Interacting with Mobile Devices via VoiceOver: Usability and Accessibility Issues Barbara Leporini CNR - ISTI v. Moruzzi, 1 - 56124 Pisa Italy
[email protected]
Maria Claudia Buzzi, Marina Buzzi CNR - IIT v. Moruzzi, 1 - 56124 Pisa, Italy {Claudia, Marina}
[email protected]
ABSTRACT
accessible to all users. In this study we evaluate the usability and accessibility of the gestures offered by the Apple touchscreen devices when the screen reader VoiceOver is running. Although recent versions of Android devices (v. 4.0) offer new advances in accessibility support for sightless persons, this work refers only to Apple devices since our research has been carried out in Italy where Apple provides the majority of devices sold (in the blind community), as our study confirms: only a few participants to the survey have used Android devices (3 of 55).
In this paper we analyze the interaction of blind users with Apple touchscreen devices iPad, iPhone and iPod touch, accessible to the visually-impaired thanks to their pre-installed VoiceOver screen reader or magnifier. Specifically, we focus on the gestures offered by VoiceOver to simplify interaction for blind users. A usability inspection of the devices’ user interfaces has been performed and integrated with user feedback collected via an online survey taken by 55 totally blind users. Results confirm that VoiceOver makes the Apple devices basically accessible to blind users, but there are still some issues related to usability. Users normally believe that accessibility integrated with VoiceOver is an important innovation, but some operations, such as the writing of long text, take too long or are not comfortable. Results suggest that a multimodal approach on mobile touchscreen devices does not offer a simple and satisfactory interaction paradigm for all and it deserves further investigation. Three possible solutions for improving user interface interaction and offering a simpler and more comfortable experience for blind individuals were proposed to the survey participants, gathering their positive feedback.
This paper first describes some accessibility and usability issues encountered when interacting with an Apple touchscreen devices (iPad, iPhone and iPod touch). Next, it presents a survey aimed at collecting qualitative feedback from blind users on the accessibility and usability of Apple touchscreen mobile devices interacting via VoiceOver. Finally, the study collects users’ opinions on possible solutions for improving the interaction. The paper is organized as follows: after a short introduction to the main investigation on mobile accessibility and the Apple VoiceOver gesture-based screen reading software, we introduce the evaluation methodology used to identify potential issues encountered when using a mobile device via VoiceOver. In Section 4 we summarize the main issues observed by users interacting via VoiceOver, while in Section 5 we report some interaction examples to better describe what is observed when using iPad, iPhone and iPod. A short discussion on the topic concludes the paper.
Author Keywords
Mobile accessibility, usability, screen reader, blind users, VoiceOver. ACM Classification Keywords
H.5.2 User Interfaces, Input devices and strategies (e.g., mouse, touchscreen); H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous
RELATED WORK
Mobile accessibility research is in constant evolution, reflecting rapid changes in technology. Research on mobile accessibility has moved from adaptation of User Interface (UI) content in a small screen to new features and interaction paradigms such as touchscreens and screen pads (single- and multi-touch input), mobile screen readers, haptic and gesture interfaces, and voice navigation functions, that can adapt to different user abilities. Schultz et al. discussed the concept of a Universal Mobile Device intended as a multifunctional accessible and usable mobile device for everyday user activity, and described technologies currently available for developing such solutions (Schulz et al., 2010) Although blind users rely on touch when using mobile phones, few studies have investigated “the influence of touch in non-visual mobile interaction” (Benedito et al., 2010) and creating accessible touchscreen interfaces for the blind user remains a challenge (Arroba et al., 2011; Bonner et al., 2010; McGookin et al., 2008).
INTRODUCTION
Mobile devices are increasingly used in daily life for activity including learning, entertainment and access to egovernment services. Smart phones and tablet devices are a cheap alternative to computers for increasing the literacy of people in developing countries, which unfortunately includes a considerable number of disabled persons. Mobile accessibility should be available to anyone anywhere, enabling e-inclusion and reducing the digital divide. Specifically, when considering mobile accessibility we need to focus on the entire input interaction paradigm (touch, gesture or voice interaction) as well as output modalities, that is, how to convey the displayed textual and graphical content. The latest smart phones as well as PC tablets based on Android OS (www.android.com) or Apple IOS 5 (http://www.apple.com/ios/) systems are designed to be OZCHI’12, November 26–30, 2012 - ACCEPTED VERSION
1
Previous studies have investigated the use of tactile overlays or customized hardware associated with speech. However, tactile overlays limit interface flexibility since elements have to match the physical coverage. To resolve these issues, Kane et al. proposed a specialized speechbased interface optimized for non-visual browsing on multi-touch devices. Although no visual representation is used, objects on the screen are spatially arranged using linear lists. Users navigate by scanning their fingers down the device screen, and use gestures to interact with objects (Kane et al., 2008).
accuracy and agility, and is able to quickly discover and absorb its organization. Koskinen et al. investigated the most pleasant tactile clicks, comparing piezo actuators vs a vibration motor, finding that subjectively the first was preferred (Koskinen et al., 2008). In agreement with previous studies, results showed that tactile feedback improves the usability of virtual buttons pressed with the fingers, since the user is able to feel the object of interaction. Brewster and Brown proposed the use of a new type of tactile output: tactons, or tactile icons, structured, abstract messages that can be used to communicate information. A Tacton is characterized by parameters such as frequency, amplitude and duration of a tactile pulse, but also rhythm and location (Brewster et al., 2004). Using tactons could enhance accessibility of mobile devices for blind users as well as for sighted people in movement. Qian et al. identified the salient features of tactons when integrated with a cellular telephone interface. Findings indicated that the best results use simple static rhythms, with differences in each pulse’s duration. However, to ensure accurate perception, the dimensions in which paired tactons differ should be limited (Qian et al., 2009). Yatani and Truong proposed the use of multiple vibration motors embedded in the back of the mobile touch-screen device for conveying tactile feedback providing semantic information about the touched object. They show that users can accurately distinguish ten vibration patterns, and that the proposed system enables better interaction in an eyes-free setting than devices without tactile feedback or using only a single vibration motor (Yatani et al., 2009).
In the field of accessible design, Wobbrock et al. argued that "just as user-centered design shifted the focus of interactive system design from systems to users, abilitybased design shifts the focus of accessible design from disability to ability". Thus, the focus moves to empowering users by designing for his/her abilities (Wobbrock et al., 2011). When interacting with mobile devices, individual differences between blind persons have great impact on their proficiency levels. In a study with 13 blind people, Guerreiro et al. observed that different ability levels have a significant impact on task effectiveness and efficiency. With a detailed characterization of how different users relate to different interaction modalities, it is possible derive a model for predictions regarding the performance of particular user/modality pairs (Guerreiro et al., 2011). Oliveira et al. present two new text input methods that transform the meaning of the mobile device keypad while exploring the user’s capabilities. NavTap allows the user to navigate through the alphabet using the device keypad. BrailleTap modifies keypad functionalities on the basis of the Braille alphabet. User studies validated the proposed approach, which also earned user approval (Oliveira et al., 2011).
VoiceOver and Gestures
Apple devices are becoming more and more accessible to people with disabilities, thanks to assistive technologies included in its products. VoiceOver is the Apple gesturebased screen reader available in Mac OS and iOS operating systems, that makes products such as iPhone, iPad and iPod touch accessible (www.apple.com/accessibility/). Using VoiceOver, these products can provide spoken feedback so people who cannot see the screen can still use them. When a user selects an element, VoiceOver reads the name or describes the item. The VoiceOver gestures can be used to physically interact with items on the screen, allowing the user to move around the screen and control the selectable individual elements. VoiceOver gestures use one, two or more fingers to tap, drag or flick1.
Kane et al. compare how blind and sighted people use touchscreen gestures and propose guidelines for designing accessible touchscreens. Blind subjects prefer gestures that use screen corners, edges, and multi-touch (enabling quicker and easier identification) and suggest new gestures in well-known spatial layouts (such as a qwerty keyboard) (Kane et al., 2011; Kane et al., 2008). Recently, following accessibility principles Bonner at al. developed an eyes-free text entry system that uses multitouch input and audio output. The system, implemented on Apple’s iPhone and tested with ten users, showed better performance in terms of speed, accuracy and user preference than the text entry component of VoiceOver (Bonner at al., 2010). Arroba et al. proposed a methodology for making mobile platforms with touchscreen input accessible for visually impaired people. The proposed solution is based on a functional gesture specification, a set of guidelines to assure consistency of mobile platforms and the customization of input application (Arroba et al., 2011).
Starting from IOS 5, new recent Apple portable devices are equipped with Siri, i.e., a voice recognition functionality (http://www.apple.com/ios/siri/). Siri is an 'intelligent assistant' voice-control which allows one to send messages, schedule meetings, take notes and more. Unfortunately when carrying out our survey with the users, Siri was not available in Italian. For this reason we decided to not include specific questions on speech
Concerning haptic solutions, several studies in literature describe the importance of providing the user with appropriate mechanisms and techniques for better orientation on the user interface. In (Goble et al., 2000), for the authors, organized content is of benefit for a reader only if (s)he is able to move around it with
1
2
http://manuals.info.apple.com/en_US/ipad_2_user_guide.pdf
recognition since we conducted our study only in Italy. In addition, another aspect to consider when using a speech recognition system is related to privacy issues (when it is necessary to use the Siri functionality in a public context such as on a train, on the street, or in any other situations where there are other people). Thus, making the UI accessible and usable also via gestures and any other interaction modalities that can be used in a public context is very important.
(e.g., main content) or action/function buttons (e.g., OK, Done, Cancel, Next/Previous page, etc.). Furthermore, some other relevant activities were also considered, such as checking correctness of edited input in text fields (to verify that no mistakes have been introduced), because that is a natural activity for a blind person when filling in forms or composing numbers. These aspects are crucial for enabling rapid navigation of content and allowing satisfactory interaction with the devices and their Apps.
EVALUATION OF VOICEOVER
To detect potential interaction problems, we first selected three Apps having these aspects and UI features affecting the logical visiting order of elements, forms with interactive elements to be filled and UIs with several editing fields. Next, we identified the actions and tasks to be performed in order to interact with the UI and its elements: (1) sequential navigation through “next” and “previous” (left and right swipe) gestures; (2) editing task, such as typing a phone number; (3) text and content fields/areas with considerable information (e.g., list of messages and single e-mail message).
To understand how a blind user can interact with a VoiceOver-based mobile device, we selected some actions and applications available on the devices. In this section we introduce the evaluation methodology, in the next one we summarize the main issues observed by the users, while in the Interaction examples section, we report some examples to show what has been observed. Method
To detect the most common accessibility and usability issues that arise when interacting via VoiceOver with a mobile device (iPhone, iPod or iPad), we proceeded with an inspection evaluation and a remote survey-based test. The first method was used to find the most relevant interaction issues due to touchscreen use when VoiceOver is activated. Later, we created a questionnaire to verify these issues and their impact for end users, and to collect comments and suggestions.
Online Survey
According to the aspects identified by analysis as well as the main accessibility and usability issues observed through the inspection evaluation, we drew up an electronic questionnaire to collect information and suggestions by blind people. The questionnaire was composed of 32 questions grouped in six parts:
All three authors of this paper participated in this study by conducting the inspection evaluation. They have good knowledge of accessibility and usability of user interfaces for the blind; one, totally blind since childhood, is expert in interacting with screen readers also on mobile devices (VoiceOver as well). The two sighted users interacted with the devices by using a screen curtain, i.e., a VoiceOver feature that turns off the display so no one can read it, to guarantee user privacy. In this way the sighted testers were able to simulate blind (novice) user interaction.
(1) user characterization (2) general use of the devices specific use: (3) numeric keypad, (4) text editing, (5) focus (6) comments and suggestions. The questionnaire was created using the Google Docs suite (https://docs.google.com/) in order to make it available on line through a Web-based form. It was disseminated throughout the visually impaired community in Italy via email to general and specific mailing lists of the Italian Association for the Blind (e.g., communities interested in assistive technologies or mobile devices). We also included three questions about possible features that should be added in order to enhance user interaction with a touch-screen:
Inspection Analysis
To carry out the inspection we first identified potential tasks and activities to perform on the UI to analyze critical aspects of interaction. Thus, we identified some key aspects to consider when evaluating the gestures available through VoiceOver: 1.
2.
3.
1. 2.
clarity of user interface and interactive elements; i.e., whether the elements have appropriate labels and are clearly announced as interactive items. logical navigation order of contents and elements; i.e., whether the content (phrases and paragraphs) as well as the interactive items (links, buttons, edit field, etc.) are visited in the appropriate order when moving through the “next” and “previous” gestures (left and right swipe gestures). The focus should follow the logical order of the visit in the user interface. quick and easy identification of elements or content; i.e., whether it is easy to reach a specific text portion
3.
four buttons positioned at the four corners vibro-tactile support to better detect main interactive items (e.g., buttons and links) tactile markers to help the user easily recognize the macro areas or current focused UI elements/part.
The Google Docs platform allows quick and easy creation of electronic forms automating the gathering and analysis of data, offering great advantages in terms of efficiency and effectiveness. Accessibility of the UI to create the questionnaire has not been considered in this work and merits further investigation. We have only considered the accessibility of the generated electronic questionnaire (i.e., the resulting form). In fact, using pre-defined elements may impact on accessibility via screen reader. 3
To create forms, Google docs offers the possibility of inserting grid and scale elements, but they present usability problems when accessed via screen reader, since labels are not associated to radio buttons. Thus the text is announced sequentially: first all labels are announced (first row) and then all the radio buttons are read, making it difficult for the user to link the elements (additional cognitive effort required). For this reason, we used simple radio button items (the multiple choice element), also for building the evaluation scale.
order, especially when expanding an item (e.g., contact details in the “Contacts” App). Unsuitable handling of focus. This issue especially occurs when editing a text field while filling in a form composed of several control UI elements. In fact, when activating an edit field by a double tap, the system focus moves on that field and the VoiceOver announces the editing modality. The virtual keyboard appears on the lower part of the screen, so the user understands that (s)he can write the text. By exploring and clicking on the keyboard letters, the focus moves on the keyboard and the user loses the editing point because (s)he is not able to quickly check what was edited. To do this, it is necessary to explore the UI again. This issue also arises when filling in a group of form elements. When a text is edited, to proceed with the next form control the UI must be explored again and the desired element activated. This process disorients the user and could make the action difficult and frustrating.
To render the questionnaire easy to use via screen reader, we numbered the questions, associating them with section headers, which provide questions tagged with an H2 heading element. The form title is tagged with an H1 heading element. In this way the user may quickly move from one question to the next (or previous) with a key press (e.g., “h” for the Jaws2 screen reader) and immediately listen to the question number. The question title has been kept short, since when there are multiple lines the screen reader announces the heading elements several times, annoying the user with information overload. The full title has been inserted in the question item title. In this way the user can quickly reach the question by reading only its number and a very short description; (s)he can then read the whole question by using the arrow keys. When a question is answered, by pressing the command key (s)he can skip directly to the next one. Last, we decided not to set all the questions as mandatory, risking the loss of some answers but avoiding the stress for the user of looking for any missing answers.
Questionnaire Analysis
The questionnaire was filled out by 55 totally blind individuals; 20 were women and 35 men; age ranged from 19 to 78 years, mean age 40 years. The users’ devices investigated include: iPhone 3GS/4/4S, iPod touch 4, and iPad2, as shown in Fig. 1; the iPhone4 is the most popular/widespread device of the sample. Concerning experience using the devices, 54% of the sample (30 users) have used the smart devices for less than 1 year while 33% (18 users) have done so for 1 to 2 years and 13% (7 users) for more than 2 years.
RESULTS Accessibility and Usability Issues
After the inspection usability evaluation via VoiceOver, we can affirm that Apple mobile devices are basically accessible since their content as well as the main activities can be carried out by a blind user through the VoiceOver gestures. Problems are mostly related to the usability aspects of carrying out some activities. The main observed issues can be summarized in:
Figure 1. Number of participants using iPhone, iPod, iPad
Lack of clarity of interactive elements. Usually when the focus is over interactive elements (such as buttons or links), VoiceOver announces the type of element. Unfortunately, in some cases no such indication is provided for elements. For instance, when navigating the available categories in the Apple App Store (a digital application distribution platform for iOS through the iTunes Store) there is no functional information for the user that the element can be activated and how (by a double tap). The user perceives such content as a text string and no additional data is added to make the element clearer.
The most common activities performed with the devices include: Phone calls, Read/write SMS, E-mail, News and newspapers, Music, and Internet navigation, as shown in Fig. 2. The following activities are: GPS Navigator, Audiobooks, Reading e-books in ePub format, Access education and learning content, Podcasting (radio broadcasts, lectures, news), Reading e-books in PDF format, Notes (written), Memo (audio). It should be noted that only 50% of the sample use Entertainment and Games, pointing out as these are still not very accessible for blind users. Concerning gestures for interacting with applications, most participants are satisfied: 49% (27 users) consider them very functional/easy and 45% (25 users) functional/easy, while only one user believes they are very difficult, as shown in Fig. 3 (two users do not answer). However, comments highlight that initially many users had difficulties, but thanks to manuals, help sections, information on-line, and exercises (activated by accessibility support), initial problems were overcome
Lack of logical navigation order. When navigating the content and elements sequentially via “next” (flick right) and “previous” (flick left) gestures, some incongruence occurs when visiting a correct logical 2
JAWS for Windows Screen Reading Software: http://www.freedomscientific.com/products/fs/jaws-productpage.asp
4
and as some users reported “once you master the gestures everything is natural and indispensable”.
Figure 5. Touchscreen use learnability
Concerning the usability of the keypad, typing numbers with the numeric keypad was considered less problematic than typing letters: only 4% and 25% of the sample judged that typing numbers is not functional at all or not very functional, vs 27% and 25% for typing letters. However most participants did not have difficulty: 36% and 25% of the participants judged the numeric keypad functional and very functional for typing numbers; the percentage decreased to 27% and 9% when typing letters. Only 4% of the participants thought for both questions that it is neither difficult nor easy, while 6% (numbers) and 8% (letters) did not respond (Fig. 6).
Figure 2. Activities performed on mobile devices
Figure 3. User satisfaction on the use of gesture
As seen in Fig. 4, the use of audio for signaling events such as the start/close of applications, deleting operations or confirming actions, is considered very useful by 75% of the sample (41 users), useful by 18% (10 users) while only one user believes that it is not very useful (3 users do not respond). The usability of HW keys is acknowledged as functional by 44% and very functional by 49% of the sample (7% of users do not respond).
Figure 6. Keypad usability: typing letters vs numbers
Concerning the identification of key functions on the numeric keypad (delete, calls, contacts, etc.) there are some issues since the total of persons who consider this function not very practical are 5% (very difficult), 16% (fairly difficult), 20% (with some difficulties) while 29% and 20% consider it sufficient and very simple respectively (Fig. 7); 10% of participants do not answer.
Figure 4. User satisfaction on audio feedback Figure 7. Ease of Function Keys
Regarding the initial use of the device, learning how to interact with the touchscreen was considered simple and very simple by 62% (34 users) and 24% (13 users) of the sample respectively. One user believed that the learnability of the touchscreen devices is very difficult, two considered it difficult and three that it is neither difficult nor easy (Fig. 5). Two users did not respond. In several comments, users expressed their enthusiasm for the use of touchscreens, for instance: “At first it takes time to learn the gestures and run it properly on a smooth surface but after the initial learning period it’s very instinctive and easy!”
Regarding the editing mode, both for typing a phone number (to confirm each digit) and typing a text (to confirm each letter/char) 75%-73% of participants prefer the editing mode with a single tap or finger, while only 11%-13% prefer the standard mode (double tap), as shown in Fig. 8. When typing a phone number, a visually-impaired user usually checks the string entered (to verify the number) but this action is judged not (or not very) practical for 16% and 22% of responders. Similarly, concerning the rechecking of typed text to verify whether there are mistakes, 13% and 27% of participants believe that this is not or not very practical respectively, while 36% and 20%
5
using the "contacts" application?” we gathered only a few responses (20%) since only a minority of the sample have this device (5 users of 55) and only a few participants have had the possibility of trying it; 4% of these believe that the management of focus is not usable (2% not usable at all and 2% not very usable) while 11% and 5% of the sample think that it is functional or very functional respectively (Fig. 12, first bar). The management of the focus for reading the details of a contact using the "contacts" application on iPhone and iPod devices is insufficient for 7% of the sample, good enough for 55% and very functional for 18%; 7% of the sample do not know how to answer (Fig. 12, second bar).
of the sample judge it sufficiently and very usable; 4% do not answer. Answers to these two questions are aggregated in Fig. 9.
Figure 8. Preferred editing mode: numbers vs letters
Figure 12. Focus management: iPhone/iPod vs iPad
Concerning the usability of reading emails on the iPad, 4% of the participants think that it is difficult while 5% and 9% believe that is simple enough or very simple respectively (Fig. 13, first bar). Concerning the usability of reading emails using iPhone or iPod devices, most of the users agree that is simple enough (47%) or very (31%) simple while only 2% and 5% believe it is very difficult and difficult, respectively (Fig. 13, second bar). Due to the low number of answers for the iPad device, the collected data is scant for deducing whether the iPad with its larger screen may improve or decrease usability, so further studies are needed.
Figure 9. Ease of typed text/numbers checking
Concerning the usability of the qwerty keyboards, participants have differing opinions: 2% and 27% judge it not functional at all or not very functional, 9% do not express an opinion, while 38% and 16% believe that it is sufficiently or very functional (Fig. 10); 8% of users do not answer.
Figure 10. Qwerty keyboard usability
Regarding the use of punctuation and symbols, users are divided in opinion regarding ease: 7% and 36% think that it is not or not very functional respectively, while 35% and 15% believe that it is sufficiently or very functional respectively (Fig. 11); 7% do not answer.
Figure 13. Reading email: iPhone/iPod vs iPad usability
The last three questions of the questionnaire concerned possible improvements on the interaction with VoiceOver and touchscreen devices (answers were grouped in Fig. 14). All the proposed features were judged very useful by most users. The first question asks “Imagine having in the 4 corners of the device the most frequently used (virtual) function keys, such as "OK", "Back", "Cancel", "Submit", "End", etc., according to the context of the application. Do you think that having 4 buttons in a standard (wellestablished) position could speed the use of applications?” -- 80% of the sample agree with this possibility, 13% disagree, 5% do not express a preference (answer: 'I don't know') and 2% do not answer (Fig. 14, first green bar).
Figure 11. Use of punctuation and symbols
A common complaint of participants is the difficulty typing popular punctuation chars (such as: . , : ? !) present in the qwerty keyboard since using them requires switching to another keyboard and displaying the alternate characters (with the double tap pressed), so it is time consuming. For the specific question “With the iPad, as you judge the management of the focus to read the details of a contact
6
different position of the speaker such as at the top along the border could make listening more comfortable. INTERACTING EXAMPLES
In this section we report three interaction examples on Apple devices in order to show the issues summarized above. User Interface Elements: App Store Figure 14. Potential UI improvements: Tactile references, Vibration and Virtual function keys
Exploring UI elements is possible via the “next” and “previous” (left and right swipe) gestures. Exploring the “Home” page(s) is clear, easy and no very significant problem is perceived, but some minor problems are present; for instance, when exploring the touchscreen to find an element located on the first row – i.e. immediately below the “status bar” – the user could accidentally tap one item of the status bar (e.g., the battery level or time clock icon). This may cause a focus problem: if the user proceeds through the “next” and “previous” gestures the focus remains within the status bar. Although conceptually correct, this can create some usability problems. The user has to tap another place – possibly around the center of the touch-screen or close to the upper edge. Being able to clearly detect the status bar might improve element detection.
One interaction issue for a blind user is that common frequently used functions may appear in different places in the interfaces of different apps. The possibility of having four app-independent buttons in the same positions might make common actions faster. The second question was: “Supposing there is on the mobile device a combination of vibro-tactile feedback for the identification support of the most significant buttons of the application (e.g., OK, Back, Cancel, Send SMS, Call, Edit, keypad numbers, etc.); do you think it would significantly improve the use of the device?”. Of participants, 53% agree with this proposal, 27% disagree, 13% do not express a preference (I don't know) and 7% do not answer (Fig. 14, second yellow bar). Another interaction problem for the blind person is the location of key points of the user interface. Vibro-tactile feedback might help to rapidly detect important UI items. An alternative way for rapidly locating interesting interface items is to have dynamic tactile markers. The last question was “Imagine that at the edges of the touch-screen display tactile references (e.g., dots of different sizes or shapes) can be inserted that according to the application context and to current state, may appear in certain positions; do you think that it might be useful as a reference for locating specific points of the interface, such as: an edit field or the status bar or buttons for more frequent actions (e.g., change, cancel, end, etc.)?” -- 49% of participants agree with this proposal, 33% disagree (no), 7% do not express a preference (“I don't know”) and 11% do not answer (Fig. 14, third blue bar).
Figure 15. App Store screen shot -- available categories
Other difficulties in the navigation order of the elements can observed when exploring the App Store to search for an App by category. Figure 15 shows the iPad screen with the App Store opened on the available categories. With a four-finger tap, the focus moves to the first element of the current window, i.e., onto the “Choose a category” heading, making it possible for the user to explore the page from the top. Using the “next” gesture (i.e., flick right), the UI elements are announced by the voice synthesizer and can be activated by a double-tap. Table 1 reports the list of the elements in the order they are “touched” by the “next” gesture. The item labels announced by VoiceOver are listed on the left, whereas additional information on that element is reported on the right.
Useful feedback has been collected from user comments. The main drawbacks detected by users concern the performance of some operations such as writing long text or emails. For this purpose several users employ an external keyboard in order to make editing faster. In addition, the simplification of complex gestures such as a long double tap to be confirmed with a third one to close an application, may be also performed via a virtual button. Also, the voiceover settings could be enriched in some features such as automatic language handling. Last, some users reported difficulties using the contact app in all devices. Some users observed a common issue concerning the location of the audio speakers. Specifically, listening to VoiceOver via the speakers in a noisy environment may be difficult. In fact, to hear the voice synthesizer better, the blind user brings the device close to their ear, but the speaker position, at the bottom or under the device, makes the activity not natural. A
The sequence of how elements are announced can be difficult and frustrating for the user. For each category some available Apps (generally three) are listed near the category name. Unfortunately, the visiting order is not well-designed so that the user can be confused, and just
7
reading the available categories (to explore them) can require time and effort. For instance, to read only the first three categories (Table 1, rows 4, 9 and 14), the user has to read 15 rows of confusing or useless information, such as the images for the categories (that correspond to the image of the first App of the category): these images do not provide more information for a blind user but rather make the user confused about the double reading of the item (for the image and for the list element, before and within the category, see Table 1, rows 3 and 5, 8 and 10, 13 and 15). UI labels announced
The absence of hardware keys can lead to some difficulties, for instance in editing quickly or when the user is in a noisy environment. A blind person is used to editing a number quickly and easily through hardware keys. This is feasible also in a uncomfortable situation such as a noisy environment. On the contrary, when using a virtual numpad, there is no hardware reference for simply detecting the number ‘5’ (the central one for better orienting oneself) as usually occurs for the phone with a hardware numpad. Consequently, the user is forced to rely only on an aural announcement. This could create some issues and difficulties when the user is not able to hear the voice synthesizer very well. Some additional feedback to help the user edit the numbers in a more suitable situation would be desirable.
Type of UI element announced
1
Choose a category
Heading
2
Search
Search field
3
Racconta storie
Image
4
Books (libri)
5
Racconta storie
6
Diabolic
7 8
Le bugie hanno le gambe lunghe e tacco List end 10 Splashtop Remote Desktop for iPad Image
9
Business (Economia)
10
Splashtop Remote Desktop for iPad
11
Office2HD
12 13
Quickoffice Connect Mobile Suite for List end iPad Complete Italian dictionary Image
14
Education (Istruzione)
15
Complete Italian dictionary
List start
...
...
List start
List start
Figure 16. Phone call
Other usability issue concerns the mode to check (i.e., read) the edited number. Reading the edited number allows checking the number’s correctness before making a call. This could require additional effort. When editing a number, the focus is over the numpad keys. When the user edits a digit, the VoiceOver reads aloud it and a phone tone is emitted as a non-speech feedback. This occurs digit by digit. When the user would like to check whether the entire phone number is correct (s)he must read it. So, in order to read the edited number at any time, the user has to move the focus onto the entry field (i.e., the field labeled by ‘phone number’), finding this field and tapping on it to move the focus there. As the tap is done on the field, the VoiceOver reads the number. Unfortunately, the phone number field is close to (i.e., immediately below) the status bar. Therefore, if when tapping over it the focus moves towards the status bar rather than over the phone number, the user must repeat all the steps to try again. This issue occurs in other editing situations. Some commands or gestures to obtain information on the focused edit field could overcome this inconvenience.
Table 1. VoiceOver sequence of App Store category (bold)
Other issues are related to the detection and announcement of the UI element types. The names of categories are linkable (to open the related category showing the Apps available). Unfortunately, the blind user is not provided with detailed information on the name of the category (see Table 1: nothing is announced as 'category' type - rows 4, 9, 14). The same behavior occurs for the elements reported per each category. By a double touch on a displayed element, the corresponding category is opened. However, this kind of behavior is not suitable for logical navigation, since it is a sort of inconsistence of the UI interaction. Summarizing, the issues observed for the UI elements are mainly related to their logical sequence for navigation order as well as inconsistence in the interactive elements both for labels and additional functional information.
Regarding the function buttons available at the bottom of the phone interface (contact, call, add to contacts, and so on), some improvements could make calls more userfriendly, for instance if the button “call” is easy to find, making a call is quicker and more comfortable.
Making a Call
Since the phone module is an important activity on the iPhone, we analyzed how easy it is to make a call. The user can select a contact from a name list or can type the number by using the numpad. Let us consider the second case: editing a number through the numpad available on the iPhone (a similar situation will occur for the iPad device).
Reading and writing e-mails
Let us consider the mailer App available on the iPhone/iPod device. In particular, we consider the incoming message list as well as the User Interface when a message is opened to be read or written. Related to the list of incoming messages, the reading is fully accessible: 8
through the “next” and “previous” gestures the focus moves correctly on each message, VoiceOver announces the sender’s name, a portion of subject field and the number of attachments, if any. A double tap over a message opens its interface. The focus moves correctly in a sequential way over the message components when using the “next” and “previous” gestures. However, some usability issues are encountered when quickly trying to detect the main parts, such as the message body or the address field (e.g., to check whether the mail has been sent to more recipients). Generally speaking, if the user wants to read all the fields (i.e., from, to, bcc, etc.) in a sequential way, the focus catches (and the VoiceOver reads) all of them. However, since the user had probably already read the sender’s name and subject from the message list – before opening the e-mail – (s)he might be interested in immediately reading the body content, but being able to quickly jump to this point might be not practical. The user needs to try via taps to localize the desired part (e.g., message body); this is feasible – i.e. accessible – but may require additional effort.
the top right area in order to find the two buttons. Since they are close to the status bar, it could happen that when tapping on the screen the user touches the status bar (e.g., the battery level or time clock icon); consequently proceeding via the “next” gesture makes it impossible to find them since the focus is now within the status bar. The user can only try again to tap just a bit below to find the two buttons, making the interaction not be comfortable. As a solution, the two buttons could be placed in two different corners (i.e., on the top right and top left). Additionally, a vibro-tactile response could be useful for detecting them. Moreover, no specific gesture for skipping from one message to another one is available (e.g., using three or four fingers) as for other Apps (i.e., for the home page when more than one page is available).
Figure 18. Reading a message
The same can occur when writing a message. At the top right two buttons “Send” and “Add an address” can lead to the same usability issue. Also in this case a possible solution is separate them in two different locations. However, the status bar can be a problem even for the solution based on the buttons on the corners. Another position for the status bar or some markers to help clearly identify the corners could help.
Figure 17. Writing a message
The same also occurs in the editing interface when writing a message. In this case the presence of the qwerty at the bottom of the interface requires further effort to move the focus from the keyboard over to the body to be able to check what was written up until that moment. In addition, checking the last sentence or words is not simple. In fact, the focus is over the qwerty keys; to read the last sentence the user can (1) move the editing cursor letter by letter through the up and down gestures or (2) use the focus by a double tap over the message body. In the first case, the inconvenience is due to the fact that after reading the content, the editing cursor has to be moved to the end (i.e., at the editing point) to proceed. Via the second way, the action may require several steps because when making a double tap over the message body the focus goes over the part, but it is necessary to explore the content to find what is to be verified. These two modalities can lead to frustration or dissatisfaction for the user.
Moving among the fields, adding contacts in the address field or skipping field by field can require a lot of effort. Although the UI is accessible, some additional features as well as tactile responses could improve user interaction via VoiceOver. DISCUSSION
In this paper we considered the interaction via VoiceOver with the main features available on the Apple devices that have pre-installed VoiceOver screen reader: iPad, iPhone and iPod touch. The investigation aimed at identifying the main accessibility and usability problems for the blind when using the gesture-based screen reader VoiceOver. As reported in previous sections, interaction through the available gestures reconfigured by VoiceOver is fundamentally accessible for a blind user, but not very usable. The main usability and simplicity issues are:
Another important issue occurs when reading more than one e-mail when the message interface is opened. The interface provides two buttons to be used for skipping from one message to the previous or next one, which are both located at the top right of the UI (see Fig. 19). Reaching those functional buttons can require additional effort because when a message is opened, being able to easily detect the “previous message” or “next message” buttons could require several steps. The user can try to tap
lack of clarity in providing appropriate details of interactive elements logical navigation order of elements when expanding an item or contents focus handling when filling out a form. Users also asked for the more common punctuation marks
9
designers. All these solutions aim to make blind users’ frequently performed actions more rapid and effective, reducing the number of required steps and errors, and providing the user with a more satisfying experience.
(e.g.: . , ; ?) to be directly available in the letter virtual keyboard without having to switch to another symbol virtual keyboard. Although fully accessible, these activities are difficult and laborious for a blind user. To improve user interaction for some common actions or checking, additional gestures or tactile references should be considered. In particular, a specific gesture or button should be made available to simulate the Tab key behavior and the command for reading the current line, as offered by several screen readers (e.g., JAWS for Windows). Adding control elements at fixed reference points (such as near the edges or the four corners) enables easy and rapid detection of functions. This is useful for carrying out specific actions more rapidly (e.g., reading the current line or moving to next field). As highlighted through the examples, impractical activities and uncomfortable actions are often due to the status bar that is very close to the App interface. In addition, the user has to find the function buttons such as “Done”, “OK” by exploring the entire UI. Furthermore, the lack of touchable elements makes a user’s perception of the UI element insufficient. Although users declared that the use of the numpad and qwerty is practical, they expressed positive interest regarding specific questions on potential features based on tactile response as well as on positioning buttons in specific places (such as the corners). This means that additional characteristics and accessibility support merit further investigation.
REFERENCES Apple, iPad 2 User Guide (For iOS 4.3 Software), March, 2011. http://manuals.info.apple.com/en_US/ipad_2_user_guide.pdf Arroba, P., Vallejo, J.C., Araujo, A., Fraga D., Moya, J.M. A Methodology for Developing Accessible Mobile Platforms over Leading Devices for Visually Impaired People, In Proc. Ambient Assisted Living, LNCS Vol. 6693 (2011), 209-215. Benedito, J., Guerreiro, T., Nicolau, H., Gonçalves, D. The key role of touch in non-visual mobile interaction. In Proc. MobileHCI '10, ACM, New York, USA, (2010), 379-380. Bonner, M., Brudvik, J., Abowd, G., Edwards, W.K. No-Look Notes: Accessible eyes-free multi-touch text entry. In Proc. Pervasive 2010, LNCS Vol. 6030, (2010), 409-426. Brewster, S., Brown, L. M. Tactons: structured tactile messages for non-visual information display. In Proc. AUIC '04, A. Cockburn (Ed.), Vol. 28. Australian Computer Society, Inc., Darlinghurst, Australia (2004), 15-23. Goble, C., Harper, S., Stevens, R. The travails of visually impaired web travellers. In Proc. Hypertext 2000 (2000), 110. Oliveira, J, Guerreiro, T, Nicolau, H, Jorge, J, and Gonçalves D. 2011. BrailleType: unleashing braille over touch screen mobile phones. In Proc. INTERACT 2011, LNCS Vol. Part I. Springer-Verlag, (2011), 100-107. Guerreiro, T., Oliveira, J., Benedito, J., Nicolau, H., Jorge, J., Gonçalves, D. Blind people and mobile keypads: accounting for individual differences. In Proc. INTERACT 2011, LNCS Vol. Part I (2011), 65-82.
CONCLUSIONS
Mobile devices, especially tablets, are a new frontier in technology. As laptops decrease in size, and tablets increase resources and elaboration power, in the near future this will lead to handheld devices offering natural interaction paradigms, involving more and more voice and gesture. This will open up new horizons for disabled persons who may find great opportunities for e-inclusion, if designers of devices and applications keep in mind accessibility and usability for all. This preliminary study provides qualitative data that offer better insight into issues faced by blind users when interacting with Apple touchscreen devices through VoiceOver. User feedback collected from 55 blind users expressed user satisfaction for VoiceOver mobile devices but highlighted usability problems. The usability inspection, aimed at a detailed analysis of user interaction via VoiceOver with the main App pre-installed on the three devices, confirmed the presence of these usability issues. Possible solutions based on a multimodal approach merits investigation in order to propose new opportunities for easier and more satisfactory interaction with a touch-based screen.
Kane, S.K., Wobbrock, J.O., Ladner, R.E. Usable gestures for blind people: understanding preference and performance. In Proc. of CHI '11, ACM (2011), 413-422. Kane, S.K., Bigham, J. P., Wobbrock, J.O. Slide rule: making mobile touch screens accessible to blind people using multitouch interaction techniques. In Proc. of Assets '08. ACM (2008), 73-80 Koskinen, E., Kaaresoja, T., Laitinen, P. Feel-good touch: finding the most pleasant tactile feedback for a mobile touch screen button. In Proc. ICMI '08. ACM, New York, NY, USA (2008), 297-304. McGookin, D., Brewster, S., Jiang, W. W. Investigating touchscreen accessibility for people with visual impairments. In Proc. of NordiCHI '08, ACM (2008), 298-307. Qian, H., Kuber, R., Sears, A. Towards identifying distinguishable tactons for use with mobile devices. In Proc. Assets '09. ACM, New York, NY, USA (2009), 257-258. Schulz, T., Fuglerud, K. Universal Mobile Device (UMD)— Methods, Inventory, and Dissemination, Note DART/2010/01, January 29 (2010). Wobbrock, J.O., Kane, S.K., Gajos, K.Z., Harada, S., Froehlich, J. Ability-based design: Concept, principles and examples, ACM Trans. Access. Comput. 3, 3, Article 9 (2011).
In future work, three prototypes of user interfaces incorporating the three proposed solutions for enhancing usability of touchscreen devices, i.e., using 1) virtual application-independent function keys 2) tactile point of reference, and 3) vibro-tactile feedback will be implemented and tested with blind users, in order to better evaluate aspects of great importance for mobile
Yatani, K., Truong, K. N. SemFeel: a user interface with semantic tactile feedback for mobile touch-screen devices. In Proc. UIST '09. ACM, New York, USA (2009), 111-120.
10
