VoiceOver Evaluation

Text Size: S M L XL

An Evaluation of VoiceOver, the Screen Reader in Apple's OS 10.4x Operating System

Background

This study provides an evaluation of Apple's VoiceOver screen reading application which is built into Apple's OS 10.4x operating system software. VoiceOver, part of Apple's Universal Access applications for computer users with disabilities, is provided at no extra cost despite being, as Apple claimed at the time of its release, robustly featured. The potential financial savings of a free screen reader (screen readers for Windows can cost as much as $1,500.00 CDN for a single-user license) coupled with the advantages of the OS X operating system (freedom from virus and spyware attack, Unix-based OS stability and useful features such as Spotlight searching) make VoiceOver on the Macintosh platform a potentially attractive alternative to using screen reading software on a Windows-based computer. This evaluation was therefore undertaken to help screen reader users decide whether adopting VoiceOver (and by extension, Macintosh hardware and software) is a reasonable choice based on their computing needs.

Design of Study

Our VoiceOver testing was done by a university student who is an experienced Jaws-user, in combination with a sighted observer. The role of the observer was to watch the tester interacting with the computer using VoiceOver and look for coding and elements of the user interface that the tester could not perceive or misperceived using the screen reader.

Since this testing was conducted at a university, we were particularly interested in testing VoiceOver with applications and procedures that students generally rely upon, namely: email, web browsing, productivity software, disk burning utilities, interfacing with external hard drives, etc. In order to gauge how effective the VoiceOver/Mac platform were at these tasks, we compared performance in these tasks to the current levels of access provided by Jaws version 6x-7x on a Windows XP platform. We used the Jaws as the basis for this comparison since it is the most popular screen reading program and is the one with which the tester is most familiar.

Given that individuals would have to cross computer platforms in order to use VoiceOver (there are currently no 3rd-party screen reading programs available for the Macintosh), we were also interested in finding out how intuitive the screen reading program, as well as the Macintosh OS X operating system, would feel to a Windows user. Our student tester therefore worked with VoiceOver for only 30 hours: her task was not to master the program and OS (which would have taken much longer) but to see if she could develop a basic facility with them in the time allotted.

Hardware/Software Used

We used a 1.25 GHz G4 Mac Mini with 512 MB DDR SD-RAM in combination with a Dell keyboard for our test. OS X 10.4.1 was installed on the computer.

In addition to testing built-in Macintosh applications such as Mail, Safari, TextEdit, Preview, QuickTime, DVD Player and iTunes, we also tested VoiceOver's compatibility with Adobe Reader 7.0 and Office 2004 for Mac.

Conclusion

VoiceOver was able to provide a roughly comparable level of functionality to Jaws when using email, reading PDF documents, accessing and changing system preferences, accessing external storage media and burning CDs. However, its ability to access a range of web content is more limited than Jaws and key productivity software suites, such as Microsoft Office, have not been made VoiceOver-accessible yet. Because of these limitations (as well as a number of deficiencies in terms of usability) we would not recommend the adoption of the current version to university students.*

Evaluating VoiceOver by Task

Email Access

Pros

• Tester was able to set up a mail account unaided.
• Tester quickly learned how to send and read e-mails, add/open attachments, delete email and create mail folders.
• Tester liked the auto-completion of e-mail addresses.

Cons

The tester encountered a number of minor problems:

• Intermittent problems activating hyperlinks in messages,
• Intermittent problems linking from the message list to the message text and reading this text (control-option J didn't always work)
• Tester found Apple Mail's spell check window somewhat awkward to use; wondered why spelling mistakes weren't automatically identified as in the TextEdit application.
• Process of accessing web-based e-mail applications hampered by VoiceOver not automatically identifying edit fields.

Conclusion

VoiceOver provides access to e-mail through the Apple Mail program that is comparable to Jaws.

Web Access

Pros

• VoiceOver was able to read standard web content through the Safari browser, find and select links, fill in forms, and create a list of links or items on a webpage and automatically narrow the list by typing characters.
• VoiceOver read through table cells in the proper order (although it didn't always indicate that it was reading in a table).

Cons

The tester encountered a number of limitations ranging from minor to serious:

• VoiceOver cannot read Flash.
• After identifying a frame, the program has difficulty staying within it and interacting with its content.
• VoiceOver doesn't read the alt tags in images and image links.
• Inability to copy and paste text from a webpage.
• Limited search features (e.g. can't search by headings, divisions, tables, edit fields).
• Multiple key commands to navigate/read/interact with web pages are awkward.
• System sounds that indicate an action has taken place are inconsistent (e.g. the computer doesn't always chime when a new page is loaded).
• If the user stops while reading through a page, VoiceOver automatically returns to the top of the page.
• VoiceOver is often slow to create a Link Chooser menu.

Conclusion

VoiceOver provides more limited access to Web content compared to Jaws.

PDF Access

Pros

• Tester quickly learned to read pages, move to the next page and search for a page.

Cons

• While Preview was able to copy and paste a single page it could not copy and paste a whole document.
• Preview lacks a "save as text" command currently available in Adobe Reader. The latter program is not optimised to work with VoiceOver.

Conclusion

VoiceOver provides access to documents in PDF format through Apple's Preview application that is comparable to Jaws. However, since Adobe Reader has not been optimised for VoiceOver, access to this program's additional functionality is currently unavailable.

Productivity Software

Pros

• Tester quickly learned how to read, create, edit and format documents using TextEdit.
• TextEdit has the ability to both open and save documents in Word format.
• TextEdit includes a relatively wide array of text styles, font and list/bullet options.
• Tester found the automatic auditory identification of spelling mistakes useful.

Cons

• Tester found it difficult when applying styles to text to move to the menu without losing text selection.

Conclusion

VoiceOver provides more limited access to productivity software compared to Jaws. While Apple's built-in word processing program, TextEdit, has a relatively wide feature set and can open Word documents, there is no equivalent spreadsheet program. VoiceOver is not optimised to work with the Microsoft Office Suite.

Operating System Access

Tester was able to access most relevant system settings (e.g. Universal Access, Dock, Sound, Energy Saver, Spotlight, CDs & DVDs, Keyboard and Mouse, Software Update, etc.)

Cons

• Tester had some difficulty setting the date and time within Date and Time window because VoiceOver wouldn't read until after the numerical data had been entered in the appropriate place.
• Tester had difficulty adjusting the computer sleep settings within the Energy Saver window. These are set with a slider that reads out a percentage figure and not a time value, so that the tester could only verify changes after they had been made.

Conclusion

VoiceOver gives access to system preferences that is comparable to Jaws.

Voice Quality

Pros

• The Mac OS ships with 20 built-in speech engines.

Cons

While the tester found that the quality of the more serious voices was acceptable, they felt that there wasn't enough distinction between periods and exclamation marks, and that the voices made unaccountable pauses.

Tester found it frustrating to have to search for a small number of usable voices amongst the large number of "novelty" voices that are part of the operating system.

Conclusion

VoiceOver has a more limited selection of quality voices compared to Jaws.

It should be noted, however, that better quality voices for the Mac OS were available for purchase from 3rd party companies (e.g., Acapella and Cepstral).

Customizability

Pros

• The tester quickly learned how to create keyboard shortcuts that complement VoiceOver functionality.
• The tester was able to assign different voices to different functions and train the speech engines' pronunciation of words.

Cons

• VoiceOver's speech engine settings are global and cannot be tailored to individual programs. The tester was frustrated by having to listen to auditory information that was relevant in one application but not in another (e.g. hearing "attribute" information makes sense in TextEdit but not in Mail or Safari).

Conclusion

VoiceOver has more limited options for customizing voice feedback compared to Jaws.

Multimedia

Pros

• The tester was able to open the QuickTime application with VoiceOver and use the commands to play a sound file, as well as a movie.
• The tester was able to access the DVD playback controls as well as the different tracks of a DVD through the DVD Player menus.

Cons

• iTunes has not been optimised to work with VoiceOver so many features will not work — e.g., iTunes Music Store, the ability to create a playlist of music and burn it to a CD, interface controls, etc. (It should be noted, however, that 3rd party media players like VLC can be used to play audio CDs. Also, iTunes can be accessed to some degree by using VoiceOver and Apple's new Front Row software).

Conclusion

VoiceOver gives more limited access to audio media through Apple's iTunes music application compared to Jaws and Windows Media Player. VoiceOver provides reasonable access to video media through the Apple's Quicktime player and DVD Player.

Storage Access: External drives and Burning Data CDs

Pros

• Tester was able to move data to a blank CD and burn it using VoiceOver.
• Tester was able to find and open a USB key, read the contents, move a file onto it and then eject it using VoiceOver.

Cons

• Tester was somewhat confused by the USB key showing up on the desktop and not being identified in some manner as an external drive. There's no equivalent auditory cue to the visual cue of the drive icon.

Conclusion

VoiceOver gives comparable access to such information storage devices and strategies as USB keys and data CD burning compared to Jaws.

Usability

Pros

• Through Apple's help resources, a process of trial and error and listserv support, the tester was able to learn the basic operation of VoiceOver in under 30 hours.

Cons

VoiceOver's usability is hampered by the following issues:

• VoiceOver help resources are limited
  • VoiceOver Help doesn't permit you to search for a single keyboard command.
  • Keyboard practice only tells you the name of the keys as opposed to the function of specific keyboard commands.
  • The context-sensitive help links in most application windows lack actual help information.
• Current lack of VoiceOver-accessible programs *
  • The inaccessibility of key 3rd party programs such as Microsoft Office and Adobe Reader has already been noted. Key Apple programs such as iTunes are also currently inaccessible.
• VoiceOver interface problems
  • VoiceOver frequently uses complicated, multiple-key commands, and often these commands are hard to remember because they're quite similar. The program's key lock that makes the VoiceOver keys (Control and Option) stay on, can't be left on for long because it conflicts with other key combinations.
  • Moving back and forth from the menu to the content in Safari and TextEdit is difficult.
  • The standard three-paneled finder window layout is confusing; interacting between the sidebar list and outline window can be difficult because of inadvertently changing outline content.
• VoiceOver Bugs
  • During our testing, VoiceOver was inadvertently turned off in System Preferences and once this happens there is no key command to turn it on again.
  • VoiceOver sounds in programs such as Safari and TextEdit are not always played consistently — e.g., the fairy chime sound doesn't always play when a new window opens in Safari (see above).
  • Inconsistent reading of alt tags for images and links in Safari (see above).

Conclusion

VoiceOver's usability is currently more limited compared to that which is provided by Jaws.

Suggestions for Improvement

Mail

• Address bugs, e.g., inconsistent hyperlink activation problem while in message window.
• Create an audible indication that an attachment has been successfully attached.

Safari

• Address bugs, e.g., inconsistent action sounds, problem with reading alt tags in images and hyperlinks, and inability to copy text from a web page.
• Improve VoiceOver behaviour in frames (i.e. allow a user to select a frame and stay in it in order to interact with its contents.
• Improve Flash accessibility.
• Design VoiceOver to automatically start reading text when it opens a web page.
• Enable a reader to read a page, stop in order to perform an action, and then return to the same place in the page to continue reading.
• Provide more options for searching a web page e.g., search by heading level, edit field, etc.
• Include a feature that would allow an individual to search for a word on a webpage without having to build an item list.
• Indicate audibly when a file is downloading and when the process is complete.
• Identify tables before reading them.
• Don't include pointless items in the results when using the Item Chooser menu (e.g., why include a splitter in the results? It has no navigational value).

General Suggestions

• Address bugs in VoiceOver, e.g., create a keyboard command to turn VoiceOver on if it has been turned off in System Preferences.
• Improve VoiceOver help by:
  • The ability to search VoiceOver Help for a specific key command without having to go through a list of commands.
  • The provision of more context-sensitive help information throughout the operating system.
  • A keyboard practice feature that tells users the function of specific key combinations.
• Work with Microsoft to create a VoiceOver-accessible version of Office.
• Tag currently untagged images that appear in windows throughout the Apple operating system, e.g. the magnifying glass image in Spotlight, images in alert and preferences windows, etc.
• Improve iTunes accessibility.
• Create a "save as text file" feature in PreView or work with Adobe to create a VoiceOver-accessible version of Adobe Reader.
• Identify external drives with an auditory cue i.e., VoiceOver should say "drive" or "hard drive".
• Use sounds to indicate when a USB storage device has been loaded and when it has been ejected.
• Allow users to tailor voice behaviour to specific applications instead of providing only global adjustments.
• Build in automatic identification of edit fields, pop-up menus and radio buttons throughout the operating system as well as in Safari and Mail.
• Enable VoiceOver to give auditory feedback while performing tasks like entering numerical data in date and time fields in System Preferences and not after the fact.
• Include more, good-quality voices in the operating system.
• Give automatic VoiceOver focus to all new windows.

Postscript

* Apple recently announced that the next major update to its operating system, OS 10.5 or Leopard, will include a host of improvements to VoiceOver, including:

• A high quality voice designed for screen reading
• More auditory cues to indicate what's happening on-screen
• Built-in support for braille displays and note-takers
• Improvements to keyboard navigation options

Leopard will be available for purchase in early 2007. We will need to re-evaluate VoiceOver at this time to see how many of its current limitations have been addressed.

Authors

Athol Gow,
Coordinator, Library Centre for Students with Disabilities.

Carin Headrick,
Hon. B.A., University of Guelph , 2005

The authors of the report would like to thank members of the Macvisionaries Discussion List for their advice and tips on using VoiceOver.

Top

© 2012 University of Guelph Library
Guelph, ON, Canada N1G 2W1
519-824-4120 x53617

CourseLink Web Advisor GryphMail Learning Commons

Send comments