Testing software for accessibility and usability will be needed at many stages in the lifecycle of a product. Our aim is to enable testers who are not accessibility or disability specialists to understand the principles of inclusive design, and apply what they already know about usability to a testing project.
When to test
The most effective time to test and evaluate a software product is during development, so that unnecessary barriers are not incorporated into a design. For a purchaser, it is helpful to identify any barriers that are presented by software in advance of procurement. When an interface is tailored to meet the needs of the users, it is valuable to test and evaluate as changes are made.
How to test
Behind all testing and evaluation there needs to be a set of
inclusive design guidelines or standards.
RNIB bases its software accessibility testing on an international standard, ISO 9241-171.
Testers need to apply standards as objectively as possible to the software interface, and come up with findings and recommendations that will help designers and procurers to make decisions about software in a design or procurement project.
To test a large software system effectively, there usually has to be a level of selection around what goes into the test schedule. For example, there may be several hundred screens, used to carry out a wide range of tasks by people in many different job roles. Not every screen or every task can be tested, but it is important to test a representative sample. These might be picked to represent priority tasks, or screens that are frequently accessed to carry out many tasks. It is always necessary to test the login process.
A range of screens and/or tasks can then be compared with the test statements. Some statements apply to the whole product - for example, those that refer to documentation. Often, findings about colour changes in one screen are consistent in all other screens. Other statements need to be applied to each separate screen or process. The results must be carefully logged.
It is rare that an accessibility or usability statement can be answered with a simple Yes or No. In our test results, we have the following choices: Fully Met, Partially Met, Not Met, Not Tested and Not Applicable. There has to be ample space for explanatory comments, too.
At present, we use a spreadsheet for recording purposes, where each row relates to a single inclusive design statement, and columns denote different parts of the software under test. The cell at the intersection point contains the outcome of testing and associated comments.
There is a need for a more usable recording and reporting tool for inclusive design testing that is carried out by people who are not professional testers.
Testing tools and techniques
Unlike web accessibility testing, there are no automated testing tools for software; everything must be done manually. There are one or two useful techniques that can help with certain aspects of testing on some platforms.
During testing, it's always instructive to change the way the interface is presented using the accessibility features supplied with the operating system. Enlarging fonts and other user elements on screen, and changing default platform colours can highlight aspects of the interface that may present barriers to people who rely on these adjustments for effective access to applications.
There are a number of useful online resources describing the way that computers running Windows can be adjusted to suit people with disabilities:
Also for Microsoft Windows, there is an applet called
Active Accessibility Object Inspector, available in the MSAA Software Development Kit, which can be downloaded. It is also possible to use MS Magnifier and MS Narrator to help track the visual focus, and to give an indication of whether interface information is available to assistive technologies.
The role of access technologies in testing is not straightforward. Screen readers and voice recognition systems are very complicated and feature-rich, and not intuitive to use. If a failure is indicated when using access technology in testing, it is difficult to know whether the software being tested or the user is at fault, unless the tester is very familiar with how the access technology works.
Reporting findings
The outcomes of accessibility and usability testing must obviously feed back into the design of the product, or into the selection process if testing is being done for procurement.
It is critical
to be realistic when reporting findings from testing. In one system we reviewed, containing hundreds of individual screens, not a single button had been included in the Tab order. This is a serious failure and would normally require remediation, but because of the scale of the problem it simply was not cost-effective to re-programme every button on every screen. A workaround had to be found, which was to ensure that every button had a unique and consistent shortcut key, and that these were listed accessibly for those who could not see the buttons.
The other critical factor is to
prioritise the findings. It is very important for procurers and developers to know which particular findings are the show-stoppers, so that the maximum effort can go into resolving those.
Priorities are based not only on the level of inaccessibility, but also on the frequency that users will experience the problem. A show-stopping problem in a screen used by a couple of people once or twice a year probably has a lower priority than a mildly annoying problem encountered twenty or thirty times a day by all users.
The report of findings has a different role to play in procurement, compared with during the development of a product. There will be business and functional requirements alongside the accessibility requirements in a procurement project, and all of these will guide the final selection of software and supplier. It is quite possible that a less accessible product will win the contract, although we hope that an impossibly inaccessible or unusable product would be ruled out.