By: Elizabeth Neal

October 8th, 2004

Reader Rating: 9.5

You've read the usability articles on SitePoint.com and elsewhere, and you're convinced! It's time to get serious about testing the usability of your Website. Now you just have to convince your boss to spring for an elaborate laboratory with up-to-date computer equipment on one side and video monitor screens on the other, divided by a soundproof wall and a one-way mirror. Right?

Wrong. Although the James Bond-esque appeal of a high-tech lab cannot be denied -- video cameras mounted on the walls and ceiling with pan, tilt, and zoom controls allowing you to follow participants' movements! Video editing equipment so you can create a concise presentation to summarize your findings! Auxiliary keyboards and mice to control participants' computers from the adjacent room! Keystroke logs of all user input! -- it's probably not necessary for your purposes.

Even Jakob Nielsen, one of the earliest and most vigorous proponents of usability testing, advises:

Think about it. What's the main point of a usability laboratory? To simulate a normal office environment, and find out how real users are experiencing your site. If that's the goal, then here's a modest suggestion: why not use your participants' actual office environments? All you need is a telephone. Ask them to answer a few questions for you while they navigate the site on their usual computer. Or drive over to their houses to see what your carefully crafted site looks like on their out-of-date home computers. Chances are you will find some very interesting flaws in your current design.

Drawbacks to Testing by Telephone

Interviewing people by telephone does mean that you'll miss some important non-verbal information, such as facial expressions and gestures. These non-verbal cues let you know when a participant is feeling tired, frustrated, or confused. It's also more difficult to collect objective metrics, such as the time a task requires, the error rate, which keystrokes were used for navigation, etc.

What you can learn a great deal about, though, is the user's experience:

• What are their expectations, based on the design? Ask the participants what their first impression is, and how the site makes them feel. What is the first thing they notice? Who do they think this site is intended for?
• What would they expect to be able to do on this site? (Tip: this can be a great help in formulating tasks for more traditional usability tests later.) Follow up by asking the participant, "What would be your first step to answer the question you have in mind?" Did your design put them on the right track?
• What does the language mean to them? You can take an open-ended approach: "Are any of the labels confusing to you? Which ones?" But also draw users out by asking specific questions like "What would you expect to find if you clicked on…" The answers might surprise you.

Tips for Remote Testing

Remote testing relies solely on what participants say, so you will have to encourage them to think aloud as they perform tasks. Some people are more vocal than others, and you will have to use questions to probe for more information from users who are less forthcoming. Ask participants if a feature was easy to use or confusing, and remind them to express their likes and dislikes as they move through the site.

When soliciting feedback from participants, it's important to phrase your questions neutrally. It can be helpful to offer two extreme positions -- "Did you find this feature difficult to use, or easy? -- because it encourages the user to offer an honest reply. And some participants will want to resist the extremes and qualify their response, which provides you with even more information. For example, "It was pretty easy, but I couldn't find…."

What About Email?

Wouldn't email contact with users be more focused and more useful? That depends.

Although email contact can indeed be helpful, it's important to keep the burden on participants very low. Rather than having them write up responses to your queries, you want to hear them think aloud while they maneuver their way through the site. Even better, of course, is to watch their struggle. This approach allows you to capture the initial reaction -- often, "Huh?" -- rather than the tempered-by-hindsight responses that you're more likely to receive by email.

Variety is the Spice of Usability Testing

"OK," you might be thinking. "I know it's cheaper not to build a usability lab from the ground up. But I want to do this job right." Don't worry! You will. Cheap doesn't have to mean sloppy. When it comes to usability testing, it can actually be an advantage.

Make a virtue of your budget shortfall. A telephone and a bus pass will allow you to test your site on a wider variety of computer setups. After all, one of the main usability challenges you will face in Web design is that not everyone sees your site the same way. An older woman using Internet Explorer, with her monitor set to 800 x 600 and her browser's text size preference set at "largest," may have a very different impression of your site than a young college student using Safari as his browser, with his screen resolution set to 1400 x 1050.

Some people won't be able to see your Website at all, but may still wish to use it. Blind people might be listening to your site, using screen readers that parse the page and speak it aloud in a synthesized voice. People with limited vision might be using magnification software that allows them to see only a small portion of your site at any time. Users with mobility problems might be trying to access your site without using a mouse.

It is worthwhile to consider these populations when testing your site. Gerber and Kirchner report that, "Previous usability and user's experience research has, almost without exception, been conducted with an able-bodied population, or at least with people for whom the use of assistive technologies was not noticeable." But these are growing populations. A Forrester survey commissioned by Microsoft found that 40% of computer users are likely to benefit from the use of accessible technology due to mild difficulties or impairments. That's about 51.6 million people.

You could try to create a usability lab equipped with a variety of assistive technology, but the software changes all the time. And what you want to avoid, above all, is testing users on their knowledge of your lab's assistive technology, rather than testing the site. The best way to keep the focus firmly on your Website is to take assistive technology out of the picture -- not by ignoring it, but by allowing users to test your site from their own familiar setups.

Step Outside the Lab

Even if you're not making a special effort to include disabled participants, it makes good sense to test your site in users' "real-life" settings. Gerber and Kirchner offer several reasons for conducting usability tests outside a lab environment:

1. It shows how your Website works on the computer hardware and software that people really use -- often older and slower than the setups available in a lab setting.


2. It takes place in environmental conditions that closely approximate actual usage -- inadequate light, background noise, clutter in the vicinity, family distractions, etc.


3. Users are more at ease than in the lab setting. They also tend to be more revealing about their actual behavior in seeking information, particularly if the Website's topic involves personal matters, such as health.


4. Finally, this technique clearly reduces "respondent burden" both in terms of participants not having to travel, and their comfort during the testing. This advantage makes it possible for you to widen your scope, including people who would not agree to testing in a lab because of the travel and inconvenience.

Don't Let Ambition Stop You Dead in Your Tracks!

The final, most compelling reason to just dive into usability testing without a proper usability lab? "Perfect" is the enemy of "done." The more elaborate the setup requirements are, the less likely it is you will ever get the usability testing started, much less done. So get on the telephone, or hop on the bus, and start observing!

Further Reading:

• Gerber and Kirchner, "Conducting Usability Testing With Computer Users Who Are Blind or Visually Impaired" [1]
• Nielsen, "Usability Laboratories: A 1994 Survey" [2]
• "The Market for Accessible Technology—The Wide Range of Abilities and Its Impact on Computer Use," [3] study commissioned by Microsoft Corporation and conducted by Forrester Research, Inc., in 2003

[1] http://www.afb.org/Section.asp?SectionID=57&TopicID=167&DocumentID=1718
[2] http://www.useit.com/papers/uselabs.html
[3] http://www.microsoft.com/enable/research/phase1.aspx