Professional Website Usability

Scoring and Conclusions

We'll use the five parts of the study described in Part 3 of this series and review the results based on the scoring system we used for each part. If you haven't already done so, go ahead and create a document in your word processing system now so you can begin to document your interpretations and conclusions. Then, you can build the other sections of the report around this most important section..

Part 1 -- General Survey

This survey collected general information about your participants and may have included text or check box responses. As you read through the responses to each of the questions, notice not only what was said but how many said it to form a conclusion.

For example: You might report "20 people participated in the usability study and 15 stated that they use the web every day and 5 use it once a week. From this, I can conclude that the majority of the participants are regular web users, familiar with how the web and web pages work and that they are well-versed in the capabilities and functions of web-related navigation and materials." If these 15 were priority customers or an audience that you are purposefully driving to your website, you can make a more educated assumption about their skill level and use it to improve the site.

Look for logical groupings of feedback and report them accordingly. For example: You might report "20 people participated in the study, and all 20 stated that they visit our website to retrieve operating forms. From this, I can conclude that we should do an extra-special job of providing navigation and labeling that leads our users quickly to forms." If only 10 of the 20 came to your site for forms, your conclusion may be different depending on what the other 10 came for.

Part 2 -- Treasure Hunt

In the Treasure Hunt, you provided very specific direction and instructions and the participant carried them out. They may have been asked to find a specific piece of information, use an interactive functionality, download an application or document, etc. (Note that I could have used the term, "Wild Goose Chase," but I chose "Treasure Hunt" to make it a little more appealing!) You also may have sent them after a piece of information that you felt was particularly buried in the site or that you had a difficult time naming or placing on the site. You'll have to read text answers, but if you used the scoring system a) I completed this task, b) I'm not sure if I completed this task, c) I did not complete this task or d) I gave up, you can count how many participants responded according to these responses.

For example: Of the 20 participants, 7 completed the task, 4 weren't sure if they completed the task, 6 did not complete this task and 3 gave up. What can you conclude from these numbers? The best way is to see if they convert to the intuitive scoring system discussed in Part 3. While you won't want to go into all the details of how the scoring system works, you will want to let the reader know that you've determined an equally balanced low, medium and high intuitive range and how these results relate to those ranges. For example, you might report, "The Treasure Hunt produces results that relate to how intuitive a particular activity was. There were a total of 20 participants, so a range of 0-20 will be used to help you understand these results. The ranges for the Treasure Hunt for intuitiveness have been determined as follows: 0-7=low intuitive, 7-14=medium intuitive, 15-20=high intuitive. You can see that the more people complete the task, the more intuitive we've apparently built that specific component (we've been successful and our assumptions paid off!). Likewise, fewer people completing the task would be evidence that more work is needed to increase intuition for that item."

By reading the text responses and manually counting for the intuitive score, you can give the reader a good picture of the user experience while finding information and performing activities on your site.

Part 3 -- Anticipation/Intuition

Web developers make many assumptions about labeling and navigation on websites, so is section will help clarify and validate that users can actually use those paths to reach their desired destination. You may have text responses and a scoring system if you provided a statement and asked the participant to circle their range of agreement and disagreement with the statement. Of course, being able to produce a number helps us again as we use it for our scoring and conclusions.

For example: By counting all the 1, 2, 3, 4, and 5 scores (assuming it's a five-point scale you used), you can again see how many participants found a certain statement to be true (3-agree somewhat, 4-agree, 5-strongly agree) or through the range to false (2-disagree, 1-disagree strongly). Totaling all these and dividing by the number of participants can produce an average score. With this score you can form conclusions or prioritize which work you might want to perform first. In the example used in the previous article, you could total up all the individual scores and multiply by the number of participants to produce the average. With 10 participants, if 5 participants strongly agree (5x5=25), 3 participants agree somewhat (3x3=9) and 2 participants strongly disagree (2x1=2), add up the totals (25+9+2=36) and divide by 10 (your number of participants). 36 divided by 10 = 3.6. From this, we can conclude that 3.6 against the 1-5 scoring would mean that more work needs to be done, but it's in the middle range. If you asked the participants to provide more information on items they scored 3 or lower, you can zoom on the areas in the website that need work. Closer to the 1 would mean dissatisfaction and that more work is required; closer to the 5 means that the participant was more satisfied and less work is required. It depends, of course, how you structure your statements as to how you use this type of scoring.

Use a conversational tone in your report with language that's easy for the reader to understand. You might report, "Results were averaged among all participants and produced a 3.6 total for this item. With the scoring system we've used, an average closer to a 1 would mean that we have much work left to do; closer to a 5 would mean less. Regardless, several improvements have been recommended and they are as follows..."

Part 4 -- Terms and Language Use

In this section you probably asked specific questions to see if specific or generic terms made sense and were understood and if language use was appropriate. The feedback provided here will likely be text or check boxes, so you'll have to read through each participants' feedback. As you uncover trends, note how many people contributed to that trend. It helps justify why the change is needed if you have more people who made similar recommendations.

For example: If there were 10 participants, and 3 noted that they preferred the word 'car' to 'automobile,' you could decide if it was meaningful that 33% of your participants asked for the same change. If all 10 found some glaring item that they asked be changed, you'd have 100% of your participants asking for it, and you'd likely make it. Again, as these trends appear, count them -- it really helps justify changes if priorities are tight or resources are running thin. In your conclusion, decide if you want to report that 3 participants felt a certain way or if 33% felt a certain way. There may be cases where you believe reporting 33% is more compelling than a measly old 3.

Part 5 -- Look and Feel

If you asked participants to comment on the look of the site, you probably asked about graphic usage, colors, font sizes, readability of text, foreground versus background compatibility, use of a metaphor, ability to move through the site, etc. You can use the 1-5 scoring system again here as well. (1-disagree strongly, 2-disagree, 3-agree somewhat, 4-agree or 5-strongly agree). How you phrase the statements will determine how this scoring will work for you.

For example: If the statement was, "I like the bright purple background with yellow checkered squares." Count how many participants chose which responses (1-5) and do the math as you did previously. With 10 participants, if 5 participants strongly agree (5x5=25), 3 participants agree somewhat (3x3=9) and 2 participants strongly disagree (2x1=2), add up the totals (25+9+2=36) and divide by 10 (your number of participants). 36 divided by 10 = 3.6. From this, we can conclude that 3.6 against the 1-5 scoring would mean that some work needs to be done, but it's in the middle range. Closer to the 1 would mean dissatisfaction and that more work is required; closer to the 5 means that the participant was more satisfied and less work is required. You'll be able to produce an average and discuss the areas needing work if you asked for more detail on items scored 3 and lower. You might report "The bright purple background with yellow checkered squares was met with mixed emotion. An average of 3.6 was produced by this item which may mean that it's not an ideal color/shape combination. Participants who scored this item 3 or lower provided the following ideas for improving the color and shapes of the background..."

What's Your Approach?

In closing, each of us has a unique way that we approach our projects and usability is no different. It's subjective, driven by the experiences or lack of experience of those involved in designing the strategy, plan, evaluation tool and interpreting the results, etc. So, be creative and do some research on your own until you find a system that works for you. There are certainly world-renowned experts, books, websites and associations on the subject, so you have many resources to explore as you find your unique approach. Just search for the keyword 'usability' on the web and you'll see what I mean.

Through experimentation and research, you'll soon become a usability expert within your own organization. This field adds incredible value to our business processes, products and systems, and I believe that management has not yet fully understood and embraced this fact. If usability is a pay now or pay later proposition, why not do it up-front, before coding has begun, while a website is in the planning or development stage, to validate the designer's assumptions and make sure user gets what they need? Failing to do so is an error that will have negative repercussions to customers, your corporate brand, your system's credibility and your staff-it could be a fatal error and not recoverable. Stress to your management and the management in your information systems areas that usability is a key step in the development process and that it must remain in the plan no matter how short timeframes are or how expensive it is perceived to be. The usability process can be shortened, fewer participants can be sought, internal staff can be used to administer the test and other means can be taken to reduce the cost and still produce meaningful results.

Thank you for the opportunity to share my approaches with you! I hope you've found this series helpful and that you'll experiment with some of my strategies the next time you have a website usability opportunity.