Basic Usability Testing at Home - notes from the workshop at Akademy 2013

In the weeks before Akademy 2013 I convinced KDE Usability Guru Björn Balazs (from User Prompt) to lend his experience to a usability workshop I wanted to organize at the conference. The goal was to teach developers how to do 'basic usability testing at home' by guiding users through their application and watching the process. To help developers who didn't make it (and those who did but can use a reminder) I hereby share a description of the process and some tips and notes.

User Application Testing

The goal of this exercise is to get input from a user about the applications being tested. The test works by putting the user in front of the application, giving them a task and letting them execute it, guided by the developer asking questions.

Process

The process works as follows: The user is put in front of the opening screen of the application/tool. The developer or usability expert guiding the process gives them a goal, something like 'play a song from Cold Play' or 'connect to the wireless network'. You can use this nice guide by the Canonical usability team on picking tasks for the users to execute.

Then the following protocol is executed by the person guiding the process, asking the user:

1. What is your first impression?
2. What do you think you can do here?
3. What would you do to reach your goal?
4. What do you expect to happen when you do it?
5. Please do it... (let user execute action)
6. Did it do what you expected?
7. if not: was it better or worse?
8. If task is not completed yet, go back to 1 

Important:

• Make clear that the user is ONLY to execute a SINGLE action in step 5. This means that most of the time, you are checking the expectations and ideas of the user instead of watching him/her clicking around.
• As guide you're not supposed to give any hints as to what button the user should click or what he/she should do, other than perhaps remind him/her of the end goal he/she has been given
• Despite the above point, there is of course no reason to let the user helplessly waste his/her time. If the user get stuck, help them to get unstuck. 
• Try to closely stick to the protocol, not skipping questions. Of course, if a user has already answered question two in his answer on question one, there is no need to ask that question again...

If the task is finished, asked what the user thought of it all, discuss improvements etc.



Note that this kind of testing is very good to find grave issues in applications; it is not very good at fine tuning the interaction with your application. You also have to be a bit wary of the feedback from a single user: repeat the process at least once with somebody else to make sure that what is an issue for one person is really a problem. Ideally, run the test a few times with various people to increase the reliability of the feedback.

Extra tip: it is very valuable to record the sessions. Often you see things later on that you were not aware off during the session..

Results

In practice, it can be a bit hard to stay disciplined, not giving hints or telling the user what to do... And to stick to the protocol instead of just quickly letting the user click to the end. Try to keep to the structure anyway, it works best.

This process works incredibly well at showing which things are unexpectedly non-obvious for users. For your enjoyment, here is a youtube video of the Network Manager session with myself as subject:
Part 1:

Part 2:


I think the results speak for themselves and you can probably imagine the frustration on the side of the developers.

I've put the notes of the workshop also on the wiki.

Thanks to User Prompt for supporting the travel and support of Björn and Heiko Tietze as they were running the actual workshop, I was just there as participant ;-)