A usability test shows where people get stuck
A usability test is a session where a real person tries real tasks on a product while you watch and measure. It matters because it replaces arguments about taste with evidence you can count.
The lab holds a made-up test of the teapot shop: twelve people, four tasks. You will read its numbers.
By the end of this lesson
Note
I can run a small usability test and read its success rate, times and errors.
Key words
| Word | What it means |
|---|---|
| Participant | a person who takes part in the test |
| Success rate | the share of people who finished a task |
| Median | the middle value when the numbers are in order |
| Error | a slip or wrong step a person makes during a task |
| Mean | the total of the values divided by how many there are |
The cells students fill in are left blank in this preview.
A test needs tasks, people and measures
Plan three things. First, the tasks: two to six jobs from the tasks that matter. Second, the participants: people like your users, not your team. Third, the measures: what you will count.
Say each task as a goal, not as steps: "buy the green teapot", not "press Buy, then Pay". Then stay quiet and watch.
Worked example: what each measure tells you
| Measure | How you get it | What it tells you |
|---|---|---|
| Success rate | People who finished, divided by all people | Whether the task can be done at all |
| Time on task | Seconds from start to finish, per person | How hard or slow it was |
| Errors | Slips and wrong steps you saw | Where the page misleads people |
| Notes | What people said and did | Why the numbers look as they do |
The cells students fill in are left blank in this preview.
Success rate is people who finished over everybody
The success rate is the number of people who finished a task, divided by the number who tried it, as a percentage. Ten of twelve is 83 per cent.
Count a person as finished only if they reached the goal without your help. A person who gave up is a failure of the page, not of the person.
Read the success rate and median time
docker exec jupyter python3 steps/tasks.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the lab prints for the four tasks
| Task | People who finished | Success rate | Median seconds |
|---|---|---|---|
| find_teapot | 12 of 12 | 100% | 27.5 |
| add_to_basket | 10 of 12 | 83% | 36.5 |
| fill_form | 11 of 12 | 92% | 97.5 |
| fix_error | 6 of 12 | 50% | 110.0 |
The cells students fill in are left blank in this preview.
Use the median for time, not the mean
The median is the middle time when you put the times in order. With an even count it is the mean of the two middle ones.
One very slow person drags the mean upwards but barely moves the median. Times 40, 45, 50, 55 and 300 have a mean of 98 and a median of 50. The median says more about a usual person.
Add up the errors for each task
docker exec jupyter python3 steps/errors.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Errors say where the page misleads people
The script adds up the errors for each task. It prints the totals from most to fewest, then the name of the task with the most.
fix_error has 34, fill_form 18, add_to_basket 11 and find_teapot 7. The two numbers agree. The task that fewest people finished is also the task with the most errors. Together they point at the page that needs work first.
A common mistake
Warning
Many people think a test needs hundreds of people. A small test finds the big faults, and twelve people are plenty to see a task failing. But twelve people cannot give an exact figure, so say "about half finished" and not "exactly 50.0 per cent".
Going further
Plan a test of your college's library page with four tasks and five classmates. Write each task as a goal and decide what you will count.
Run it, then use your function to find the success rate and median time of each task. Which task do you fix first?