Usability Testing Basics

A shop has the teapots people want, a fair price and a good name, and still they leave at the checkout. Nobody is rude about it. They just could not work out what to press. A page that is hard to use loses people quietly, so the only way to know is to watch some of them try.

• Usability: how easily and how well people can reach their goal with a page.
• Accessibility: whether people with poor sight, a screen reader, a keyboard only or a small phone can use it too.
• Contrast ratio: how different the text colour is from the colour behind it, from 1 (the same) to 21 (black on white).
• Alt text: words that stand in for a picture when it cannot be seen.
• Success rate: the share of people who finished a task.
• Median: the middle value when the times are put in order.

ls lists the work folder: the shop, the two tables of results, the notes, the starter notebook and the scripts. cat notes.md prints what the observers wrote while twelve people tried the shop. Read them against the Preview: grey text hard to read, a Buy button pressed four times, a person asking what Error 7 is. Notes are not numbers yet, but they tell you where to look.

The script reads the colours in shop/style.css and works out the contrast ratio of each text and background pair. The usual contrast guideline asks for at least 4.5 for ordinary text. The body text, grey on white, scores 2.85, which is too low. The headline, dark grey on white, scores 15.91, which is plenty. Three more pairs fail, among them the link colour at 1.89.

The audit reads the two pages and the style sheet and counts four common problems. On the home page 3 of 4 pictures have no alt text and 5 of 6 links say only things like "click here" or "read more", so a person listening to the links hears nothing about where each goes. On the checkout page 3 of 4 boxes have no label, and the one error message is only a code. In the style sheet one rule removes the focus outline, the line that shows a keyboard user where they are.

Twelve people each tried four tasks. pandas groups the rows by task and gives the success rate and the median time. Finding a teapot worked for everybody, in a median of 27.5 seconds. Paying for it was the hard part: only 50 per cent got through the last task, in a median of 110 seconds. The median is used because one very slow person would drag a mean upwards.

After the tasks each person answered ten questions from 1 to 5. The usual way of scoring turns the ten answers into one number from 0 to 100 for each person. The twelve scores run from 32.5 to 80.0, with a mean of 55.0, and scores around 68 are usually read as average, so people found this shop harder than most. The second script counts mistakes: the last task has 34, more than any other.

Pick one problem, for example the focus outline, in JupyterLab open shop/style.css, change it, save, and refresh the Preview. Then run the script below. It fixes nothing itself: it runs the audit and the contrast check again, so the numbers change only if your edit did. A fix you cannot see in the numbers or in the Preview is a fix you cannot be sure of.

Open JupyterLab from the bar and double-click start.ipynb. It loads the two tables and works out the same success rates, times and mistakes, one cell at a time. Click a cell and press Shift and Enter to run it. The last cell is where your own question goes: does the slowest person also give the lowest questionnaire score?