Ticket day
Tickets for a big concert go on sale at 9:00. Fifty thousand fans open the website at the same time.
At 9:01 the site stops. Every button worked when the team tested it. So what went wrong?
The team tested what the site does. They did not test how well it works when many people use it.
Key words
| Word | What it means | Example |
|---|---|---|
| Non-functional testing | Tests that check how well the software works, not what it does. | Is it fast enough? |
| Availability | How much of the time the software is up and working. | Up for 23 hours out of 24 |
| Compatibility | Does it work on different devices and different software? | Phone and laptop |
| Configuration | The settings and set-up the software runs with. | Language, memory, time zone |
| Load | How much work the software has at one time. | 50,000 users at once |
| Response time | How long the user waits for an answer. | 2 seconds |
| Average | Add all the numbers, then divide by how many numbers there are. | 1, 2 and 6 add up to 9. 9 divided by 3 is 3. |
| Crash | The software stops working suddenly. | A blank screen |
| Server | A computer that runs a website or app for many users. | The ticket website computer |
The cells students fill in are left blank in this preview.
Four non-functional tests
| Test | The question it asks | Example |
|---|---|---|
| Availability testing | Is it up and working when users need it? | Check the site every minute for a week |
| Compatibility testing | Does it work with other devices, systems and software? | Open the app on an old phone and a new one |
| Configuration testing | Does it work with each set of settings? | Try it in English, then in Japanese |
| Load testing | Does it work when many users use it at once? | Send 5,000 fake users to the site |
The cells students fill in are left blank in this preview.
Availability: how often is it up?
Users want the service when they need it. Availability is the time it is up, compared with the whole time.
A day has 1440 minutes. If a site is down for 14 minutes, it is up for 1426 minutes. That is very good. A site that is down for 2 hours is not.
Worked example: one day
| Step | What we do | Number |
|---|---|---|
| 1 | Minutes in one day: 24 hours times 60 | 1440 |
| 2 | Minutes the site was down | 14 |
| 3 | Minutes the site was up: 1440 minus 14 | 1426 |
| 4 | We say | Up for 1426 minutes out of 1440 |
The cells students fill in are left blank in this preview.
Load testing
A load test sends many fake users to the software. We start with a few users and add more and more.
We watch the response time. We find the point where it gets slow, or where it crashes. Now the team knows how many users the site can handle.
Try it: response times
# Time in seconds for five users
times = [1, 2, 2, 9, 1]
print("Slowest:", max(times))
print("Total:", sum(times))
print("Average:", sum(times) / len(times))
# Change 9 to 2 and run again. What happens to the average?
Average and slowest
data-callouts.tip.label
The average hides a very slow user. In the Try it block the average is 3, but one user waited 9 seconds. A good load test looks at the slowest time too.
Compatibility and configuration
Compatibility is about other things around your software: the phone, the system, the browser.
Configuration is about the settings inside it: the language, the time zone, the amount of memory. A bug can appear with one setting and not with another. So we test the settings that people really use.