One shop app, many machines
A distributed system is a group of computers that work together and look like one service to the people using it. Every large shop app, bank app and video site is one.
You need to know why they exist and what goes wrong in them. A program that works on one machine can fail in surprising ways when it runs on ten.
By the end of this lesson
Note
I can say why systems run on many machines and what goes wrong.
Key words
| Word | What it means | Example |
|---|---|---|
| Distributed system | computers that work together as one service | |
| Node | one machine in the system | |
| Latency | the time a request takes to get its answer | 80 ms |
| Availability | the share of time a service works | |
| Partial failure | some parts break while the rest keep working | |
| Scale out | add more machines instead of a bigger one |
The cells students fill in are left blank in this preview.
Three reasons to use more than one machine
A single machine has limits, and a distributed system gets round three of them.
Speed: many machines answer many requests at the same time. Size: data too big for one disk is shared out. Failure: if one machine breaks, another carries on.
Adding machines is called scaling out. The other way, buying one bigger machine, soon gets very expensive and still has one weak point.
Worked example: machines for a busy hour
| Machines | Requests per second they can answer | Busiest hour needs 250 |
|---|---|---|
| 1 | 100 | Too slow, requests queue up |
| 2 | 200 | Still too slow |
| 3 | 300 | Enough, with a little spare |
The cells students fill in are left blank in this preview.
Counting the machines you need
# How many machines does a busy hour need?
peak = 250 # requests per second at the busiest time
per_machine = 100 # what one machine can answer each second
machines = 0
capacity = 0
while capacity < peak:
machines = machines + 1
capacity = capacity + per_machine
print(machines, "machines give", capacity, "requests per second")
# Change peak to 410 and run again.
Every trip over the network costs time
On one machine a function call takes a tiny fraction of a millisecond. Between machines the request must travel, and the latency grows with the distance.
The figures in the table are typical, and real ones vary. The lesson is the size of the gap. A page that needs five trips waits for all five.
Typical time for one trip
| Where the other machine is | About how long one trip takes |
|---|---|
| Same machine, in memory | less than 0.001 ms |
| Same data centre | 0.5 ms |
| Another city in the same country | 10 ms |
| Across an ocean | 80 ms |
The cells students fill in are left blank in this preview.
Make a tiny client called v
v() ( printf '%s\r\n' "$*" | nc -w 1 valkey 6379 )Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask the store if it is alive
v PINGShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Send a word to the store and back
v ECHO helloShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Adding up the trips
# One page needs several trips over the network. Times are in milliseconds.
trips = [2, 2, 2, 80, 80]
total = 0
for ms in trips:
total = total + ms
print("The page waits", total, "ms")
# Change the last 80 to 2 (the same data centre) and run again.
One part of the system can fail alone
One computer is either working or not. A distributed system is stranger: a machine can stop, a cable can break or a message can be lost while the others carry on. This is partial failure.
The worst part is that you often cannot tell. If a machine does not answer, it may be broken, busy or just slow. Designs must expect this.
Needing both is worse than needing one
# One reading a minute for ten minutes: is each machine up or down?
a = ["up", "up", "down", "up", "up", "up", "up", "down", "up", "up"]
b = ["up", "up", "up", "up", "down", "up", "up", "up", "up", "up"]
both = 0
either = 0
for minute in range(10):
if a[minute] == "up" and b[minute] == "up":
both = both + 1
if a[minute] == "up" or b[minute] == "up":
either = either + 1
print("Both up:", both, "minutes of 10")
print("At least one up:", either, "minutes of 10")
# Change a[2] to "up" and run again.
Check whether the lab store has any copy
v INFO replication | grep -E "role|connected_slaves"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think more machines always mean fewer failures. Each machine can break, and the network between them can too. More machines help only when the design lets the others carry on. If every machine is needed, more machines make failure more likely.
Going further: a lone store
The lab store prints connected_slaves:0, so it has no copy. Write one sentence on what a customer would see if this one machine stopped, and what one extra machine would change.
The next lesson builds exactly that extra machine, a replica.