"The internet is down"
Network troubleshooting means finding where a connection fails by testing one step at a time, from the machine outward. It turns a vague complaint into one fault you can fix.
On Monday morning a manager rings the help desk. The internet is down. Ten minutes later it turns out only one printer was offline.
A good engineer does not guess. She tests each step in order and stops at the first one that fails.
By the end of this lesson
Note
I can find where a connection fails by testing one step at a time.
Key words
| Word | What it means |
|---|---|
| Symptom | what the user sees go wrong |
| Default route | the way a machine sends traffic when no other route matches |
| Unreachable | no route exists to that network |
| Timeout | no answer came back in time |
| Refused | the machine answered and said no |
| Traceroute | a tool that lists each router on the way |
The cells students fill in are left blank in this preview.
Test one step at a time
A connection needs several things in a row: an address, a way to the gateway, a route, a path, a name and an open port. A fault at one step hides everything after it.
So test in that order. The first step that fails is where to look, and nothing after it is worth testing yet.
Worked example: the six steps
| Step | The question | A command |
|---|---|---|
| 1 Address | Does this machine have an address? | ip -br -4 addr |
| 2 Gateway | Can it reach its own gateway? | ping the gateway |
| 3 Route | Does it know a way to the target? | ip route get |
| 4 Path | Where do the packets stop? | traceroute -n |
| 5 Name | Does the name turn into an address? | dig, getent hosts |
| 6 Port | Does something answer on the port? | nc -zv, curl |
The cells students fill in are left blank in this preview.
Step 1: read the LAN machine's address
sudo ip netns exec lan ip -br -4 addrShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Step 2: ping the gateway
sudo ip netns exec lan ping -c 1 -W 1 10.0.0.1Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Step 3: ask for the route
sudo ip netns exec lan ip route get 203.0.113.2Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Break one step, then find it
Now you remove the LAN machine's default route yourself. Then you test the steps in order, and see which one fails first.
The last Run of this part puts the route back, so do all of it.
Remove the default route
sudo ip netns exec lan ip route del defaultShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Step 2 again: the gateway still answers
sudo ip netns exec lan ping -c 1 -W 1 10.0.0.1 | tail -n 2Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Step 3 again: ask for the outside server
sudo ip netns exec lan ip route get 203.0.113.2Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the web request says now
sudo ip netns exec lan curl --noproxy '*' -sS -m 3 http://203.0.113.2:8000/; echo "curl exit code $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Put the default route back
sudo ip netns exec lan ip route add default via 10.0.0.1Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Each failure word says something different
Unreachable comes at once and means there is no route. Refused comes fast and means the machine answered and said no.
A timeout comes slowly and means silence: a drop, or a reply that cannot find its way back. The next three Runs show the last two.
A refusal: the machine answered
sudo ip netns exec gw nc -zv -w 2 10.0.0.2 9999; echo "nc exit code $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A timeout: nobody answers the LAN
sudo ip netns exec lan curl --noproxy '*' -sS -m 3 http://203.0.113.2:8000/; echo "curl exit code $?"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Step 4: see where the packets stop
sudo ip netns exec lan traceroute -n -q 1 -w 1 -m 3 203.0.113.2Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think that if ping works the connection is fine. Ping proves one step only. The web request still needs a route back, a name and an open port. Here the gateway answers ping, and the web request still times out.
Worked example: from symptom to next step
| What you see | What it means | Where to look next |
|---|---|---|
| Network is unreachable, at once | No route to that network | The routing table and the default route |
| Could not resolve host | The name gave no address | The hosts file and the DNS server |
| Connection refused, fast | The machine is up and nothing listens | The program, its port and a reject rule |
| Timed out, slowly | No answer came back at all | A drop rule, or the way back |
The cells students fill in are left blank in this preview.
Going further: your own order
Write down the six steps from this lesson in your own words, with one command for each.
Then pick a fault you have met, at work or at home, and say which step would have found it first. Keep the page: it is the start of your own checklist.