A broken pod tells you why
A broken pod is one that never reaches the Running state. Kubernetes records the reason, but you have to read the right place.
The STATUS column names the kind of fault, the pod's events say what Kubernetes tried, and the pod's log says what the program said.
By the end of this lesson
Note
I can read describe and logs and name the fault of a broken pod.
Key words
| Word | What it means | Example |
|---|---|---|
| Status | the state a pod is in | Pending |
| Events | the list of what Kubernetes did with a pod | |
| Log | what the program in the pod printed | |
| CrashLoopBackOff | the program keeps stopping and is retried, with longer waits | |
| ImagePullBackOff | the image cannot be fetched, so it waits and retries | |
| Describe | the kubectl command that shows one object in full |
The cells students fill in are left blank in this preview.
The STATUS column names the kind of fault
Three words cover most broken pods. Pending means no node has taken the pod yet. ImagePullBackOff means the image could not be fetched. CrashLoopBackOff means the program starts, stops, and is tried again.
The word is a clue, not the cause. The cause is in the events or the log.
Worked example: status, meaning, where to look
| STATUS | What has happened | Look at first |
|---|---|---|
| Pending | No node has taken the pod | The events, in describe |
| ErrImagePull, ImagePullBackOff | The image was not fetched | The events, in describe |
| CrashLoopBackOff | The program started and stopped | The log |
The cells students fill in are left blank in this preview.
Make a namespace for faults
kubectl create namespace faultsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Start a program that always fails
kubectl create deployment crashy -n faults --image=busybox:1.37.0 -- sh -c 'echo cannot open orders.db; exit 2'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask for a tag that does not exist
kubectl create deployment oldtag -n faults --image=busybox:1.99.0 -- sleep 3600Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Start a pod no node suits
kubectl run stuck -n faults --image=busybox:1.37.0 --overrides='{"spec":{"nodeSelector":{"disk":"fast"}}}' -- sleep 3600Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Wait until the crashing pod is backing off
kubectl wait --for=jsonpath='{.status.containerStatuses[0].state.waiting.reason}'=CrashLoopBackOff pod -l app=crashy -n faults --timeout=120sShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the STATUS column
kubectl get pods -n faultsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Logs and events say what went wrong
kubectl logs prints what the program wrote. It is the first place to look when the program ran and stopped.
kubectl describe pod prints the pod in full, and its Events at the bottom. They are the first place to look when the program never ran. For a pod stuck in Pending, the last event says why no node took it.
Read the failing program's log
kubectl logs -n faults -l app=crashyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the last events of the stuck pod
kubectl describe pod stuck -n faults | tail -n 3Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think CrashLoopBackOff is the fault. It is only the pattern: start, stop, wait, try again. The fault is whatever made the program stop, and the log has it. Read the log before you change anything.
Fix the fault the events name
A wrong image tag is fixed by giving the Deployment a tag that exists. A pod with a node selector no node meets waits until a node does.
The next Runs give oldtag the image the lab has. Then they label the node disk=fast so stuck can be placed, and take the label away again.
Give oldtag an image that exists
kubectl set image deployment/oldtag busybox=busybox:1.37.0 -n faultsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Label the node for the pod
kubectl label nodes --all disk=fastShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Wait until stuck is ready
kubectl wait --for=condition=Ready pod/stuck -n faults --timeout=60sShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Take the label off the node again
kubectl label nodes --all disk-Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Remove the namespace
kubectl delete namespace faultsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.