Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Resource limits with cgroups v2

After this lesson: I can set memory, CPU and process limits on a program with cgroups.

60 min lesson5 commandsSandboxing lab: run untrusted code inside namespaces, cgroups, seccomp and gVisor on one private Ubuntu VM

A limit the program cannot argue with

A control group, or cgroup, is a set of processes that the Linux kernel counts and limits together. It limits their memory, their processor time and how many processes they start. Version 2 is the one current systems use.

That matters because untrusted code that grabs all the memory takes the machine down for everyone. A cgroup stops it, and today you watch the kernel do it.

By the end of this lesson

Note

I can set memory, CPU and process limits on a program with cgroups.

Key words

WordWhat it meansExample
Control groupprocesses the kernel counts and limits together
memory.maxthe most memory the group may hold, in bytes67108864
cpu.maxprocessor time allowed in each period20000 100000
pids.maxthe most processes the group may hold20
OOM killthe kernel killing a process that is out of memory
Fork bomba program that starts processes without end

The cells students fill in are left blank in this preview.

A cgroup is a folder of files

Every cgroup is a folder under /sys/fs/cgroup. Making a folder makes a group. Writing a number into one of its files sets a limit, and writing a process number into cgroup.procs moves that process in. Every child it starts joins the same group.

The lab's helper, sbx limit, does exactly that and nothing more. With --show it prints each file it writes. Read /usr/local/bin/sbx to check.

Go to the lab folder

Command
cd ~/lab

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

See the files sbx limit writes

Command
sudo sbx limit --memory 64M --cpu 20 --pids 20 --show -- true

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: three limits as files

OptionFileWhat sbx wroteWhy that number
--memory 64Mmemory.max6710886464 times 1024 times 1024 bytes
--cpu 20cpu.max20000 10000020 ms of every 100 ms
--pids 20pids.max20At most 20 processes and threads

The cells students fill in are left blank in this preview.

Over the memory limit, the kernel kills

A program inside a group can ask for more memory than memory.max allows. The kernel first tries to reclaim memory, then kills a process in the group. The kill is counted in the file memory.events, on the line oom_kill.

The agent below asks for 200 MB, 10 MB at a time, inside a 64 MB limit. Predict what it prints before you press Run.

Ask for 200 MB inside 64 MB

Command
sudo sbx limit --memory 64M -- python3 agent/agent.py memory 200

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Ask for 200 MB with no limit

Command
python3 agent/agent.py memory 200

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people think a program over its memory limit gets an error it can handle. On Linux it usually does not. The kernel kills it with a signal it cannot catch, so the program never prints its last line. Exit status 137 is that kill.

A process limit stops a fork bomb

A fork bomb starts processes until the machine has no room for one more, and then nothing can start, not even your shell. pids.max counts every process and thread in the group.

When the group is full, the next fork fails with Resource temporarily unavailable, and pids.events counts it on its max line. The program carries on with what it has.

Start 30 processes inside 20

Command
sudo sbx limit --pids 20 -- python3 agent/agent.py spawn 30

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Try it: read a cpu.max line

Code: python
# cpu.max holds two numbers: how much time the group may use in each period,
# and the period, both in microseconds.
period = 100000
for quota in [20000, 50000, 200000]:
    share = quota * 100 // period
    print(quota, period, "is", share, "percent of one processor")
# 200000 is more than one period: what does that mean on a machine with two processors?

About this lab and its source: Sandboxing lab: run untrusted code inside namespaces, cgroups, seccomp and gVisor on one private Ubuntu VM

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.