study › Docker › cgroup Memory Limits

cgroup Memory Limits

--memory=50m is not advice to your program — it's a number in a kernel file, enforced with SIGKILL. Watch a 50 MB container allocate 90 MB before dying, docker stats report 3% while the kernel says 85%, and a dd holding one megabyte get OOM-killed.

Concept

docker run --memory=50m does not tell your program anything. It writes a number into a file — /sys/fs/cgroup/memory.max — and from that moment the Linux kernel accounts every page the container touches against it. When the container can't stay under the number, the kernel does not raise an exception, call a handler, or return NULL from malloc. It sends SIGKILL. Your process gets no turn.

This exercise makes the whole chain visible: the limit as a file the container can read, the kernel's live counter climbing toward it in docker stats, the kill, and the kernel's own tally of what it did. Along the way three things will probably not match your prediction — a container limited to 50 MB will allocate 90 MB before dying, docker stats will report 3% while the kernel says 85%, and a dd whose own resident heap is one megabyte will get OOM-killed.

Provenance

Linux kernel documentation, Documentation/admin-guide/cgroup-v2.rst — "Controllers → Memory → Memory Interface Files" (rendered). The claim under test is that section's one-sentence definition of memory.max:

Memory usage hard limit. This is the main mechanism to limit memory usage of a cgroup. If a cgroup's memory usage reaches this limit and can't be reduced, the OOM killer is invoked in the cgroup.

and, for contrast, its definition of memory.high:

Memory usage throttle limit. If a cgroup's usage goes over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure.

The two memory.events counters that let you tell those apart, from the same section:

max — The number of times the cgroup's memory usage was about to go over the max boundary. If direct reclaim fails to bring it down, the cgroup goes to OOM state.

oom_kill — The number of processes belonging to this cgroup killed by any kind of OOM killer.

Read max carefully: it counts near misses, not kills. Parts 4 and 5 turn entirely on that distinction, and on memory.current's definition — "the total amount of memory currently being used by the cgroup and its descendants", which is not the same thing as the memory its processes allocated.

The swap arithmetic in Part 2 comes from Docker's "Runtime options with Memory, CPUs, and GPUs" → --memory-swap details (docs):

If --memory-swap is unset, and --memory is set, the container can use as much swap as the --memory setting, if the host container has swap memory configured. […] If --memory-swap is set to the same value as --memory, and --memory is set to a positive integer, the container doesn't have access to swap.

cgroups(7) describes the hierarchy and the delegation rules, not these files. It's the right man page for what a cgroup is, the wrong one for what memory.max does.

Prerequisites

Two ideas carry almost all of the result; the other two are background.

cgroup v2 memory accounting — load-bearing. memory.current is a charge counter over pages, not a sum of process RSS. Page cache, kernel slab, socket buffers, and tmpfs are all charged to whichever cgroup first faulted them in, and there is no separate budget for "cache". Every surprise in Part 5 falls out of this one rule. → kernel docs: "Memory Ownership"
Page reclaim, clean vs. dirty — load-bearing. A clean file-backed page can be dropped instantly; a dirty one must be written back first, and the kernel has to wait. This is the entire difference between step 12 (survives) and step 13 (killed). Anonymous pages have no backing file, so reclaiming them means swapping — which is why Part 2 and Part 3 differ. → Mel Gorman, Understanding the Linux Virtual Memory Manager, ch. 10, "Page Frame Reclamation"
SIGKILL is uncatchable. Used in Part 3 to explain why there is no traceback. SIGKILL and SIGSTOP are the two signals that cannot be caught, blocked, or ignored. → signal(7)
Exit status 128 + N. Used everywhere 137 appears. A shell reports a signal-terminated child as 128 + signal number; SIGKILL is 9. → Bash reference, "Exit Status"

Environment

Host: macOS 26.5.2 (build 25F84), Apple Silicon (arm64)
Docker: Docker Desktop, Engine 29.6.2 — Linux VM kernel 6.12.76-linuxkit, 16 GiB allocated to the VM
cgroup: v2, driver cgroupfs, cgroupns enabled
VM swap: 2048 MB configured. Part 2 depends on this existing; on a VM with swap disabled the container in Part 2 dies at ~45 MB like Part 3's.
Alpine image: alpine:3.20 — Alpine 3.20.10, digest sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
Python image: python:3-alpine — Python 3.14.6 on Alpine 3.24.1, digest sha256:26730869004e2b9c4b9ad09cab8625e81d256d1ce97e72df5520e806b1709f92
Captured: 2026-07-26

Mental model: five files and one signal

Everything below is cgroup v2 (/sys/fs/cgroup/memory.*). If you find memory.limit_in_bytes on your machine you're on v1 and the file names differ; check with docker info | grep -i Cgroup.

File	Docker flag	What the kernel does at this threshold	Reach for it when…
memory.max	`--memory` / `-m`	Hard wall. On the allocation that would cross it, the kernel first tries to reclaim (evict page cache, swap out anonymous pages). If reclaim can't free enough, it invokes the cgroup OOM killer and `SIGKILL`s a task in the cgroup.	Always, on anything you don't fully trust. This is the only knob that actually bounds a container's blast radius — the one that stops a leaking sidecar from taking the node down with it.
memory.high not exposed by Docker	—	Soft wall. The kernel reclaims aggressively and throttles the allocating task (puts it to sleep proportionally to the overage). It never kills.	When you'd rather a memory-hungry batch job crawl than die. Kubernetes doesn't expose it either; systemd does, as `MemoryHigh=`. Set it below `memory.max` so you get backpressure before the cliff.
memory.low	`--memory-reservation`	Reclaim protection, not a limit. Under host-level memory pressure the kernel tries to avoid reclaiming this cgroup below the value. It never throttles and never kills; you can sail past it freely.	Best-effort prioritization on a packed host. Worth knowing mostly because `--memory-reservation` is routinely mistaken for a soft limit — it is not one.
memory.swap.max	`--memory-swap` (a combined memory+swap total)	Caps how much of this cgroup's memory may live in swap. Docker's default is `--memory-swap = 2 × --memory`, i.e. an equal amount of swap on top of RAM.	Set `--memory-swap` equal to `--memory` to switch swap off, which is the only way to make `--memory=50m` mean "50 MB total". Part 2 is what happens when you don't.
memory.current memory.events	(`docker stats`, partly)	Not thresholds — the kernel's live byte counter and its running tally of `high` throttles, `max` wall-hits, and `oom_kill`s.	Diagnosis. `memory.events` is the only place that tells you the kernel killed something and how hard it fought first.

memory.current is not RSS It counts anonymous memory and page cache, kernel slab, socket buffers, and tmpfs — every page charged to the cgroup, whoever touched it first. A container that reads a large file has a high memory.current while its processes allocate nothing. Part 5 pushes this until something dies.

The kill is a signal, not an error SIGKILL cannot be caught, blocked, or handled. There is no MemoryError, no std::bad_alloc, no OutOfMemoryError, no chance to flush a log line — the language runtime is never consulted, because the kernel refuses the page rather than the allocation. What you get is an exit status of 137 = 128 + 9, the shell's convention for "terminated by signal 9". A JVM's own OutOfMemoryError is the opposite thing entirely: that's the runtime enforcing its own -Xmx ceiling in userspace, and it fires whether or not a cgroup exists.

your code malloc / new / bytearray() │ ▼ page fault ────▶ kernel charges the page to memory.current │ ├── under memory.max ──────────────▶ page granted │ └── would exceed memory.max │ ├─ reclaim succeeds ──────▶ page granted, memory.events "max" +1 │ (evict page cache, swap out anon) │ └─ reclaim fails ─────────▶ SIGKILL, memory.events "oom_kill" +1 exit code 137

Setup

Two terminals side by side. Everything runs in containers named cg-*, so cleanup at the end is by name only.

Make a working directory and write the allocator:

mkdir -p ~/cgroup-lab && cd ~/cgroup-lab
cat > alloc.py <<'EOF'
import os, sys, time

MB = 1024 * 1024
pause = float(sys.argv[1]) if len(sys.argv) > 1 else 0.0

def rd(name):
    with open("/sys/fs/cgroup/" + name) as f:
        return int(f.read()) // MB

held = []
for i in range(1, 41):
    held.append(os.urandom(5 * MB))
    print(f"allocated {i * 5:3d} MB   memory.current = {rd('memory.current'):3d} MB"
          f"   memory.swap.current = {rd('memory.swap.current'):3d} MB", flush=True)
    time.sleep(pause)
print("done: allocated 200 MB, never killed")
EOF

It allocates 5 MB at a time up to a hard ceiling of 200 MB, holding every chunk, and after each one prints the kernel's own counters read from inside its own cgroup. The 200 MB ceiling is deliberate — a bounded allocator is both safer than tail /dev/zero and better teaching material, because you can see exactly how far it got.

Why os.urandom and not zeros Zero-filled pages get special-cased on their way to swap, which makes the arithmetic in Part 2 come out wrong by tens of megabytes. Incompressible bytes keep the accounting honest. (Try it the other way in "Go deeper".)

The script is piped in on stdin, so nothing is mounted and no image is built:

docker run --rm -i --memory=50m python:3-alpine python - < alloc.py

Part 1 — the limit is a file, and the container can read it

What this part tests: where the limit actually lives. Per the mental model it's memory.max, a plain file in the container's own cgroup — not a property of the Docker daemon, and not something your runtime is told about.

Step 1 · ask the container what it's allowed

Terminal A

docker run --rm --name cg-see --memory=50m alpine:3.20 sh -c '
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events
free -m'

Predict: can a process inside the container discover the 50 MB number? And what will free report?

52428800 1699840 low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 total used free shared buff/cache available Mem: 15973 628 515 369 14830 14726 Swap: 2048 0 2048

52428800 is exactly 50 × 1024 × 1024. The limit is right there, readable by anything in the container, and memory.current (1699840 ≈ 1.6 MB) is the kernel's live tally of what the container is using right now. memory.events is all zeros: nothing has hit any threshold yet.

And then free -m says 15973 MB. /proc/meminfo is not namespaced — it reports the host's RAM, and the container is 320× wrong about its own budget. This is precisely why the JVM needed UseContainerSupport (on by default since JDK 10) and why Node's old default heap sizing killed containers: every runtime that sized itself from /proc/meminfo sized itself to the host.

(Only the total column is the point. used/free/buff/cache are the whole Docker VM's, so they move with whatever else you have running; memory.current is the container's own and does not.)

Part 2 — allocate past it (and don't die when you expect to)

What this part tests: the hard wall. memory.max is 50 MB and the script wants 200 MB, so the container cannot possibly finish. The question is where it stops.

Step 2 · run the allocator under the limit

Terminal A

docker run -i --name cg-swap --memory=50m python:3-alpine python - < alloc.py
echo "exit: $?"

Predict: at roughly what "allocated N MB" line does the script die?

allocated 5 MB memory.current = 12 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 17 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 22 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 27 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 32 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 37 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 42 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 47 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 50 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 55 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 60 MB memory.current = 49 MB memory.swap.current = 18 MB allocated 65 MB memory.current = 49 MB memory.swap.current = 24 MB allocated 70 MB memory.current = 49 MB memory.swap.current = 28 MB allocated 75 MB memory.current = 49 MB memory.swap.current = 34 MB allocated 80 MB memory.current = 49 MB memory.swap.current = 37 MB allocated 85 MB memory.current = 49 MB memory.swap.current = 43 MB allocated 90 MB memory.current = 49 MB memory.swap.current = 48 MB exit: 137

It allocated 90 MB inside a 50 MB limit before dying. Read the two counters as a pair and it's obvious what happened:

Up to 40 MB, memory.current tracks the allocation exactly, a constant interpreter overhead above it.
At 45 MB it hits the wall. memory.current stops climbing — and never exceeds 49 again for the rest of the run. That flat line is the limit being enforced, allocation by allocation.
From that point memory.swap.current does the climbing instead: 0 → 14 → 24 → 34 → 43 → 48 MB. Every 5 MB the script asks for, the kernel reclaims 5 MB by pushing older pages out to swap.
The kill comes when both budgets are full: 49 MB resident + 48 MB swapped = 97 MB, against a total budget of 100 MB.

(The exact swap figures move by a few MB run to run; the shape — resident pinned at 49 while swap climbs to ~48 — does not.)

100 MB, because --memory-swap defaults to twice --memory, and --memory-swap is the combined ceiling. --memory=50m alone quietly grants the container 50 MB of swap on top of its 50 MB of RAM.

Step 3 · confirm the default

Terminal A

docker run --rm --name cg-see --memory=50m alpine:3.20 \
  sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)'
docker run --rm --name cg-see --memory=50m --memory-swap=50m alpine:3.20 \
  sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)'

max=52428800 swap=52428800 max=52428800 swap=0

Part 3 — the clean kill

What this part tests: the same wall with swap removed, so the number in the flag is the whole budget. Also what the process experiences, which per the mental model is nothing at all.

Step 4 · a control run with no limit

Terminal A

docker run --rm -i --name cg-nolimit python:3-alpine python - < alloc.py

lines removed allocated 190 MB memory.current = 197 MB memory.swap.current = 0 MB allocated 195 MB memory.current = 202 MB memory.swap.current = 0 MB allocated 200 MB memory.current = 207 MB memory.swap.current = 0 MB done: allocated 200 MB, never killed

Straight through all 200 MB, memory.current tracking it the whole way, exit 0. Nothing in the program changes for the rest of this exercise.

Step 5 · the same script, swap off

Terminal A

docker run -i --name cg-hard --memory=50m --memory-swap=50m python:3-alpine python - < alloc.py
echo "exit: $?"
docker inspect cg-hard --format 'OOMKilled={{.State.OOMKilled}}  ExitCode={{.State.ExitCode}}'

Predict: where does it stop — and what does the output look like at the end? A traceback? A MemoryError? An exit message?

allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 14 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 OOMKilled=true ExitCode=137

Nine lines, then nothing. No traceback, no MemoryError, no "killed" message from Python — the output simply stops mid-stream, because the 45 MB line was flushed and the process was gone before it could print a tenth. memory.swap.current stayed at 0 the entire run, so the 50 MB in the flag really was the whole budget this time.

137 is the only thing your orchestrator sees: 128 + 9, terminated by SIGKILL. OOMKilled=true is Docker's own record, read back out of the container's state afterwards — the one place that distinguishes "the kernel shot it" from "it exited 137 for some other reason".

Part 4 — watching it from the outside

What this part tests: that the enforcement is observable live, from a different terminal, by something that knows nothing about the program. Per the mental model, docker stats reads memory.current and memory.events is the kernel's own tally.

This part needs both terminals Open them side by side. Terminal A runs the container and the allocator; Terminal B does nothing but watch.

Step 6 · a container that survives the kill

Terminal A

docker run -d --name cg-watch --memory=50m --memory-swap=50m python:3-alpine sleep 3600
docker exec cg-watch cat /sys/fs/cgroup/memory.events

low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0

A clean tally to compare against.

Step 7 · start watching, then allocate

Terminal B — leave this running:

docker stats --format '{{.Name}}   {{.MemUsage}}   {{.MemPerc}}' cg-watch

Terminal A — run the allocator inside the existing container, with a 0.6 s pause per chunk so B can keep up:

docker exec -i cg-watch python - 0.6 < alloc.py
echo "exit: $?"

Predict: what does B show at the moment of the kill, and what does it show one second later?

Terminal A — the allocator

allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB lines removed allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137

Terminal B — successive values docker stats redrew

cg-watch 524KiB / 50MiB 1.02% cg-watch 9.215MiB / 50MiB 18.43% cg-watch 14.01MiB / 50MiB 28.02% cg-watch 24.04MiB / 50MiB 48.07% cg-watch 34.07MiB / 50MiB 68.13% cg-watch 39.08MiB / 50MiB 78.16% cg-watch 49.3MiB / 50MiB 98.59% cg-watch 608KiB / 50MiB 1.19%

The climb is visible right up to 98.59%, and the next sample is 1.19%. There is no intermediate state — no "shutting down", no "cleaning up". One second the memory is charged, the next it isn't, because the process holding it ceased to exist between samples. (Which sample lands closest to 100% is luck of the polling interval; the cliff to ~1% is not.)

Step 8 · ask the kernel what it did

Terminal A

docker exec cg-watch cat /sys/fs/cgroup/memory.events
docker inspect cg-watch --format 'Running={{.State.Running}}  OOMKilled={{.State.OOMKilled}}'

Predict: how many times did this cgroup hit its limit? (It was killed once.)

low 0 high 0 max 37 oom 1 oom_kill 1 oom_group_kill 0 Running=true OOMKilled=true

max 37 — the cgroup hit memory.max thirty-seven times and survived, by reclaiming. The kill was not the first time it touched the wall; it was the first time reclaim came up empty. (The count lands in the thirties run to run; what matters is that it is dozens, not one.)
oom 1 — the OOM killer was invoked once.
oom_kill 1 — it killed exactly one task.

And Running=true OOMKilled=true: the container is alive (the sleep was never touched), but Docker has flagged it. The OOM killer picks a victim within the cgroup by score, and a sleep holding 500 KB is a poor candidate next to a Python process holding 49 MB. In a real container this is the failure mode where PID 1 survives and a worker vanishes — the container looks healthy and half of it is dead.

Step 9 · optional — the kernel's own account

Terminal A

dmesg is the whole VM's ring buffer, shared by every container on the daemon, so filter it to your own cgroup rather than reading the tail — the cgroup id is in the oom_memcg= field of every memcg OOM report:

CID=$(docker inspect --format '{{.Id}}' cg-ddkill)
docker run --rm --privileged --name cg-dmesg alpine:3.20 dmesg \
  | grep -B4 -A1 "oom_memcg=/docker/$CID"

[125629.069168] Tasks state (memory values in pages): [125629.069168] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [125629.069182] [ 27269] 0 27269 427 204 0 204 0 45056 0 0 sleep [125629.069185] [ 27294] 0 27294 683 476 256 220 0 40960 0 0 dd [125629.069187] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,mems_allowed=0,oom_memcg=/docker/1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,task_memcg=/docker/1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,task=dd,pid=27294,uid=0 [125629.069257] Memory cgroup out of memory: Killed process 27294 (dd) total-vm:2732kB, anon-rss:1024kB, file-rss:880kB, shmem-rss:0kB, UID:0 pgtables:40kB oom_score_adj:0

(This capture is of the Part 5 dd kill rather than the Python one — same report, and the numbers in it are the point of Part 5, so run it after step 13. Swap cg-ddkill for cg-watch to see the Python one.) constraint=CONSTRAINT_MEMCG is the kernel saying this was a cgroup OOM, not the host running out of memory, and oom_memcg=/docker/1288b2d31596… names the container. The scoring table above it lists every task that was a candidate.

Part 5 — page cache counts, and `docker stats` lies about it

What this part tests: the accounting caveat from the mental model — memory.current is not RSS. This is the part most likely to break your intuition, so take the predictions seriously.

Step 10 · write a 40 MB file with a 1 MB buffer

Terminal A

docker run -d --name cg-cache --memory=50m --memory-swap=50m python:3-alpine sleep 600
docker exec cg-cache sh -c '
  dd if=/dev/urandom of=/big bs=1M count=40
  sync
  echo memory.current=$(cat /sys/fs/cgroup/memory.current)
  grep -E "^(anon|file|inactive_file) " /sys/fs/cgroup/memory.stat'

Predict: dd allocated 1 MB. What is memory.current now, out of a 50 MB budget?

40+0 records in 40+0 records out 41943040 bytes (40.0MB) copied, 0.098399 seconds, 406.5MB/s memory.current=44498944 anon 98304 file 41955328 inactive_file 41955328

memory.current is 44498944 — 42.4 MiB, or 85% of the limit — for a container running sleep and a dd that allocated 1 MB. anon is 98304, ninety-six kilobytes: essentially none of it is anybody's heap. file 41955328 is the 40 MB file to within a few pages. The page cache is charged to the cgroup that faulted it in, and there is no distinction in memory.max between "cache" and "heap".

Step 11 · now ask Docker

Terminal B

docker stats --no-stream --format '{{.Name}}   {{.MemUsage}}   {{.MemPerc}}' cg-cache

Predict: the kernel says 85% of 50 MB. What percentage does docker stats report?

cg-cache 1.719MiB / 50MiB 3.44%

3.44%. The kernel says 85%, Docker says 3%. Neither is lying: Docker subtracts inactive_file from memory.current before reporting, on the theory that reclaimable cache isn't "really" used — and here inactive_file is 41955328, essentially the whole charge. That is a defensible dashboard choice and a terrible debugging one, because the number memory.max is compared against is the unsubtracted one.

If you are chasing an OOM kill, docker stats is the wrong instrument Read memory.current. A container can sit at 3% in your dashboard and be one page away from the wall.

Step 12 · allocate 30 MB into a container that has 8 MB "free"

Terminal A

docker exec cg-cache python -c "
import os
x = os.urandom(30*1024*1024)
print('allocated 30 MB, still alive')
print('memory.current =', open('/sys/fs/cgroup/memory.current').read().strip())
for l in open('/sys/fs/cgroup/memory.stat'):
    if l.split()[0] in ('anon','file','pgscan','pgsteal'): print(l.strip())
print(open('/sys/fs/cgroup/memory.events').read().strip())
"

Predict: memory.current is at 42 MB of 50, and this wants 30 MB more. Does it get killed?

allocated 30 MB, still alive memory.current = 52428800 anon 35160064 file 15855616 pgscan 6400 pgsteal 6400 low 0 high 0 max 100 oom 0 oom_kill 0 oom_group_kill 0

It survived. file fell from 41955328 to 15855616 — the kernel threw away 25 MB of clean page cache to make room — while anon rose to 35160064. pgscan 6400 / pgsteal 6400 is the reclaim: 6400 pages scanned, 6400 successfully freed, a 100% hit rate because clean file pages are free to drop. (6400 pages × 4 KiB = 25 MiB — exactly the cache that disappeared.)

max 100, oom_kill 0 It hit the wall a hundred times and was never killed once. memory.max is not "kill at 50 MB". It's "never exceed 50 MB", and killing is only what happens when the kernel runs out of other ways to obey. memory.current = 52428800 is memory.max to the byte — the cgroup sitting exactly flat against its ceiling, working.

Step 13 · the same trick at 300 MB

Terminal A — a fresh container so nothing above muddies it:

docker run -d --name cg-ddkill --memory=50m --memory-swap=50m alpine:3.20 sleep 600
docker exec cg-ddkill dd if=/dev/urandom of=/big bs=1M count=300
echo "exit: $?"
docker exec cg-ddkill sh -c '
  ls -lh /big
  echo memory.current=$(cat /sys/fs/cgroup/memory.current)
  grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat
  cat /sys/fs/cgroup/memory.events'

Predict: clean cache was free to reclaim in step 12, and dd still only holds 1 MB at a time. So this is fine, right?

exit: 137 -rw-r--r-- 1 root root 47.0M Jul 25 16:08 /big memory.current=52260864 anon 167936 file 49250304 low 0 high 0 max 22 oom 1 oom_kill 1 oom_group_kill 0

dd printed nothing — not even its usual records in / records out — it was OOM-killed with an anon charge for the whole cgroup of 167936 bytes, 164 kilobytes. The kernel's own report from step 9 puts it even more starkly: Killed process 27294 (dd) total-vm:2732kB, anon-rss:1024kB. A process whose entire resident heap is one megabyte, killed for exceeding a fifty megabyte limit.

The difference from step 12 is dirty. Reclaiming a clean page means dropping it; reclaiming a dirty page means waiting for writeback to finish first. dd was producing dirty pages at the ~400 MB/s measured in step 10 and the disk could not retire them that fast, so the cgroup filled with cache the kernel was not yet allowed to drop, hit the wall 22 times trying, and then killed the only task making it worse. The file stops at 47.0 MB of the requested 300.

(How far dd gets and how many max hits it takes vary with disk speed and load — 47 MB and 22 here. That it dies at all, holding a 1 MB buffer, does not.)

The shape of a whole class of production incidents A container that is fine all day and dies during the nightly export, backup, or log rotation. Nothing leaked. It wrote a file.

Part 6 — the same number as a soft limit

What this part tests: the hard/soft distinction from the mental model, with everything else held constant. memory.high is the same 50 MB threshold, enforced by throttling instead of killing.

Docker doesn't expose memory.high, so this needs --privileged to remount the cgroup filesystem read-write and set the file by hand. That is a lab technique, not a deployment pattern — in the real world this is systemd's MemoryHigh=.

Step 14 · 50 MB, softly

Terminal A

docker run --rm -i --privileged --name cg-soft python:3-alpine sh -c '
  mount -o remount,rw /sys/fs/cgroup
  echo 50M > /sys/fs/cgroup/memory.high
  python -
  echo "python exit: $?"
  cat /sys/fs/cgroup/memory.events' < alloc.py

Predict: memory.high = 50M, memory.max left at max, same script wanting 200 MB. What happens?

allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB lines removed allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 50 MB memory.swap.current = 17 MB lines removed allocated 95 MB memory.current = 49 MB memory.swap.current = 53 MB lines removed allocated 140 MB memory.current = 49 MB memory.swap.current = 98 MB lines removed allocated 195 MB memory.current = 49 MB memory.swap.current = 157 MB allocated 200 MB memory.current = 49 MB memory.swap.current = 162 MB done: allocated 200 MB, never killed python exit: 0 low 0 high 159 max 0 oom 0 oom_kill 0 oom_group_kill 0

(Trimmed — the run prints all forty lines; memory.current never leaves the 43–50 range after the tenth.)

Same program, same 50 MB number, opposite outcome. All 200 MB allocated, exit 0, and memory.current pinned in the high forties for the entire run — the ceiling was honoured just as strictly as memory.max honoured it. The difference is what happened at the ceiling: memory.swap.current climbed to 162 MB, three times the RAM limit, because with no memory.max there was nothing to cap it.

memory.events	memory.max = 50M	memory.high = 50M
max	37	0
high	0	159
oom_kill	1	0
exit code	137	0
allocated before stopping	45 MB	200 MB

high 159 counts the times the kernel throttled the allocating task — put it to sleep to let reclaim catch up. That is the entire mechanism: the same threshold, applied as backpressure rather than as a wall.

What you should see

A container that reads its own limit out of /sys/fs/cgroup/memory.max as 52428800 while free -m insists it has 15973 MB. An allocator that gets 90 MB into a 50 MB container before dying, because --memory=50m silently grants 50 MB of swap as well — and memory.current that flattens at 49 and never once exceeds it. With swap off, a clean kill at 45 MB with no traceback, exit 137, OOMKilled=true. docker stats climbing to 98.59% and reading 1.19% one second later. memory.events reporting max 37, oom_kill 1 — thirty-seven survived collisions before the fatal one. A dd with a 1 MB buffer pushing memory.current to 85% of the limit while docker stats reports 3.44%, and then getting SIGKILLed at anon-rss:1024kB. And the same 50 MB expressed as memory.high letting the identical script finish all 200 MB with high 159, oom_kill 0, exit 0.

Why

Because the limit is enforced at page-fault time, in the kernel, against a counter the kernel maintains — and there is no path from there back into your program. When a fault would push memory.current past memory.max, the kernel's options are to make room or to stop the cgroup growing. Making room means reclaim: drop clean page cache, write back dirty pages, swap out anonymous ones. That succeeded 37 times in Part 4 and 100 times in Part 5, invisibly, which is why max counts in the dozens and hundreds on a container that is merely busy.

Only when reclaim returns empty-handed does the second option apply, and the kernel cannot implement it by failing an allocation. The allocation already succeeded — os.urandom returned, malloc returned, the page table entry exists. What's failing is the fault, on memory the process already believes it owns. Handing that back as an error would mean inventing a failure at an instruction that has no error path. So the kernel removes the process instead, with the one signal that cannot be caught, and your program learns nothing because there is nobody left to tell.

That is also why memory.high can exist at all. Throttling is something you can do to a running task — sleep it, let reclaim catch up, resume it — without ever needing an error path. It's strictly more graceful, it's what you actually want for batch work, and Docker exposes no flag for it.

And the page cache results follow from the same accounting rather than from any special rule. Charging cache to the cgroup that faulted it in is the only choice that makes limits meaningful — otherwise any container could pin unlimited host memory by reading files. The consequence is that memory.current is a measure of pages the cgroup is responsible for, not pages its processes allocated, and the OOM killer's victim is chosen by RSS among tasks that may have had nothing to do with the pressure. dd at anon-rss:1024kB was not the cause of the problem. It was just the biggest thing in the room.

Where the effect vanishes Give the cgroup swap it can actually use and the wall stops being a wall for anonymous memory — Part 2's container reached 90 MB, Part 6's reached 200. Make the pressure entirely clean page cache and the kill vanishes too: step 12 hit the limit a hundred times without a scratch, because every page the kernel wanted was droppable. The kill needs both conditions at once — nowhere to put the pages, and pages that can't be thrown away.

Go deeper

Drop the sync in step 10 and re-run the 30 MB allocation immediately after the dd. With the 40 MB still dirty and in flight, the same allocation that survived at step 12 gets exit 137 instead — the same distinction as step 13, at a scale small enough to run in a second. On this machine it reproduced on all five attempts; on a faster disk the writeback may win the race, so run it a few times.
Swap the allocator's os.urandom(5 * MB) back to bytearray(5 * MB) and re-run Part 2. Zero-filled pages get special-cased in the swap path, and the container survives to 125 MB instead of 90 (three runs, 125 MB every time) — the same limit, honoured identically, with the arithmetic thrown off by page content. A good reminder that synthetic memory benchmarks are easy to get wrong.
Try --oom-kill-disable on Part 3's run and read the warning: WARNING: Your kernel does not support OomKillDisable. OomKillDisable discarded. The flag is a cgroup v1 feature (memory.oom_control, which paused the cgroup at the limit instead of killing it). v2 removed the ability to disable the OOM killer, Docker keeps the flag for compatibility, and it silently does nothing. Worth knowing before you find it in someone's docker-compose.yml and assume it's protecting anything.
Run two allocators in the same container at once and watch which one the OOM killer picks, in dmesg's scoring table. Then bias the choice: echo 500 > /proc/self/oom_score_adj in one of them makes it the preferred victim. Note the asymmetry — raising the score works unprivileged, but echo -1000 (the way you protect a supervisor from its own workers) fails with sh: write error: Permission denied unless the container has CAP_SYS_RESOURCE. You may volunteer to die; you may not opt out.
Compare with the CPU side: --cpus=0.5 is enforced by the CFS bandwidth controller, which throttles rather than kills, because a process can always be given less CPU later but cannot be given back memory it already wrote to. Same cgroup, two very different enforcement stories.

Sources: Linux kernel docs: cgroup v2 memory interface files — the memory.high section spells out the throttling behaviour Part 6 counts · Docker: runtime options with memory, CPUs, and GPUs for --memory-swap's combined-total semantics

Cleanup

docker rm -f cg-watch cg-cache cg-ddkill cg-swap cg-hard

cg-see, cg-nolimit, cg-soft and cg-dmesg were all --rm and are already gone. The allocator lives in ~/cgroup-lab/alloc.py; delete the directory if you're done.

cgroup Memory Limits

Concept

Provenance

Prerequisites

Environment

Mental model: five files and one signal

Setup

Part 1 — the limit is a file, and the container can read it

Part 2 — allocate past it (and don't die when you expect to)

Part 3 — the clean kill

Part 4 — watching it from the outside

Part 5 — page cache counts, and docker stats lies about it

Part 6 — the same number as a soft limit

What you should see

Why

Go deeper

Cleanup

Part 5 — page cache counts, and `docker stats` lies about it