# Docker: cgroup memory limits ## Concept `docker run --memory=50m` does not tell your program anything. It writes a number into a file — `/sys/fs/cgroup/memory.max` — and from that moment the Linux kernel accounts every page the container touches against it. When the container can't stay under the number, the kernel does not raise an exception, call a handler, or return `NULL` from `malloc`. It sends `SIGKILL`. Your process gets no turn. This exercise makes the whole chain visible: the limit as a file the container can read, the kernel's live counter climbing toward it in `docker stats`, the kill, and the kernel's own tally of what it did. Along the way three things will probably not match your prediction — a container limited to 50 MB will allocate 90 MB before dying, `docker stats` will report 3% while the kernel says 85%, and a `dd` whose own resident heap is one megabyte will get OOM-killed. ## Provenance **Linux kernel documentation, `Documentation/admin-guide/cgroup-v2.rst` — "Controllers → Memory → Memory Interface Files"** ([rendered](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files)). The claim under test is that section's one-sentence definition of `memory.max`: > Memory usage hard limit. This is the main mechanism to limit memory usage > of a cgroup. If a cgroup's memory usage reaches this limit and can't be > reduced, the OOM killer is invoked in the cgroup. and, for contrast, its definition of `memory.high`: > Memory usage throttle limit. If a cgroup's usage goes over the high > boundary, the processes of the cgroup are throttled and put under heavy > reclaim pressure. The two `memory.events` counters that let you tell those apart, from the same section: > **max** — The number of times the cgroup's memory usage was about to go > over the max boundary. If direct reclaim fails to bring it down, the cgroup > goes to OOM state. > > **oom_kill** — The number of processes belonging to this cgroup killed by > any kind of OOM killer. Read `max` carefully: it counts *near misses*, not kills. Parts 4 and 5 turn entirely on that distinction, and on `memory.current`'s definition — "the total amount of memory currently being used by the cgroup and its descendants", which is not the same thing as the memory its *processes* allocated. The swap arithmetic in Part 2 comes from Docker's **"Runtime options with Memory, CPUs, and GPUs" → `--memory-swap` details** ([docs](https://docs.docker.com/engine/containers/resource_constraints/)): > If `--memory-swap` is unset, and `--memory` is set, the container can use > as much swap as the `--memory` setting, if the host container has swap > memory configured. […] If `--memory-swap` is set to the same value as > `--memory`, and `--memory` is set to a positive integer, the container > doesn't have access to swap. (`cgroups(7)` describes the hierarchy and the delegation rules, not these files. It's the right man page for *what a cgroup is*, the wrong one for what `memory.max` does.) ## Prerequisites Two ideas carry almost all of the result; the other two are background. - **cgroup v2 memory accounting — load-bearing.** `memory.current` is a charge counter over *pages*, not a sum of process RSS. Page cache, kernel slab, socket buffers, and tmpfs are all charged to whichever cgroup first faulted them in, and there is no separate budget for "cache". Every surprise in Part 5 falls out of this one rule. → [kernel docs: "Memory Ownership"](https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-ownership) - **Page reclaim, clean vs. dirty — load-bearing.** A clean file-backed page can be dropped instantly; a dirty one must be written back first, and the kernel has to wait. This is the entire difference between step 27 (survives) and step 30 (killed). Anonymous pages have no backing file, so reclaiming them means swapping — which is why Part 2 and Part 3 differ. → [Mel Gorman, *Understanding the Linux Virtual Memory Manager*, ch. 10, "Page Frame Reclamation"](https://www.kernel.org/doc/gorman/html/understand/understand013.html) - **`SIGKILL` is uncatchable.** Used in Part 3 to explain why there is no traceback. `SIGKILL` and `SIGSTOP` are the two signals that cannot be caught, blocked, or ignored. → [`signal(7)`](https://man7.org/linux/man-pages/man7/signal.7.html) - **Exit status 128 + N.** Used everywhere `137` appears. A shell reports a signal-terminated child as `128 + signal number`; `SIGKILL` is 9. → [Bash reference, "Exit Status"](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html) ## Environment - **Host:** macOS 26.5.2 (build 25F84), Apple Silicon (arm64) - **Docker:** Docker Desktop, Engine 29.6.2 — Linux VM kernel `6.12.76-linuxkit`, 16 GiB allocated to the VM - **cgroup:** v2, driver `cgroupfs`, `cgroupns` enabled - **VM swap:** 2048 MB configured. Part 2 depends on this existing; on a VM with swap disabled the container in Part 2 dies at ~45 MB like Part 3's. - **Alpine image:** `alpine:3.20` — Alpine 3.20.10, digest `sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc` - **Python image:** `python:3-alpine` — Python 3.14.6 on Alpine 3.24.1, digest `sha256:26730869004e2b9c4b9ad09cab8625e81d256d1ce97e72df5520e806b1709f92` - **Captured:** 2026-07-26 ## Mental model: five files and one signal Everything below is cgroup **v2** (`/sys/fs/cgroup/memory.*`). If you find `memory.limit_in_bytes` on your machine you're on v1 and the file names differ; check with `docker info | grep -i cgroup`. | File | Docker flag | What the kernel does at this threshold | Reach for it when… | | --- | --- | --- | --- | | `memory.max` | `--memory` / `-m` | **Hard wall.** On the allocation that would cross it, the kernel first tries to reclaim (evict page cache, swap out anonymous pages). If reclaim can't free enough, it invokes the cgroup OOM killer and `SIGKILL`s a task in the cgroup. | Always, on anything you don't fully trust. This is the only knob that actually bounds a container's blast radius — the one that stops a leaking sidecar from taking the node down with it. | | `memory.high` | *not exposed by Docker* | **Soft wall.** The kernel reclaims aggressively and *throttles* the allocating task (puts it to sleep proportionally to the overage). It never kills. | When you'd rather a memory-hungry batch job crawl than die. Kubernetes doesn't expose it either; systemd does, as `MemoryHigh=`. Set it below `memory.max` so you get backpressure before the cliff. | | `memory.low` | `--memory-reservation` | **Reclaim protection, not a limit.** Under *host*-level memory pressure the kernel tries to avoid reclaiming this cgroup below the value. It never throttles and never kills; you can sail past it freely. | Best-effort prioritization on a packed host. Worth knowing mostly because `--memory-reservation` is routinely mistaken for a soft limit — it is not one. | | `memory.swap.max` | `--memory-swap` (as a *combined* memory+swap total) | Caps how much of this cgroup's memory may live in swap. Docker's default is `--memory-swap = 2 × --memory`, i.e. an equal amount of swap on top of RAM. | Set `--memory-swap` equal to `--memory` to switch swap off, which is the only way to make `--memory=50m` mean "50 MB total". Part 2 is what happens when you don't. | | `memory.current` / `memory.events` | (`docker stats`, partly) | Not thresholds — the kernel's live byte counter and its running tally of `high` throttles, `max` wall-hits, and `oom_kill`s. | Diagnosis. `memory.events` is the only place that tells you the kernel killed something *and* how hard it fought first. | Two things about that accounting that catch people out: **`memory.current` is not RSS.** It counts anonymous memory *and* page cache, kernel slab, socket buffers, and tmpfs — every page charged to the cgroup, whoever touched it first. A container that reads a large file has a high `memory.current` while its processes allocate nothing. Part 5 pushes this until something dies. **The kill is a signal, not an error.** `SIGKILL` cannot be caught, blocked, or handled. There is no `MemoryError`, no `std::bad_alloc`, no `OutOfMemoryError`, no chance to flush a log line — the language runtime is never consulted, because the kernel refuses the page rather than the allocation. What you get is an exit status of **137 = 128 + 9**, the shell's convention for "terminated by signal 9". A JVM's own `OutOfMemoryError` is the opposite thing entirely: that's the runtime enforcing its *own* `-Xmx` ceiling in userspace, and it fires whether or not a cgroup exists. ``` your code malloc / new / bytearray() │ ▼ page fault ────▶ kernel charges the page to memory.current │ ├── under memory.max ──────────────▶ page granted │ └── would exceed memory.max │ ├─ reclaim succeeds ──────▶ page granted, memory.events "max" +1 │ (evict page cache, swap out anon) │ └─ reclaim fails ─────────▶ SIGKILL, memory.events "oom_kill" +1 exit code 137 ``` ## Setup Two terminals side by side. Everything runs in containers named `cg-*`, so cleanup at the end is by name only. Make a working directory and write the allocator: ```bash mkdir -p ~/cgroup-lab && cd ~/cgroup-lab cat > alloc.py <<'EOF' import os, sys, time MB = 1024 * 1024 pause = float(sys.argv[1]) if len(sys.argv) > 1 else 0.0 def rd(name): with open("/sys/fs/cgroup/" + name) as f: return int(f.read()) // MB held = [] for i in range(1, 41): held.append(os.urandom(5 * MB)) print(f"allocated {i * 5:3d} MB memory.current = {rd('memory.current'):3d} MB" f" memory.swap.current = {rd('memory.swap.current'):3d} MB", flush=True) time.sleep(pause) print("done: allocated 200 MB, never killed") EOF ``` It allocates 5 MB at a time up to a hard ceiling of 200 MB, holding every chunk, and after each one prints the kernel's own counters *read from inside its own cgroup*. The 200 MB ceiling is deliberate — a bounded allocator is both safer than `tail /dev/zero` and better teaching material, because you can see exactly how far it got. `os.urandom` rather than zeros is also deliberate: zero-filled pages get special-cased on their way to swap, which makes the arithmetic in Part 2 come out wrong by tens of megabytes. Incompressible bytes keep the accounting honest. The script is piped in on stdin, so nothing is mounted and no image is built: ```bash docker run --rm -i --memory=50m python:3-alpine python - < alloc.py ``` ## Steps ### Part 1 — the limit is a file, and the container can read it What this part tests: where the limit actually lives. Per the mental model it's `memory.max`, a plain file in the container's own cgroup — not a property of the Docker daemon, and not something your runtime is told about. 1. **Predict**: a container started with `--memory=50m`. Can a process *inside* it discover that number? And what will `free` report? 2. **Terminal A**: ```bash docker run --rm --name cg-see --memory=50m alpine:3.20 sh -c ' cat /sys/fs/cgroup/memory.max cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.events free -m' ``` 3. **Observe**: ``` 52428800 1699840 low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 total used free shared buff/cache available Mem: 15973 628 515 369 14830 14726 Swap: 2048 0 2048 ``` `52428800` is exactly 50 × 1024 × 1024. The limit is right there, readable by anything in the container, and `memory.current` (1699840 ≈ 1.6 MB) is the kernel's live tally of what the container is using *right now*. `memory.events` is all zeros: nothing has hit any threshold yet. And then `free -m` says **15973 MB**. `/proc/meminfo` is not namespaced — it reports the host's RAM, and the container is 320× wrong about its own budget. This is precisely why the JVM needed `UseContainerSupport` (on by default since JDK 10) and why Node's old default heap sizing killed containers: every runtime that sized itself from `/proc/meminfo` sized itself to the host. (Only the `total` column is the point. `used`/`free`/`buff/cache` are the whole Docker VM's, so they move with whatever else you have running; `memory.current` is the container's own and does not.) ### Part 2 — allocate past it (and don't die when you expect to) What this part tests: the hard wall. `memory.max` is 50 MB and the script wants 200 MB, so the container cannot possibly finish. The question is *where* it stops. 4. **Predict**: at roughly what "allocated N MB" line does the script die? 5. **Terminal A**: ```bash docker run -i --name cg-swap --memory=50m python:3-alpine python - < alloc.py echo "exit: $?" ``` 6. **Observe**: ``` allocated 5 MB memory.current = 12 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 17 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 22 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 27 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 32 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 37 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 42 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 47 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 50 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 55 MB memory.current = 49 MB memory.swap.current = 14 MB allocated 60 MB memory.current = 49 MB memory.swap.current = 18 MB allocated 65 MB memory.current = 49 MB memory.swap.current = 24 MB allocated 70 MB memory.current = 49 MB memory.swap.current = 28 MB allocated 75 MB memory.current = 49 MB memory.swap.current = 34 MB allocated 80 MB memory.current = 49 MB memory.swap.current = 37 MB allocated 85 MB memory.current = 49 MB memory.swap.current = 43 MB allocated 90 MB memory.current = 49 MB memory.swap.current = 48 MB exit: 137 ``` It allocated **90 MB inside a 50 MB limit** before dying. Read the two counters as a pair and it's obvious what happened: - Up to 40 MB, `memory.current` tracks the allocation exactly, a constant interpreter overhead above it. - At 45 MB it hits the wall. `memory.current` stops climbing — and *never exceeds 49 again for the rest of the run*. That flat line is the limit being enforced, allocation by allocation. - From that point `memory.swap.current` does the climbing instead: 0 → 14 → 24 → 34 → 43 → 48 MB. Every 5 MB the script asks for, the kernel reclaims 5 MB by pushing older pages out to swap. - The kill comes when both budgets are full: 49 MB resident + 48 MB swapped = 97 MB, against a total budget of 100 MB. (The exact swap figures move by a few MB run to run; the shape — resident pinned at 49 while swap climbs to ~48 — does not.) 100 MB, because `--memory-swap` defaults to **twice** `--memory`, and `--memory-swap` is the *combined* ceiling. `--memory=50m` alone quietly grants the container 50 MB of swap on top of its 50 MB of RAM. 7. **Terminal A**: confirm the default: ```bash docker run --rm --name cg-see --memory=50m alpine:3.20 \ sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)' docker run --rm --name cg-see --memory=50m --memory-swap=50m alpine:3.20 \ sh -c 'echo max=$(cat /sys/fs/cgroup/memory.max) swap=$(cat /sys/fs/cgroup/memory.swap.max)' ``` ``` max=52428800 swap=52428800 max=52428800 swap=0 ``` ### Part 3 — the clean kill What this part tests: the same wall with swap removed, so the number in the flag is the whole budget. Also what the *process* experiences, which per the mental model is nothing at all. 8. **Terminal A**: baseline first — the identical script with no limit: ```bash docker run --rm -i --name cg-nolimit python:3-alpine python - < alloc.py ``` ``` ⋮ allocated 190 MB memory.current = 197 MB memory.swap.current = 0 MB allocated 195 MB memory.current = 202 MB memory.swap.current = 0 MB allocated 200 MB memory.current = 207 MB memory.swap.current = 0 MB done: allocated 200 MB, never killed ``` Straight through all 200 MB, `memory.current` tracking it the whole way, exit 0. Nothing in the program changes for the rest of this exercise. 9. **Predict**: now with `--memory=50m --memory-swap=50m`. Where does it stop, and what does the *output* look like at the end — a traceback? a `MemoryError`? an exit message? 10. **Terminal A**: ```bash docker run -i --name cg-hard --memory=50m --memory-swap=50m python:3-alpine python - < alloc.py echo "exit: $?" docker inspect cg-hard --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' ``` 11. **Observe**: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB allocated 10 MB memory.current = 14 MB memory.swap.current = 0 MB allocated 15 MB memory.current = 19 MB memory.swap.current = 0 MB allocated 20 MB memory.current = 24 MB memory.swap.current = 0 MB allocated 25 MB memory.current = 29 MB memory.swap.current = 0 MB allocated 30 MB memory.current = 34 MB memory.swap.current = 0 MB allocated 35 MB memory.current = 39 MB memory.swap.current = 0 MB allocated 40 MB memory.current = 44 MB memory.swap.current = 0 MB allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 OOMKilled=true ExitCode=137 ``` Nine lines, then nothing. No traceback, no `MemoryError`, no "killed" message from Python — the output simply stops mid-stream, because the 45 MB line was flushed and the process was gone before it could print a tenth. `memory.swap.current` stayed at `0` the entire run, so the 50 MB in the flag really was the whole budget this time. `137` is the only thing your orchestrator sees: 128 + 9, terminated by `SIGKILL`. `OOMKilled=true` is Docker's own record, read back out of the container's state afterwards — the one place that distinguishes "the kernel shot it" from "it exited 137 for some other reason". ### Part 4 — watching it from the outside What this part tests: that the enforcement is observable live, from a different terminal, by something that knows nothing about the program. Per the mental model, `docker stats` reads `memory.current` and `memory.events` is the kernel's own tally. This part needs **both terminals**. 12. **Terminal A**: start a long-lived container so it survives the kill, and check the tally is clean: ```bash docker run -d --name cg-watch --memory=50m --memory-swap=50m python:3-alpine sleep 3600 docker exec cg-watch cat /sys/fs/cgroup/memory.events ``` ``` low 0 high 0 max 0 oom 0 oom_kill 0 oom_group_kill 0 ``` 13. **Terminal B**: start watching. Leave this running. ```bash docker stats --format '{{.Name}} {{.MemUsage}} {{.MemPerc}}' cg-watch ``` 14. **Terminal A**: run the allocator *inside* the existing container, with a 0.6 s pause per chunk so B can keep up: ```bash docker exec -i cg-watch python - 0.6 < alloc.py echo "exit: $?" ``` 15. **Predict**: what does B show at the moment of the kill, and what does it show one second later? 16. **Observe** — A: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB ⋮ allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB exit: 137 ``` and B, which redraws one line in place about once a second. The successive values it showed: ``` cg-watch 524KiB / 50MiB 1.02% cg-watch 9.215MiB / 50MiB 18.43% cg-watch 14.01MiB / 50MiB 28.02% cg-watch 24.04MiB / 50MiB 48.07% cg-watch 34.07MiB / 50MiB 68.13% cg-watch 39.08MiB / 50MiB 78.16% cg-watch 49.3MiB / 50MiB 98.59% cg-watch 608KiB / 50MiB 1.19% ``` The climb is visible right up to **98.59%**, and the next sample is **1.19%**. There is no intermediate state — no "shutting down", no "cleaning up". One second the memory is charged, the next it isn't, because the process holding it ceased to exist between samples. (Which sample lands closest to 100% is luck of the polling interval; the cliff to ~1% is not.) 17. **Terminal A**: ask the kernel what it did. ```bash docker exec cg-watch cat /sys/fs/cgroup/memory.events docker inspect cg-watch --format 'Running={{.State.Running}} OOMKilled={{.State.OOMKilled}}' ``` 18. **Observe**: ``` low 0 high 0 max 37 oom 1 oom_kill 1 oom_group_kill 0 Running=true OOMKilled=true ``` Three numbers worth reading carefully: - `max 37` — the cgroup hit `memory.max` **thirty-seven times** and survived, by reclaiming. The kill was not the first time it touched the wall; it was the first time reclaim came up empty. (The count lands in the thirties run to run; what matters is that it is dozens, not one.) - `oom 1` — the OOM killer was invoked once. - `oom_kill 1` — it killed exactly one task. And `Running=true OOMKilled=true`: the container is alive (the `sleep` was never touched), but Docker has flagged it. The OOM killer picks a victim within the cgroup by score, and a `sleep` holding 500 KB is a poor candidate next to a Python process holding 49 MB. In a real container this is the failure mode where PID 1 survives and a worker vanishes — the container looks healthy and half of it is dead. 19. **Terminal A** (optional): read the kernel's own account of it. `dmesg` is the whole VM's ring buffer, shared by every container on the daemon, so filter it to your own cgroup rather than reading the tail — the cgroup id is in the `oom_memcg=` field of every memcg OOM report: ```bash CID=$(docker inspect --format '{{.Id}}' cg-ddkill) docker run --rm --privileged --name cg-dmesg alpine:3.20 dmesg \ | grep -B4 -A1 "oom_memcg=/docker/$CID" ``` ``` [125629.069168] Tasks state (memory values in pages): [125629.069168] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [125629.069182] [ 27269] 0 27269 427 204 0 204 0 45056 0 0 sleep [125629.069185] [ 27294] 0 27294 683 476 256 220 0 40960 0 0 dd [125629.069187] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,mems_allowed=0,oom_memcg=/docker/1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,task_memcg=/docker/1288b2d31596a4351f2de0e34d33df7d207aeda5f5a36f51161c2a6c1af505ff,task=dd,pid=27294,uid=0 [125629.069257] Memory cgroup out of memory: Killed process 27294 (dd) total-vm:2732kB, anon-rss:1024kB, file-rss:880kB, shmem-rss:0kB, UID:0 pgtables:40kB oom_score_adj:0 ``` (This capture is of the Part 5 `dd` kill rather than the Python one — same report, and the numbers in it are the point of Part 5, so run it after step 29. Swap `cg-ddkill` for `cg-watch` to see the Python one.) `constraint=CONSTRAINT_MEMCG` is the kernel saying this was a *cgroup* OOM, not the host running out of memory, and `oom_memcg=/docker/1288b…` names the container. The scoring table above it lists every task that was a candidate. ### Part 5 — page cache counts, and `docker stats` lies about it What this part tests: the accounting caveat from the mental model — `memory.current` is not RSS. This is the part most likely to break your intuition, so take the predictions seriously. 20. **Terminal A**: a fresh container, and write a 40 MB file into it. `dd` with `bs=1M` allocates a *one megabyte* buffer. ```bash docker run -d --name cg-cache --memory=50m --memory-swap=50m python:3-alpine sleep 600 docker exec cg-cache sh -c ' dd if=/dev/urandom of=/big bs=1M count=40 sync echo memory.current=$(cat /sys/fs/cgroup/memory.current) grep -E "^(anon|file|inactive_file) " /sys/fs/cgroup/memory.stat' ``` 21. **Predict**: `dd` allocated 1 MB. What is `memory.current` now, out of a 50 MB budget? And what will `docker stats` say? 22. **Observe**: ``` 40+0 records in 40+0 records out 41943040 bytes (40.0MB) copied, 0.098399 seconds, 406.5MB/s memory.current=44498944 anon 98304 file 41955328 inactive_file 41955328 ``` `memory.current` is **44498944 — 42.4 MiB, or 85% of the limit** — for a container running `sleep` and a `dd` that allocated 1 MB. `anon` is `98304`, ninety-six kilobytes: essentially none of it is anybody's heap. `file 41955328` is the 40 MB file to within a few pages. The page cache is charged to the cgroup that faulted it in, and there is no distinction in `memory.max` between "cache" and "heap". 23. **Terminal B**: ask Docker. ```bash docker stats --no-stream --format '{{.Name}} {{.MemUsage}} {{.MemPerc}}' cg-cache ``` 24. **Observe**: ``` cg-cache 1.719MiB / 50MiB 3.44% ``` **3.44%.** The kernel says 85%, Docker says 3%. Neither is lying: Docker subtracts `inactive_file` from `memory.current` before reporting, on the theory that reclaimable cache isn't "really" used — and here `inactive_file` is `41955328`, essentially the whole charge. That is a defensible dashboard choice and a terrible debugging one, because the number `memory.max` is compared against is the *unsubtracted* one. If you are chasing an OOM kill, `docker stats` is the wrong instrument; read `memory.current`. 25. **Predict**: `memory.current` is at 42 MB of 50. Now allocate 30 MB of anonymous memory in the same container. There is nowhere near 30 MB free. Does it get killed? 26. **Terminal A**: ```bash docker exec cg-cache python -c " import os x = os.urandom(30*1024*1024) print('allocated 30 MB, still alive') print('memory.current =', open('/sys/fs/cgroup/memory.current').read().strip()) for l in open('/sys/fs/cgroup/memory.stat'): if l.split()[0] in ('anon','file','pgscan','pgsteal'): print(l.strip()) print(open('/sys/fs/cgroup/memory.events').read().strip()) " ``` 27. **Observe**: ``` allocated 30 MB, still alive memory.current = 52428800 anon 35160064 file 15855616 pgscan 6400 pgsteal 6400 low 0 high 0 max 100 oom 0 oom_kill 0 oom_group_kill 0 ``` It survived. `file` fell from 41955328 to 15855616 — the kernel threw away 25 MB of clean page cache to make room — while `anon` rose to 35160064. `pgscan 6400 / pgsteal 6400` is the reclaim: 6400 pages scanned, 6400 successfully freed, a 100% hit rate because clean file pages are free to drop. (6400 pages × 4 KiB = 25 MiB — exactly the cache that disappeared.) And `max 100, oom_kill 0`. It hit the wall a hundred times and was never killed once. **`memory.max` is not "kill at 50 MB". It's "never exceed 50 MB", and killing is only what happens when the kernel runs out of other ways to obey.** `memory.current = 52428800` is `memory.max` to the byte — the cgroup sitting exactly flat against its ceiling, working. 28. **Predict**: so if clean cache is free to reclaim, writing a *300* MB file into a 50 MB container should also be fine — `dd` still only holds 1 MB at a time. 29. **Terminal A**, in a fresh container so nothing above muddies it: ```bash docker run -d --name cg-ddkill --memory=50m --memory-swap=50m alpine:3.20 sleep 600 docker exec cg-ddkill dd if=/dev/urandom of=/big bs=1M count=300 echo "exit: $?" docker exec cg-ddkill sh -c ' ls -lh /big echo memory.current=$(cat /sys/fs/cgroup/memory.current) grep -E "^(anon|file) " /sys/fs/cgroup/memory.stat cat /sys/fs/cgroup/memory.events' ``` 30. **Observe**: ``` exit: 137 -rw-r--r-- 1 root root 47.0M Jul 25 16:08 /big memory.current=52260864 anon 167936 file 49250304 low 0 high 0 max 22 oom 1 oom_kill 1 oom_group_kill 0 ``` `dd` printed nothing — not even its usual `records in / records out` — it was **OOM-killed with an `anon` charge for the whole cgroup of 167936 bytes, 164 kilobytes.** The kernel's own report from step 19 puts it even more starkly: `Killed process 27294 (dd) total-vm:2732kB, anon-rss:1024kB`. A process whose entire resident heap is one megabyte, killed for exceeding a fifty megabyte limit. The difference from step 27 is *dirty*. Reclaiming a clean page means dropping it; reclaiming a dirty page means waiting for writeback to finish first. `dd` was producing dirty pages at the ~400 MB/s measured in step 22 and the disk could not retire them that fast, so the cgroup filled with cache the kernel was not yet allowed to drop, hit the wall 22 times trying, and then killed the only task making it worse. The file stops at 47.0 MB of the requested 300. (How far `dd` gets and how many `max` hits it takes vary with disk speed and load — 47 MB and 22 here. That it dies at all, holding a 1 MB buffer, does not.) This is the shape of a whole class of production incidents: a container that is fine all day and dies during the nightly export, backup, or log rotation. Nothing leaked. It wrote a file. ### Part 6 — the same number as a soft limit What this part tests: the hard/soft distinction from the mental model, with everything else held constant. `memory.high` is the same 50 MB threshold, enforced by throttling instead of killing. Docker doesn't expose `memory.high`, so this needs `--privileged` to remount the cgroup filesystem read-write and set the file by hand. That is a lab technique, not a deployment pattern — in the real world this is systemd's `MemoryHigh=`. 31. **Predict**: `memory.high = 50M` with `memory.max` left at `max`. The same script wants 200 MB. What happens? 32. **Terminal A**: ```bash docker run --rm -i --privileged --name cg-soft python:3-alpine sh -c ' mount -o remount,rw /sys/fs/cgroup echo 50M > /sys/fs/cgroup/memory.high python - echo "python exit: $?" cat /sys/fs/cgroup/memory.events' < alloc.py ``` 33. **Observe**: ``` allocated 5 MB memory.current = 9 MB memory.swap.current = 0 MB ⋮ allocated 45 MB memory.current = 49 MB memory.swap.current = 0 MB allocated 50 MB memory.current = 50 MB memory.swap.current = 17 MB ⋮ allocated 95 MB memory.current = 49 MB memory.swap.current = 53 MB ⋮ allocated 140 MB memory.current = 49 MB memory.swap.current = 98 MB ⋮ allocated 195 MB memory.current = 49 MB memory.swap.current = 157 MB allocated 200 MB memory.current = 49 MB memory.swap.current = 162 MB done: allocated 200 MB, never killed python exit: 0 low 0 high 159 max 0 oom 0 oom_kill 0 oom_group_kill 0 ``` (Trimmed — the run prints all forty lines; `memory.current` never leaves the 43–50 range after the tenth.) Same program, same 50 MB number, opposite outcome. All 200 MB allocated, **exit 0**, and `memory.current` pinned in the high forties for the entire run — the ceiling was honoured just as strictly as `memory.max` honoured it. The difference is what happened at the ceiling: `memory.swap.current` climbed to 162 MB, three times the RAM limit, because with no `memory.max` there was nothing to cap it. And the tally is the mirror image of Part 4: | | `memory.max = 50M` | `memory.high = 50M` | | --- | --- | --- | | `max` | 37 | 0 | | `high` | 0 | 159 | | `oom_kill` | 1 | 0 | | exit code | 137 | 0 | | allocated before stopping | 45 MB | 200 MB | `high 159` counts the times the kernel throttled the allocating task — put it to sleep to let reclaim catch up. That is the entire mechanism: the same threshold, applied as backpressure rather than as a wall. ## What you should see A container that reads its own limit out of `/sys/fs/cgroup/memory.max` as `52428800` while `free -m` insists it has 15973 MB. An allocator that gets 90 MB into a 50 MB container before dying, because `--memory=50m` silently grants 50 MB of swap as well — and `memory.current` that flattens at 49 and never once exceeds it. With swap off, a clean kill at 45 MB with no traceback, exit `137`, `OOMKilled=true`. `docker stats` climbing to `98.59%` and reading `1.19%` one second later. `memory.events` reporting `max 37, oom_kill 1` — thirty-seven survived collisions before the fatal one. A `dd` with a 1 MB buffer pushing `memory.current` to 85% of the limit while `docker stats` reports `3.44%`, and then getting SIGKILLed at `anon-rss:1024kB`. And the same 50 MB expressed as `memory.high` letting the identical script finish all 200 MB with `high 159, oom_kill 0, exit 0`. ## Why Because the limit is enforced at page-fault time, in the kernel, against a counter the kernel maintains — and there is no path from there back into your program. When a fault would push `memory.current` past `memory.max`, the kernel's options are to make room or to stop the cgroup growing. Making room means reclaim: drop clean page cache, write back dirty pages, swap out anonymous ones. That succeeded 37 times in Part 4 and 100 times in Part 5, invisibly, which is why `max` counts in the dozens and hundreds on a container that is merely *busy*. Only when reclaim returns empty-handed does the second option apply, and the kernel cannot implement it by failing an allocation. The allocation already succeeded — `os.urandom` returned, `malloc` returned, the page table entry exists. What's failing is the *fault*, on memory the process already believes it owns. Handing that back as an error would mean inventing a failure at an instruction that has no error path. So the kernel removes the process instead, with the one signal that cannot be caught, and your program learns nothing because there is nobody left to tell. That is also why `memory.high` can exist at all. Throttling is something you can do *to* a running task — sleep it, let reclaim catch up, resume it — without ever needing an error path. It's strictly more graceful, it's what you actually want for batch work, and Docker exposes no flag for it. And the page cache results follow from the same accounting rather than from any special rule. Charging cache to the cgroup that faulted it in is the only choice that makes limits meaningful — otherwise any container could pin unlimited host memory by reading files. The consequence is that `memory.current` is a measure of pages the cgroup is responsible for, not pages its processes allocated, and the OOM killer's victim is chosen by RSS among tasks that may have had nothing to do with the pressure. `dd` at `anon-rss:1024kB` was not the cause of the problem. It was just the biggest thing in the room. **Where the effect vanishes.** Give the cgroup swap it can actually use and the wall stops being a wall for anonymous memory — Part 2's container reached 90 MB, Part 6's reached 200. Make the pressure entirely *clean* page cache and the kill vanishes too: step 27 hit the limit a hundred times without a scratch, because every page the kernel wanted was droppable. The kill needs both conditions at once — nowhere to put the pages, and pages that can't be thrown away. ## Go deeper - Drop the `sync` in step 20 and re-run the 30 MB allocation immediately after the `dd`. With the 40 MB still dirty and in flight, the same allocation that survived at step 27 gets `exit 137` instead — the same distinction as step 30, at a scale small enough to run in a second. On this machine it reproduced on all five attempts; on a faster disk the writeback may win the race, so run it a few times. - Swap the allocator's `os.urandom(5 * MB)` back to `bytearray(5 * MB)` and re-run Part 2. Zero-filled pages get special-cased in the swap path, and the container survives to 125 MB instead of 90 (three runs, 125 MB every time) — the same limit, honoured identically, with the arithmetic thrown off by page *content*. A good reminder that synthetic memory benchmarks are easy to get wrong. - Try `--oom-kill-disable` on Part 3's run and read the warning: `WARNING: Your kernel does not support OomKillDisable. OomKillDisable discarded.` The flag is a cgroup **v1** feature (`memory.oom_control`, which paused the cgroup at the limit instead of killing it). v2 removed the ability to disable the OOM killer, Docker keeps the flag for compatibility, and it silently does nothing. Worth knowing before you find it in someone's `docker-compose.yml` and assume it's protecting anything. - Run two allocators in the same container at once and watch which one the OOM killer picks, in `dmesg`'s scoring table. Then bias the choice: `echo 500 > /proc/self/oom_score_adj` in one of them makes it the preferred victim. Note the asymmetry — *raising* the score works unprivileged, but `echo -1000` (the way you protect a supervisor from its own workers) fails with `sh: write error: Permission denied` unless the container has `CAP_SYS_RESOURCE`. You may volunteer to die; you may not opt out. - Compare with the CPU side: `--cpus=0.5` is enforced by the CFS bandwidth controller, which *throttles* rather than kills, because a process can always be given less CPU later but cannot be given back memory it already wrote to. Same cgroup, two very different enforcement stories. - Read the kernel's own documentation for these files: https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files — in particular the `memory.high` section, which spells out the throttling behaviour Part 6 counts. ## Cleanup ```bash docker rm -f cg-watch cg-cache cg-ddkill cg-swap cg-hard ``` (`cg-see`, `cg-nolimit`, `cg-soft` and `cg-dmesg` were all `--rm` and are already gone. The allocator lives in `~/cgroup-lab/alloc.py`; delete the directory if you're done.)