Union Filesystem Layers

Delete a 50 MB file and a credentials file in the last RUN of your Dockerfile — then read the credentials back out of the shipped image, and find the 50 MB still sitting on your disk. An image is a stack of immutable diffs; a container is one thin layer on top.

Concept

A Docker image is not a disk image. It's a stack of tarballs — one per build step — and the filesystem your container sees is a union mount that overlays them, topmost wins. Every layer is immutable and content-addressed, so unrelated images share the ones they have in common. A running container adds exactly one thin writable layer on top, which is why docker diff can tell you what changed and why two containers from the same image cost almost nothing extra.

This exercise makes all of that literal. You'll build an image whose last RUN deletes a 50 MB file and a credentials file — and then read that credentials file back out of the shipped image. You'll watch a two-byte append cost 4.6 MB. And you'll end up looking at the actual kernel overlay mount, with its six lowerdir= paths and a character device standing in for a deleted file.

Provenance

Docker Engine docs, Storage drivers — sections Images and layers, Container and layers, Container size on disk, and Sharing promotes smaller images:

"When you create a new container, you add a new writable layer on top of the underlying layers. This layer is often called the 'container layer'."

"Shared image layers are only stored once in /var/lib/docker/ and are also shared when pushing and pulling an image to an image registry."

On docker ps -s: size is "the amount of data (on disk) that's used for the writable layer of each container"; virtual size is "the amount of data used for the read-only image data used by the container plus the container's writable layer 'size'."

Docker Engine docs, OverlayFS storage driver — sections How the overlay2 driver works and How container reads and writes work with overlay2. This is the sentence Parts 2, 4 and 5 exist to make you look at:

"When a file is deleted within a container, a whiteout file is created in the container (upperdir). The version of the file in the image layer (lowerdir) is not deleted (because the lowerdir is read-only). However, the whiteout file prevents it from being available to the container."

And for copy-up, from the same page:

"The first time a container writes to an existing file, that file does not exist in the container (upperdir). The overlay2 driver performs a copy_up operation to copy the file from the image (lowerdir) to the container (upperdir)."

The on-disk form of a whiteout is the kernel's, not Docker's — Linux kernel documentation, Overlay Filesystem, section whiteouts and opaque directories:

"A whiteout is created as a character device with 0/0 device number or as a zero-size regular file with the xattr trusted.overlay.whiteout."

The tar form is the OCI's — OCI Image Layer Specification, section Whiteouts:

"A whiteout file is an empty file with a special filename that signifies a path should be deleted. A whiteout filename consists of the prefix .wh. plus the basename of the path to be deleted."

Part 5's claim about the l/ symlink directory is from the overlay2 page's Image and container layers on-disk: the short identifiers exist "to avoid hitting the page size limitation on arguments to the mount command," and "the overlay2 driver natively supports up to 128 lower OverlayFS layers."

Prerequisites

Four pieces of background, one line each on where they're load-bearing:

• overlayfs whiteouts and copy-up — the mechanism behind Part 4's docker diff letters and every character device you find in Part 5. This one carries most of the exercise. Kernel: Overlay Filesystem (read "whiteouts and opaque directories" and "Non-directories").
• BuildKit's layer cache — Part 3's shared digests come from cache reuse, not from two builds independently producing identical bytes. Getting this backwards is the single easiest way to misread Part 3. Docker: build cache.
• Content-addressable storage — why a layer's name is a SHA-256 of its tar bytes, and why that makes "do I already have this?" a string compare. OCI Image Spec: descriptors and digests.
• tar as the layer wire format — Part 2 unpacks the shipped artifact with nothing but tar; the whiteout is an ordinary zero-byte member. OCI Image Layer Specification.

Environment

• Host: macOS 26.5.2 (build 25F84), Apple Silicon (arm64)
• Docker: Docker Desktop, Engine 29.6.2 — Linux VM kernel 6.12.76-linuxkit, 16 GiB allocated to the VM
• Alpine image: alpine:3.20 — Alpine 3.20.10, digest sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
• Captured: 2026-07-26

Storage driver overlay2, builder buildkit.dockerfile.v0. Layer digests and overlay2 directory IDs are generated per build and per machine — yours will differ, and you'll substitute your own throughout Part 5.

Mental model: a stack, a union, and a tombstone

Four terms carry the whole exercise:

The shape you're about to see on disk:

Deleting layer 3's file from layer 1 changes the top row. It does not change row 3.

Setup

One terminal is enough. You need Docker, and roughly 150 MB of free image space (the exercise ends up with four images). Everything here is tagged layers-demo:* and named layers-* so you can clean up precisely at the end.

Work in a scratch directory, not in a repo:

mkdir -p /tmp/layers && cd /tmp/layers

Create Dockerfile. Note that each RUN is deliberately its own step, and the last one is the "cleanup" line you've seen in a hundred real Dockerfiles:

FROM alpine:3.20

RUN echo "build-id=1" > /etc/app-release

RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null

RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials

RUN rm -f /opt/build-cache.bin /root/.aws-credentials

CMD ["sleep", "3600"]

/dev/urandom rather than zeros on purpose: incompressible bytes, so nothing downstream can quietly make the problem look smaller than it is.

for pat in build-id build-cache aws-credentials app-release; do
  docker buildx prune -a --force --filter "description~=$pat"
done

Part 1 — one instruction, one layer

What this part tests: the claim that the image is a stack of per-step diffs, not a single blob. If that's true, docker history should show one row per instruction with a size attached, and those sizes should add up.

docker build -t layers-demo:leaky .
docker history layers-demo:leaky

docker image inspect layers-demo:leaky --format '{{.Size}}'
docker image inspect alpine:3.20       --format '{{.Size}}'

docker run --rm layers-demo:leaky sh -c 'ls -la /opt; cat /root/.aws-credentials'

Part 2 — the file you deleted is still in the image

What this part tests: the mental model's headline — a whiteout hides, it doesn't remove. The union view above is honest; it just isn't the whole artifact. docker save gives you the artifact, exactly as a registry would receive it.

mkdir -p save && docker save layers-demo:leaky -o save/leaky.tar
cd save && tar -xf leaky.tar
for f in blobs/sha256/*; do
  if tar -tf "$f" 2>/dev/null | grep -q 'aws-credentials'; then
    echo "FOUND IN: $f"
    tar -xOf "$f" root/.aws-credentials 2>/dev/null
  fi
done

tar -tvf blobs/sha256/fe77f750...   # the layer that created the files
tar -tvf blobs/sha256/38df108f...   # the layer that "deleted" them

FROM alpine:3.20

RUN echo "build-id=1" > /etc/app-release

RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null \
 && echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials \
 && rm -f /opt/build-cache.bin /root/.aws-credentials

CMD ["sleep", "3600"]

docker build -f Dockerfile.fixed -t layers-demo:squashed .
docker image inspect layers-demo:squashed --format '{{.Size}}'

Part 3 — layers are content-addressed, and therefore shared

What this part tests: the claim that a layer is named by its contents, so Docker stores one copy of a layer no matter how many images list it.

docker image inspect layers-demo:leaky    --format '{{json .RootFS.Layers}}'
docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}'

docker image inspect alpine:3.20 --format '{{json .RootFS.Layers}}'

docker build -t layers-demo:leaky .           # unchanged, for contrast
sed -i '' 's/build-id=1/build-id=2/' Dockerfile
docker build --progress=plain -t layers-demo:rebuilt .

docker build --no-cache -f Dockerfile.fixed -t layers-demo:nocache .
docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}'
docker image inspect layers-demo:nocache  --format '{{json .RootFS.Layers}}'

docker system df -v | grep -E 'SHARED SIZE|^layers-demo'

Part 4 — the writable layer is genuinely thin

What this part tests: that a running container adds only a diff. Per the mental model, the upperdir starts empty and gains one entry per file you touch — which is exactly the information docker diff reports.

docker run -d --name layers-box layers-demo:leaky sleep 3600
docker diff layers-box

docker exec layers-box sh -c '
  echo "build-id=2" > /etc/app-release
  mkdir -p /srv/data && echo hello > /srv/data/new.txt
  rm /etc/motd'
docker diff layers-box

docker ps -s --filter name=layers-box --format 'table {{.Names}}\t{{.Size}}'

docker run -d --name layers-box2 layers-demo:leaky sleep 3600
docker exec layers-box2 cat /etc/app-release
docker ps -s --filter name=layers-box --format '{{.Names}} {{.Size}}'

docker exec layers-box2 sh -c 'ls -l /lib/libcrypto.so.3'
docker exec layers-box2 sh -c 'echo x >> /lib/libcrypto.so.3'
docker ps -s --filter name=layers-box2 --format '{{.Names}} {{.Size}}'
docker diff layers-box2

Part 5 — look at the actual mount

What this part tests: everything above, at the level the kernel sees it. docker inspect will hand you the real paths.

docker inspect layers-box --format '{{json .GraphDriver}}'

docker inspect layers-box2 --format '{{.GraphDriver.Data.LowerDir}}' | tr ':' '\n'

docker run --rm --privileged --pid=host alpine:3.20 \
  nsenter -t 1 -m -- <command>

docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
  find /var/lib/docker/overlay2/1d5c8c44b19a8b…/diff -exec ls -ld {} +

docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
  sh -c 'grep 1d5c8c44 /proc/mounts'

docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- sh -c \
  'for d in s0boitxkt0lcv2ltxa1p6108u td38xshhvgkyrnryq0z1fhq8r \
            uqvao75hp68davd9jea3taw7m bsf07qsb707txc2nz2x55y0h2; do
     echo "=== $d ==="
     find /var/lib/docker/overlay2/$d/diff -exec ls -ld {} +
   done'

docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
  cat /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff/root/.aws-credentials

What you should see

A docker history whose per-layer sizes sum to exactly the image size (8,822,697 + 11 + 52,428,800 + 21 + 0 = 61,251,529), with the RUN rm step contributing 0B. A container that agrees the deleted files are gone, and a docker save tarball that hands you AKIAEXAMPLESECRETKEY anyway, alongside two zero-byte .wh. tombstones. The same build with one && instead of three RUNs at 8,822,708 bytes — 6.9× smaller for identical behavior. Two images agreeing on their first two layer digests character-for-character, docker system df -v reporting :squashed at 8.82MB size and 0B unique size, and a one-character edit invalidating every layer above it. A --no-cache rebuild of that same RUN producing a different digest for an identical 11-byte file, which is how you learn the match was cache reuse rather than reproducibility. A running container whose writable layer is 17B (virtual 61.3MB), a second one at 0B, and a two-byte append that costs 4.6 MB. And finally a real overlay line in /proc/mounts with six lowerdirs, a character device at …/diff/etc/motd, and a 52,428,800-byte file alive in a layer directory that nothing can reach.

Why

Because the layer is the unit of transfer, and immutability is what makes transfer cheap. If a layer could be edited, its digest would change, every image referencing it would need updating, and the "already have that one" check that makes docker pull fast would have nothing stable to compare. So a layer is frozen the moment it's written, named by its contents, and shared by anyone whose stack happens to contain the same bytes. Everything else follows from refusing to break that.

Deletion is where the cost shows up. You cannot remove a file from a read-only layer, so the only honest thing an upper layer can do is record "stop looking below this path" — a whiteout. The union honors it, ls honors it, your application honors it. The registry does not, because the registry ships layers, not the union. RUN rm on a later line therefore buys you nothing but two tombstones, and this is not a Docker bug to be fixed; it is the direct price of the property that makes images shareable at all. The fix is always to arrange for the bytes never to be committed to a layer in the first place — one RUN, a multi-stage build, or a build mount.

Copy-up is the same trade in the other direction. Making an image layer writable per-container would mean copying it per-container, which would undo the sharing. So the layer stays read-only and the copy happens lazily, per file, on first write — near-zero cost for a container that mostly reads, and a full file copy the instant it isn't. 17B (virtual 61.3MB) and 4.6MB for two bytes are the two ends of that one mechanism.

Where the effect vanishes. All of this is a property of the union, not of containers. Mount a volume or a bind mount over a path and that subtree is no longer part of the overlay: no copy-up, no whiteouts, no contribution to docker diff, and no growth in the writable layer no matter how much you write. That is why the copy-up surprise never bites a database that keeps its data directory on a volume — and always bites one that doesn't.

Go deeper

• Rebuild :leaky as a multi-stage build — do the dd and the credentials work in a FROM alpine AS build stage, then FROM alpine and COPY --from=build only what you want. Check RootFS.Layers: the discarded stage contributes no layers at all, because it was never part of the final image's stack. This is the actual answer to Part 2's problem.
• Try RUN --mount=type=secret,id=creds cat /run/secrets/creds with docker build --secret id=creds,src=./creds. The file is present during the step and absent from the layer — verify with the Part 2 tarball scan and confirm nothing matches.
• Add WORKDIR, ENV, LABEL and EXPOSE lines and re-read docker history. All 0B: they're recorded in the image config (docker inspect --format '{{json .Config}}'), which is itself a blob in the tarball. The config plus the ordered layer list is the entire image — there is no other file.
• Write a file to a -v volume from inside layers-box and re-run docker diff. It doesn't appear, because a volume is a bind mount over the union, not part of it — which is the whole reason volume data survives docker rm.
• Get the shared/unique split in exact bytes rather than the rounded column, still scoped to your own images:

  sock=$(docker context inspect --format '{{.Endpoints.docker.Host}}' | sed 's|unix://||')
  curl -s --unix-socket "$sock" http://localhost/v1.45/system/df | python3 -c '
  import json, sys
  for i in json.load(sys.stdin)["Images"]:
      for t in i.get("RepoTags") or []:
          if t.startswith("layers-demo"):
              print(t, i["Size"], i["SharedSize"], i["Size"] - i["SharedSize"])'

Sources: Docker Docs: Storage drivers · Docker Docs: OverlayFS storage driver · Linux kernel: Overlay Filesystem — its "whiteouts and opaque directories" section defines the 0,0 character device you found · OCI Image Layer Specification for the .wh. tar convention

Cleanup

docker rm -f layers-box layers-box2
docker rmi layers-demo:leaky layers-demo:squashed layers-demo:rebuilt layers-demo:nocache
rm -rf /tmp/layers

The build cache outlives the images. Drop just this exercise's entries — the description filter keeps everyone else's cache intact:

for pat in build-id build-cache aws-credentials app-release; do
  docker buildx prune -a --force --filter "description~=$pat"
done

Run it twice, for the same reason as in Setup, then confirm nothing of yours is left and everyone else's cache still is:

docker buildx du --verbose --filter 'description~=build-cache' | grep -c '^ID:'   # → 0
docker buildx du | tail -2