# Docker: union filesystem layers

## Concept

A Docker image is not a disk image. It's a *stack of tarballs* — one per
build step — and the filesystem your container sees is a union mount that
overlays them, topmost wins. Every layer is immutable and
content-addressed, so unrelated images share the ones they have in common.
A running container adds exactly one thin writable layer on top, which is
why `docker diff` can tell you what changed and why two containers from
the same image cost almost nothing extra.

This exercise makes all of that literal. You'll build an image whose last
`RUN` deletes a 50 MB file and a credentials file — and then read that
credentials file back out of the shipped image. You'll watch a two-byte
append cost 4.6 MB. And you'll end up looking at the actual kernel
`overlay` mount, with its six `lowerdir=` paths and a character device
standing in for a deleted file.

## Provenance

Docker Engine docs, **[Storage drivers](https://docs.docker.com/engine/storage/drivers/)**
— sections *Images and layers*, *Container and layers*, *Container size on
disk*, and *Sharing promotes smaller images*:

> "When you create a new container, you add a new writable layer on top of
> the underlying layers. This layer is often called the 'container layer'."
>
> "Shared image layers are only stored once in `/var/lib/docker/` and are
> also shared when pushing and pulling an image to an image registry."
>
> On `docker ps -s`: *size* is "the amount of data (on disk) that's used for
> the writable layer of each container"; *virtual size* is "the amount of
> data used for the read-only image data used by the container plus the
> container's writable layer 'size'."

Docker Engine docs, **[OverlayFS storage driver](https://docs.docker.com/engine/storage/drivers/overlayfs-driver/)**
— sections *How the `overlay2` driver works* and *How container reads and
writes work with `overlay2`*. This is the sentence Parts 2, 4 and 5 exist
to make you look at:

> "When a *file* is deleted within a container, a *whiteout* file is created
> in the container (`upperdir`). The version of the file in the image layer
> (`lowerdir`) is not deleted (because the `lowerdir` is read-only). However,
> the whiteout file prevents it from being available to the container."

And for copy-up, from the same page:

> "The first time a container writes to an existing file, that file does not
> exist in the container (`upperdir`). The `overlay2` driver performs a
> `copy_up` operation to copy the file from the image (`lowerdir`) to the
> container (`upperdir`)."

The on-disk *form* of a whiteout is the kernel's, not Docker's — Linux
kernel documentation, **[Overlay Filesystem](https://docs.kernel.org/filesystems/overlayfs.html)**,
section *whiteouts and opaque directories*:

> "A whiteout is created as a character device with 0/0 device number or as
> a zero-size regular file with the xattr `trusted.overlay.whiteout`."

The *tar* form is the OCI's — **[OCI Image Layer Specification](https://github.com/opencontainers/image-spec/blob/main/layer.md)**,
section *Whiteouts*:

> "A whiteout file is an empty file with a special filename that signifies a
> path should be deleted. A whiteout filename consists of the prefix `.wh.`
> plus the basename of the path to be deleted."

Part 5's claim about the `l/` symlink directory is from the overlay2 page's
*Image and container layers on-disk*: the short identifiers exist "to avoid
hitting the page size limitation on arguments to the `mount` command," and
"the `overlay2` driver natively supports up to 128 lower OverlayFS layers."

## Prerequisites

Four pieces of background, one line each on where they're load-bearing:

- **overlayfs whiteouts and copy-up** — the mechanism behind Part 4's
  `docker diff` letters and every character device you find in Part 5.
  *This one carries most of the exercise.*
  [Kernel: Overlay Filesystem](https://docs.kernel.org/filesystems/overlayfs.html)
  (read "whiteouts and opaque directories" and "Non-directories").
- **BuildKit's layer cache** — Part 3's shared digests come from cache
  *reuse*, not from two builds independently producing identical bytes.
  Getting this backwards is the single easiest way to misread Part 3.
  [Docker: build cache](https://docs.docker.com/build/cache/).
- **Content-addressable storage** — why a layer's name is a SHA-256 of its
  tar bytes, and why that makes "do I already have this?" a string compare.
  [OCI Image Spec: descriptors and digests](https://github.com/opencontainers/image-spec/blob/main/descriptor.md).
- **tar as the layer wire format** — Part 2 unpacks the shipped artifact
  with nothing but `tar`; the whiteout is an ordinary zero-byte member.
  [OCI Image Layer Specification](https://github.com/opencontainers/image-spec/blob/main/layer.md).

## Environment

- **Host:** macOS 26.5.2 (build 25F84), Apple Silicon (arm64)
- **Docker:** Docker Desktop, Engine 29.6.2 — Linux VM kernel `6.12.76-linuxkit`, 16 GiB allocated to the VM
- **Alpine image:** `alpine:3.20` — Alpine 3.20.10, digest `sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc`
- **Captured:** 2026-07-26

Storage driver `overlay2`, builder `buildkit.dockerfile.v0`. Layer digests
and `overlay2` directory IDs are generated per build and per machine —
yours will differ, and you'll substitute your own throughout Part 5.

## Mental model: a stack, a union, and a tombstone

Four terms carry the whole exercise:

| Term | What it actually is |
| --- | --- |
| **Layer** | A tar archive of the filesystem *changes* one build step made — files added or modified, plus tombstones for files removed. Immutable once written, and named by the SHA-256 of its contents (its *diff ID*), which is what makes sharing possible. |
| **Union / overlay mount** | The kernel stacking those layers into one directory tree. In overlay2's vocabulary: the read-only image layers are `lowerdir`s, the container's writable layer is the `upperdir`, and the combined view is the `merged` directory — which is what becomes the container's `/`. |
| **Whiteout** | How you delete a file you don't own. Since lower layers are read-only, an upper layer records a marker at that path — a character device with major/minor `0,0` on disk, or a `.wh.<name>` entry inside a layer tarball — and the union hides everything below it. |
| **Copy-up** | How you *modify* a file you don't own. On first write, overlayfs copies the entire file from the lower layer into the upper layer, then edits the copy. Costs are per-file, not per-byte-changed. |

The consequence that costs people real money:

> **A later layer can hide a file, but it can never remove it.** The bytes
> stay in the earlier layer forever, and that layer still ships. `RUN rm`
> in its own step reduces your image size by zero and reduces the
> recoverability of your secret by zero.

The shape you're about to see on disk:

```
   container /  ← what your process sees (the "merged" view)
  ═══════════════════════════════════════════
   upperdir     writable layer   [C /etc/app-release] [A /srv/data] [D /etc/motd]
  ───────────────────────────────────────────  ← container starts here
   lowerdir 1   RUN rm ...       whiteouts only, 0 B
   lowerdir 2   RUN echo creds   /root/.aws-credentials, 21 B
   lowerdir 3   RUN dd 50MB      /opt/build-cache.bin, 52,428,800 B
   lowerdir 4   RUN echo build   /etc/app-release, 11 B
   lowerdir 5   FROM alpine:3.20 the whole distro, 8,822,697 B
```

Deleting layer 3's file from layer 1 changes the top row. It does not
change row 3.

## Setup

One terminal is enough. You need Docker, and roughly 150 MB of free image
space (the exercise ends up with four images). Everything here is tagged
`layers-demo:*` and named `layers-*` so you can clean up precisely at the
end.

Work in a scratch directory, not in a repo:

```bash
mkdir -p /tmp/layers && cd /tmp/layers
```

Create `Dockerfile`. Note that each `RUN` is deliberately its own step,
and the last one is the "cleanup" line you've seen in a hundred real
Dockerfiles:

```dockerfile
FROM alpine:3.20

RUN echo "build-id=1" > /etc/app-release

RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null

RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials

RUN rm -f /opt/build-cache.bin /root/.aws-credentials

CMD ["sleep", "3600"]
```

`/dev/urandom` rather than zeros on purpose: incompressible bytes, so
nothing downstream can quietly make the problem look smaller than it is.

**Part 3 assumes you have never built this Dockerfile before.** If you
have, BuildKit already holds several cache entries for these exact `RUN`
lines and Steps 15 and 18 will pick an arbitrary one. Clear just this exercise's
entries first — never a blanket `prune`, which takes everyone else's cache
with it:

```bash
for pat in build-id build-cache aws-credentials app-release; do
  docker buildx prune -a --force --filter "description~=$pat"
done
```

Run that loop twice: freeing a record can make its parent prunable, so the
first pass usually leaves a few behind.

## Steps

### Part 1 — one instruction, one layer

What this part tests: the claim that the image is a stack of per-step
diffs, not a single blob. If that's true, `docker history` should show one
row per instruction with a size attached, and those sizes should add up.

1. **Predict**: the Dockerfile has five instructions after `FROM`. How
   many rows will `docker history` show, and what size will the last
   `RUN` (the `rm`) report?
2. Build and look:

   ```bash
   docker build -t layers-demo:leaky .
   docker history layers-demo:leaky
   ```

3. **Observe**:

   ```
   IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
   ad1aa751238f   7 seconds ago   CMD ["sleep" "3600"]                            0B        buildkit.dockerfile.v0
   <missing>      7 seconds ago   RUN /bin/sh -c rm -f /opt/build-cache.bin /r…   0B        buildkit.dockerfile.v0
   <missing>      8 seconds ago   RUN /bin/sh -c echo "AKIAEXAMPLESECRETKEY" >…   21B       buildkit.dockerfile.v0
   <missing>      8 seconds ago   RUN /bin/sh -c dd if=/dev/urandom of=/opt/bu…   52.4MB    buildkit.dockerfile.v0
   <missing>      9 seconds ago   RUN /bin/sh -c echo "build-id=1" > /etc/app-…   11B       buildkit.dockerfile.v0
   <missing>      3 months ago    CMD ["/bin/sh"]                                 0B        buildkit.dockerfile.v0
   <missing>      3 months ago    ADD alpine-minirootfs-3.20.10-aarch64.tar.gz…   8.82MB    buildkit.dockerfile.v0
   ```

   Seven rows: your five plus the two the base image contributed — history
   is inherited, so you're reading `alpine`'s build steps at the bottom.
   `CMD` rows are `0B` because they change metadata, not files. And the
   `rm` step is `0B` too, which is the first hint of the whole lesson.

4. Now check the arithmetic:

   ```bash
   docker image inspect layers-demo:leaky --format '{{.Size}}'
   docker image inspect alpine:3.20       --format '{{.Size}}'
   ```

   → `61251529` and `8822697`

   The layer sizes are `8,822,697 + 11 + 52,428,800 + 21 + 0`, which is
   `61,251,529` exactly. The image's size *is* the sum of its layers'
   diffs; there's nothing else in there.

5. **Predict**: the last `RUN` deleted both files. From inside a container,
   is `/opt/build-cache.bin` there?
6. Look at the union view:

   ```bash
   docker run --rm layers-demo:leaky sh -c 'ls -la /opt; cat /root/.aws-credentials'
   ```

7. **Observe**:

   ```
   total 8
   drwxr-xr-x    1 root     root          4096 Jul 25 16:14 .
   drwxr-xr-x    1 root     root          4096 Jul 25 16:14 ..
   cat: can't open '/root/.aws-credentials': No such file or directory
   ```

   Both gone, exactly as a filesystem should behave. The image is still
   61 MB.

### Part 2 — the file you deleted is still in the image

What this part tests: the mental model's headline — a whiteout hides, it
doesn't remove. The union view above is honest; it just isn't the whole
artifact. `docker save` gives you the artifact.

8. **Predict**: `docker save` writes the image exactly as it would be
   pushed to a registry. Can you get `AKIAEXAMPLESECRETKEY` out of it?
9. Unpack it and grep the layers:

   ```bash
   mkdir -p save && docker save layers-demo:leaky -o save/leaky.tar
   cd save && tar -xf leaky.tar
   for f in blobs/sha256/*; do
     if tar -tf "$f" 2>/dev/null | grep -q 'aws-credentials'; then
       echo "FOUND IN: $f"
       tar -xOf "$f" root/.aws-credentials 2>/dev/null
     fi
   done
   ```

10. **Observe**:

    ```
    FOUND IN: blobs/sha256/38df108fbea373e8ed78e4e686c365fd7bf0bb15950d2a51b410865f95c26824
    FOUND IN: blobs/sha256/fe77f750c34353660716bbcb0ad108ecc8c1300457a00b7e44a487a50cde1a1f
    AKIAEXAMPLESECRETKEY
    ```

    Two layers matched the *name*; only one has the file, and the loop
    walks `blobs/sha256/*` in digest order, so which of the two prints
    first is luck — here the empty one did, which is why the secret shows
    up under the *second* `FOUND IN` line. The credential is plaintext in
    a layer that ships with the image, retrievable by anyone who can pull
    it — no exploit, no privilege, just `tar`.

11. Look at what the two matching layers actually contain (substitute your
    own digests):

    ```bash
    tar -tvf blobs/sha256/fe77f750...   # the layer that created the files
    tar -tvf blobs/sha256/38df108f...   # the layer that "deleted" them
    ```

12. **Observe**:

    ```
    drwxr-xr-x  0 0      0           0 Jul 26 00:14 etc/
    drwx------  0 0      0           0 Jul 26 00:14 root/
    -rw-r--r--  0 0      0          21 Jul 26 00:14 root/.aws-credentials
    ```

    ```
    drwxr-xr-x  0 0      0           0 Jul 26 00:14 etc/
    drwxr-xr-x  0 0      0           0 Jul 26 00:14 opt/
    -rw-------  0 0      0           0 Jul 26 00:14 opt/.wh.build-cache.bin
    drwx------  0 0      0           0 Jul 26 00:14 root/
    -rw-------  0 0      0           0 Jul 26 00:14 root/.wh..aws-credentials
    ```

    There it is, in plain sight: the `rm` layer contains two **zero-byte
    files whose names begin with `.wh.`**. That is what a delete is in an
    image. `rm` didn't produce a smaller image, it produced a *slightly
    larger* one — two extra tombstones on top of everything it was
    hiding.

    (`root/.wh..aws-credentials` has two dots because the original
    filename already started with one: prefix `.wh.` plus `.aws-credentials`.)

13. **Predict**: move all four commands into a single `RUN` chained with
    `&&`, so the file is created and deleted inside one step. Does the
    image get smaller?
14. Build the alternative as `Dockerfile.fixed` (back in `/tmp/layers`):

    ```dockerfile
    FROM alpine:3.20

    RUN echo "build-id=1" > /etc/app-release

    RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null \
     && echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials \
     && rm -f /opt/build-cache.bin /root/.aws-credentials

    CMD ["sleep", "3600"]
    ```

    ```bash
    docker build -f Dockerfile.fixed -t layers-demo:squashed .
    docker image inspect layers-demo:squashed --format '{{.Size}}'
    ```

15. **Observe** — the build first:

    ```
    #4 [1/3] FROM docker.io/library/alpine:3.20
    #4 DONE 0.0s
    #5 [2/3] RUN echo "build-id=1" > /etc/app-release
    #5 CACHED
    #6 [3/3] RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null  && echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials  && rm -f /opt/build-cache.bin /root/.aws-credentials
    #6 DONE 0.4s
    #7 writing image sha256:0e915405e51483e6b245c21d98ba46ac1915f887456ef00fabb1e15c57d4678e done
    #7 naming to docker.io/library/layers-demo:squashed done
    #7 DONE 0.0s
    ```

    (`#5 CACHED` — the `echo` step is character-for-character the one
    `:leaky` already ran, so BuildKit did not run it again. Remember this;
    Part 3 turns on it.)

    Then the size: `8822708`.

    | | `:leaky` (rm in its own layer) | `:squashed` (one RUN) |
    | --- | --- | --- |
    | Image size | 61,251,529 B (61.3 MB) | 8,822,708 B (8.82 MB) |
    | Layers | 5 | 3 |
    | 50 MB cache file on disk | yes, layer 3 | no |
    | Secret recoverable from image | yes | no |

    Byte-identical container filesystems, 6.9× the size. The commands that
    ran are the same, in the same order, producing the same result. The
    only difference is *where the layer boundaries fell*, and layer
    boundaries are the unit at which content becomes permanent.

    `8,822,708` is `8,822,697 + 11 + 0`: the alpine rootfs, the
    `app-release` line, and a combined step whose net diff is *nothing*.
    Its `docker history` row reads `0B` — the file it created never
    outlived the step, so not even a tombstone was needed.

### Part 3 — layers are content-addressed, and therefore shared

What this part tests: the claim that a layer is named by its contents, so
Docker stores one copy of a layer no matter how many images list it.

16. **Predict**: `:leaky` and `:squashed` are separate images built from
    separate Dockerfiles. How many of their layer digests match?
17. Compare:

    ```bash
    docker image inspect layers-demo:leaky    --format '{{json .RootFS.Layers}}'
    docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}'
    ```

18. **Observe** (one digest per line for readability; the command prints
    them as a JSON array):

    ```
    leaky:                                                              squashed:
    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f  ← same
    7b511a713a666939376b9940ccab0b4b119de3f31be772c94d0d976ad95808d4    7b511a713a666939376b9940ccab0b4b119de3f31be772c94d0d976ad95808d4  ← same
    8d85c35474bef3ae31f1d5fc960b33c66db7d3971ac674d51dc63d24453d1b19    06a204555e48d10c594ef7fa321f5725f0f696b5d013667550ce7722611bd66b
    fe77f750c34353660716bbcb0ad108ecc8c1300457a00b7e44a487a50cde1a1f
    38df108fbea373e8ed78e4e686c365fd7bf0bb15950d2a51b410865f95c26824
    ```

    Two identical strings. `88b4fba6…` is the alpine rootfs, and you can
    prove that without leaving the shell — it is the *only* layer
    `alpine:3.20` has:

    ```bash
    docker image inspect alpine:3.20 --format '{{json .RootFS.Layers}}'
    ```

    ```
    ["sha256:88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f"]
    ```

    `7b511a71…` is the `echo "build-id=1"` layer. Hold on to the question
    of *why* it matches — Step 24 answers it, and the obvious answer is
    wrong.

19. **Predict**: rebuild `:leaky` unchanged. Then change *only* the first
    `RUN` to `build-id=2` and rebuild. Which steps re-run?
20. Try both:

    ```bash
    docker build -t layers-demo:leaky .           # unchanged
    sed -i '' 's/build-id=1/build-id=2/' Dockerfile
    docker build --progress=plain -t layers-demo:rebuilt .
    ```

21. **Observe** — the unchanged build:

    ```
    #4 [1/5] FROM docker.io/library/alpine:3.20
    #4 DONE 0.0s
    #5 [2/5] RUN echo "build-id=1" > /etc/app-release
    #5 CACHED
    #6 [3/5] RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null
    #6 CACHED
    #7 [4/5] RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials
    #7 CACHED
    #8 [5/5] RUN rm -f /opt/build-cache.bin /root/.aws-credentials
    #8 CACHED
    #9 exporting to image
    #9 exporting layers done
    #9 writing image sha256:ad1aa751238fca192bfa4464394ff93b4cf0a823d18bf1dad0271ac81f4d338f done
    #9 naming to docker.io/library/layers-demo:leaky done
    #9 DONE 0.0s
    ```

    and after the one-character edit:

    ```
    #4 [1/5] FROM docker.io/library/alpine:3.20
    #4 CACHED
    #5 [2/5] RUN echo "build-id=2" > /etc/app-release
    #5 DONE 0.2s
    #6 [3/5] RUN dd if=/dev/urandom of=/opt/build-cache.bin bs=1M count=50 2>/dev/null
    #6 DONE 0.6s
    #7 [4/5] RUN echo "AKIAEXAMPLESECRETKEY" > /root/.aws-credentials
    #7 DONE 0.2s
    #8 [5/5] RUN rm -f /opt/build-cache.bin /root/.aws-credentials
    #8 DONE 0.2s
    ```

    Steps 3, 4 and 5 have identical text and identical inputs from your
    point of view, and all three re-ran. The digests confirm it — only
    the base layer survives:

    ```
    leaky:                                                              rebuilt:
    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f  ← same
    7b511a713a666939376b9940ccab0b4b119de3f31be772c94d0d976ad95808d4    f68171ac3205a7036dfbc43d4b87b4b59919a1ae7b3dc46cc28f8e609ed9f921
    8d85c35474bef3ae31f1d5fc960b33c66db7d3971ac674d51dc63d24453d1b19    05c17317ef39ef2da843a47892ab23c5bb3ad9a42216afa9941c55e2a7cb7218
    fe77f750c34353660716bbcb0ad108ecc8c1300457a00b7e44a487a50cde1a1f    1f6aea84e50d5ccb7dfed81c1b8ce2ee4cad1970d97a3f2c095cc6e7189eb145
    38df108fbea373e8ed78e4e686c365fd7bf0bb15950d2a51b410865f95c26824    0dc8bdf9228ba5eab256c19809b558383e9d1e0044820e910602f8c05f132d93
    ```

    A layer's cache key includes the layer it was built *on*. Change one,
    and every layer above it is a different stack, so none of them can be
    reused. That's the reason `COPY package.json` comes before
    `COPY . .` in every well-written Node Dockerfile: put the thing that
    changes every commit as high in the stack as you can.

22. **Predict**: back to Step 18. `:leaky` and `:squashed` agreed on
    `7b511a71…`, the `echo "build-id=1"` layer. Was that because the two
    builds *independently produced the same bytes*? Force that step to
    actually run again and find out — same base, same command, same
    resulting 11-byte file:
23. ```bash
    docker build --no-cache -f Dockerfile.fixed -t layers-demo:nocache .
    docker image inspect layers-demo:squashed --format '{{json .RootFS.Layers}}'
    docker image inspect layers-demo:nocache  --format '{{json .RootFS.Layers}}'
    ```

24. **Observe**:

    ```
    squashed:                                                           nocache:
    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f    88b4fba61c4c714a2fc173ddf7e9324a257304696830bf41ad28ecc58c11c95f  ← same
    7b511a713a666939376b9940ccab0b4b119de3f31be772c94d0d976ad95808d4    edfaed2daef14a6326f9f453bc611e49d370ffd76d1f87d194b35bfb4aae5fff
    06a204555e48d10c594ef7fa321f5725f0f696b5d013667550ce7722611bd66b    7b712b080d4b55f47b7e6ef69d6f833041ee7bc015b9c5a01cb573a6319a7c79
    ```

    **The `echo "build-id=1"` layer is now a different layer.** Same
    Dockerfile line, same base image, same 11 bytes of file content —
    different digest. `docker save` both images and list that one layer
    out of each to see the entire difference:

    ```
    drwxr-xr-x  0 0      0           0 Jul 26 00:14 etc/
    -rw-r--r--  0 0      0          11 Jul 26 00:14 etc/app-release
    ```

    ```
    drwxr-xr-x  0 0      0           0 Jul 26 00:16 etc/
    -rw-r--r--  0 0      0          11 Jul 26 00:16 etc/app-release
    ```

    Two minutes of mtime, and `tar -xO` returns `build-id=1` from both. A
    layer's digest is over its *tar bytes*, and a tar member carries its
    modification time, so re-running a step produces a layer that is
    content-identical and digest-different.

    So Step 18's match was not two builds converging on the same bytes: it
    was BuildKit recognising `(same parent, same command string)` and
    **reusing the already-committed layer without running anything** —
    look back at Step 15's build, where BuildKit printed `#5 … CACHED`
    for exactly that line. That
    distinction is the whole practical difference between "Docker builds
    are reproducible" (they are not) and "Docker builds are cacheable"
    (they are). What *is* content-addressed all the way down, with no
    cache involved, is `88b4fba6…`: every alpine-based image on your
    machine points at the one rootfs layer you pulled once.

25. **Predict**: you now have four images that between them advertise a
    lot of megabytes. How many of those bytes exist? Scope the question to
    your own images — a machine-wide total says nothing about this
    exercise:

    ```bash
    docker system df -v | grep -E 'SHARED SIZE|^layers-demo'
    ```

26. **Observe**:

    ```
    REPOSITORY                                TAG               IMAGE ID       CREATED         SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
    layers-demo                               nocache           19726d67590a   3 minutes ago   8.82MB    8.823MB       11B           0
    layers-demo                               rebuilt           fb1f8c3e4021   3 minutes ago   61.3MB    8.823MB       52.43MB       0
    layers-demo                               squashed          0e915405e514   4 minutes ago   8.82MB    8.823MB       0B            0
    layers-demo                               leaky             ad1aa751238f   5 minutes ago   61.3MB    8.823MB       52.43MB       2
    ```

    Read the `:squashed` row again: **8.82 MB of size, 0 B of unique
    size.** Every byte that image needs was already on disk for `:leaky`.
    Building it cost nothing at all.

    In exact bytes (from `docker image inspect` and the daemon's
    `/system/df` endpoint — the one-liner is under *Go deeper*):

    | | reported `SIZE` | `SHARED SIZE` | `UNIQUE SIZE` |
    | --- | --- | --- | --- |
    | `:leaky` | 61,251,529 | 8,822,708 | 52,428,821 |
    | `:squashed` | 8,822,708 | 8,822,708 | 0 |
    | `:rebuilt` | 61,251,529 | 8,822,697 | 52,428,832 |
    | `:nocache` | 8,822,708 | 8,822,697 | 11 |
    | **sum** | **140,148,474** | **35,290,810** | **104,857,664** |

    The four images advertise 140,148,474 bytes. 35,290,810 of that is
    shared bytes, counted once into every image that names them — and only
    8,822,708 *distinct* shared bytes exist on disk (the 8,822,697-byte
    alpine rootfs plus the 11-byte `app-release` layer that `:leaky` and
    `:squashed` agree on). Real occupancy is
    `104,857,664 + 8,822,708 = 113,680,372`: **26,468,102 bytes of the
    headline number were never stored.** Scale that to a fleet of fifty
    services on one base image and the sharing is the entire reason a
    registry is affordable.

### Part 4 — the writable layer is genuinely thin

What this part tests: that a running container adds only a diff. Per the
mental model, the `upperdir` starts empty and gains one entry per file you
touch — which is exactly the information `docker diff` reports.

27. Start a container and diff it before touching anything:

    ```bash
    docker run -d --name layers-box layers-demo:leaky sleep 3600
    docker diff layers-box
    ```

28. **Observe**: nothing at all. Empty output. A container that has done
    no writes has an empty writable layer — the process is running on a
    stack of read-only layers and a mount point.

29. **Predict**: modify one existing file, create one new file, delete one
    existing file. What three letters does `docker diff` report, and what
    does it say about the *directories* containing them?
30. Do it:

    ```bash
    docker exec layers-box sh -c '
      echo "build-id=2" > /etc/app-release
      mkdir -p /srv/data && echo hello > /srv/data/new.txt
      rm /etc/motd'
    docker diff layers-box
    ```

31. **Observe**:

    ```
    C /etc
    C /etc/app-release
    D /etc/motd
    C /srv
    A /srv/data
    A /srv/data/new.txt
    ```

    `A`dded, `C`hanged, `D`eleted. Note `/srv` and `/etc` are `C`, not
    `A` — they already existed in the image; the writable layer had to
    materialize its own copy of each directory to hold the new entries.
    That's copy-up applied to directories.

32. **Predict**: you wrote 17 bytes of content. What does Docker report as
    the container's size?
33. ```bash
    docker ps -s --filter name=layers-box --format 'table {{.Names}}\t{{.Size}}'
    ```

34. **Observe**:

    ```
    NAMES        SIZE
    layers-box   17B (virtual 61.3MB)
    ```

    `17B` is the writable layer — `11` for `app-release` plus `6` for
    `new.txt`; the tombstone for `/etc/motd` is free. `virtual` is that
    plus the shared image. Start a second container and it costs `0B`:

    ```bash
    docker run -d --name layers-box2 layers-demo:leaky sleep 3600
    docker exec layers-box2 cat /etc/app-release
    docker ps -s --filter name=layers-box --format '{{.Names}} {{.Size}}'
    ```

    ```
    build-id=1
    layers-box2 0B (virtual 61.3MB)
    layers-box 17B (virtual 61.3MB)
    ```

    Two containers reporting 61.3 MB each; 17 bytes of actual new disk
    between them. And `layers-box2` still sees `build-id=1` — the first
    container's edit lives in *its* upperdir, invisible to anyone else,
    because the layer they share is read-only for both.

35. **Predict**: now append two bytes to a 4.6 MB file that came from the
    image. What happens to the writable layer?
36. ```bash
    docker exec layers-box2 sh -c 'ls -l /lib/libcrypto.so.3'
    docker exec layers-box2 sh -c 'echo x >> /lib/libcrypto.so.3'
    docker ps -s --filter name=layers-box2 --format '{{.Names}} {{.Size}}'
    docker diff layers-box2
    ```

37. **Observe**:

    ```
    -rwxr-xr-x    1 root     root       4597896 Apr  9 22:19 /lib/libcrypto.so.3
    layers-box2 4.6MB (virtual 65.8MB)
    C /lib
    C /lib/libcrypto.so.3
    ```

    Two bytes written, 4.6 MB consumed. Copy-up has no notion of a partial
    file: to make one byte writable, overlayfs copies all 4,597,896 of them
    into the upper layer first. This is the mechanism behind "why did my
    container balloon overnight" — a log rotation or an in-place edit of
    large image files, not new data. It's also why databases in containers
    want a volume: a volume bypasses the union entirely.

### Part 5 — look at the actual mount

What this part tests: everything above, at the level the kernel sees it.
`docker inspect` will hand you the real paths.

38. ```bash
    docker inspect layers-box --format '{{json .GraphDriver}}'
    ```

39. **Observe** (pretty-printed; the command emits one line):

    ```
    {
        "Data": {
            "ID": "9330aac54137b4451f2c9253c753e25b07e9b7cc5230a01a89f5eb9ec0966e52",
            "LowerDir": "/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec-init/diff:/var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff:/var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff:/var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff:/var/lib/docker/overlay2/bsf07qsb707txc2nz2x55y0h2/diff:/var/lib/docker/overlay2/f9488352a1dfcbea7d209c9877e72d5cc48dd1a4baa75af80225bc9be0349fb5/diff",
            "MergedDir": "/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/merged",
            "UpperDir": "/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff",
            "WorkDir": "/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/work"
        },
        "Name": "overlay2"
    }
    ```

    Six lower directories, topmost first: a per-container `-init` layer
    (where Docker stages `/etc/hosts` and friends), then your five image
    layers in reverse order. One upper. That colon-separated string is
    passed straight to `mount -t overlay`.

    Compare with `layers-box2`:

    ```bash
    docker inspect layers-box2 --format '{{.GraphDriver.Data.LowerDir}}' | tr ':' '\n'
    ```

    ```
    /var/lib/docker/overlay2/5bf4aaa9e7ebe7fa12a1d5495ffd24c168fa4ac949801f04199d871c39583a83-init/diff
    /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff
    /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff
    /var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff
    /var/lib/docker/overlay2/bsf07qsb707txc2nz2x55y0h2/diff
    /var/lib/docker/overlay2/f9488352a1dfcbea7d209c9877e72d5cc48dd1a4baa75af80225bc9be0349fb5/diff
    ```

    The five image directories are character-for-character the same paths;
    only the `-init` and upper differ. "Sharing a layer" means literally
    mounting the same directory into two containers.

**On macOS/Windows, those paths are inside the Docker VM**, not on your
machine — `ls /var/lib/docker` from your shell will fail. On native Linux,
just `sudo ls` them and skip the wrapper below. Everywhere else, reach
into the VM's mount namespace:

```bash
docker run --rm --privileged --pid=host alpine:3.20 \
  nsenter -t 1 -m -- <command>
```

`--pid=host` puts you in the VM's PID namespace so PID 1 is its init;
`nsenter -t 1 -m` then enters that process's *mount* namespace, which is
where `/var/lib/docker` lives. **Substitute your own directory IDs** from
the `docker inspect` output above — they are randomly generated per
machine.

40. **Predict**: `docker diff` reported six paths for `layers-box`,
    including `D /etc/motd`. What is physically in the `UpperDir`?
41. ```bash
    docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
      find /var/lib/docker/overlay2/1d5c8c44b19a8b…/diff -exec ls -ld {} +
    ```

42. **Observe**:

    ```
    drwxr-xr-x 4 root root 4096 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff
    drwxr-xr-x 2 root root 4096 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/etc
    -rw-r--r-- 1 root root   11 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/etc/app-release
    c--------- 2 root root 0, 0 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/etc/motd
    drwxr-xr-x 3 root root 4096 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/srv
    drwxr-xr-x 2 root root 4096 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/srv/data
    -rw-r--r-- 1 root root    6 Jul 25 16:16 /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff/srv/data/new.txt
    ```

    Seven entries — the upper root, two directories it had to materialize,
    and four leaves — for a 61 MB filesystem. And look at `/etc/motd`:
    leading `c`, size `0, 0`. That is a **character device with major and
    minor both zero** — exactly the on-disk whiteout the kernel
    documentation defines. Deleting a file *added* something. The `D` in
    `docker diff` is Docker translating this back into a word.

43. The kernel's own view, to prove none of this is a Docker abstraction:

    ```bash
    docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
      sh -c 'grep 1d5c8c44 /proc/mounts'
    ```

    ```
    overlay /var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/merged overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/5UFK4BZSC23VMAIAAC2IZICORX:/var/lib/docker/overlay2/l/CTHEBUFJOM6N2DI4GPOPG3CEWG:/var/lib/docker/overlay2/l/DHPCB4AFVBFGTV7W4IMCMGOMPJ:/var/lib/docker/overlay2/l/K3ONCNKGRH423U672XBR2RNF6U:/var/lib/docker/overlay2/l/NHHLV26VO4SQJ4LW4BBNG33IDC:/var/lib/docker/overlay2/l/7XATOXVVNXECXOSKERCYNBHVWW,upperdir=/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/diff,workdir=/var/lib/docker/overlay2/1d5c8c44b19a8b1d9d5b9106966f12a44b29fff1931d5586191f25386e2180ec/work 0 0
    ```

    A plain `overlay` mount in `/proc/mounts`, indistinguishable from one
    you'd make by hand. Docker's whole "container filesystem" is this
    single line. Six `lowerdir=` entries, one `upperdir=`, one `workdir=`.
    (The `l/` paths are short symlinks Docker keeps "to avoid hitting the
    page size limitation on arguments to the `mount` command" — the same
    limit behind overlay2's documented ceiling of 128 lower layers.)

44. **Predict**: walk the image layers now, in the same order.
    `uqvao75hp6…` is the `dd` layer, `s0boitxkt0…` the `rm` layer. Is the
    50 MB file on your disk right now?
45. ```bash
    docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- sh -c \
      'for d in s0boitxkt0lcv2ltxa1p6108u td38xshhvgkyrnryq0z1fhq8r \
                uqvao75hp68davd9jea3taw7m bsf07qsb707txc2nz2x55y0h2; do
         echo "=== $d ==="
         find /var/lib/docker/overlay2/$d/diff -exec ls -ld {} +
       done'
    ```

46. **Observe**:

    ```
    === s0boitxkt0lcv2ltxa1p6108u ===
    drwxr-xr-x 5 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff
    drwxr-xr-x 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff/etc
    drwxr-xr-x 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff/opt
    c--------- 3 root root 0, 0 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff/opt/build-cache.bin
    drwx------ 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff/root
    c--------- 3 root root 0, 0 Jul 25 16:14 /var/lib/docker/overlay2/s0boitxkt0lcv2ltxa1p6108u/diff/root/.aws-credentials
    === td38xshhvgkyrnryq0z1fhq8r ===
    drwxr-xr-x 4 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff
    drwxr-xr-x 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff/etc
    drwx------ 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff/root
    -rw-r--r-- 1 root root   21 Jul 25 16:14 /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff/root/.aws-credentials
    === uqvao75hp68davd9jea3taw7m ===
    drwxr-xr-x 4 root root     4096 Jul 25 16:14 /var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff
    drwxr-xr-x 2 root root     4096 Jul 25 16:14 /var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff/etc
    drwxr-xr-x 2 root root     4096 Jul 25 16:14 /var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff/opt
    -rw-r--r-- 1 root root 52428800 Jul 25 16:14 /var/lib/docker/overlay2/uqvao75hp68davd9jea3taw7m/diff/opt/build-cache.bin
    === bsf07qsb707txc2nz2x55y0h2 ===
    drwxr-xr-x 3 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/bsf07qsb707txc2nz2x55y0h2/diff
    drwxr-xr-x 2 root root 4096 Jul 25 16:14 /var/lib/docker/overlay2/bsf07qsb707txc2nz2x55y0h2/diff/etc
    -rw-r--r-- 1 root root   11 Jul 25 16:14 /var/lib/docker/overlay2/bsf07qsb707txc2nz2x55y0h2/diff/etc/app-release
    ```

    (Every layer also carries the parent directories of the files it
    touched — that's why each one has a bare `etc/`.)

    The `rm` layer is two character devices. The `dd` layer is all
    52,428,800 bytes, sitting in a directory on your disk, under a path no
    container can see. And one command finishes the story:

    ```bash
    docker run --rm --privileged --pid=host alpine:3.20 nsenter -t 1 -m -- \
      cat /var/lib/docker/overlay2/td38xshhvgkyrnryq0z1fhq8r/diff/root/.aws-credentials
    ```

    ```
    AKIAEXAMPLESECRETKEY
    ```

    Same secret you pulled out of the tarball in Part 2, now read straight
    off the host filesystem, from an image whose every container reports
    the file as missing.

## What you should see

A `docker history` whose per-layer sizes sum to exactly the image size
(`8,822,697 + 11 + 52,428,800 + 21 + 0 = 61,251,529`), with the `RUN rm`
step contributing `0B`. A container that agrees the deleted files are
gone, and a `docker save` tarball that hands you `AKIAEXAMPLESECRETKEY`
anyway, alongside two zero-byte `.wh.` tombstones. The same build with one
`&&` instead of three `RUN`s at `8,822,708` bytes — 6.9× smaller for
identical behavior. Two images agreeing on their first two layer digests
character-for-character, `docker system df -v` reporting `:squashed` at
`8.82MB` size and `0B` **unique** size, and a one-character edit
invalidating every layer above it. A `--no-cache` rebuild of that same
`RUN` producing a *different* digest for an identical 11-byte file, which
is how you learn the match was cache reuse rather than reproducibility. A
running container whose writable layer is `17B (virtual 61.3MB)`, a second
one at `0B`, and a two-byte append that costs 4.6 MB. And finally a real
`overlay` line in `/proc/mounts` with six `lowerdir`s, a character device
at `.../diff/etc/motd`, and a 52,428,800-byte file alive in a layer
directory that nothing can reach.

## Why

Because the layer is the unit of *transfer*, and immutability is what
makes transfer cheap. If a layer could be edited, its digest would change,
every image referencing it would need updating, and the "already have
that one" check that makes `docker pull` fast would have nothing stable to
compare. So a layer is frozen the moment it's written, named by its
contents, and shared by anyone whose stack happens to contain the same
bytes. Everything else follows from refusing to break that.

Note carefully what "shared by anyone whose stack contains the same bytes"
does and does not buy you. `88b4fba6…` is shared for the strong reason:
it is the layer you pulled, and every alpine-based image on the machine
names it. `7b511a71…` is shared for a weaker reason: BuildKit saw a step
it had already run on the same parent and reused the committed layer. Make
that step actually execute again and the digest changes, because the tar
records mtimes and two runs are never at the same instant. Content
addressing gives you deduplication of *bytes you already have*; it does
not give you reproducible builds, and Step 24 is the two-line proof.

Deletion is where the cost shows up. You cannot remove a file from a
read-only layer, so the only honest thing an upper layer can do is record
"stop looking below this path" — a whiteout. The union honors it, `ls`
honors it, your application honors it. The registry does not, because the
registry ships layers, not the union. `RUN rm` on a later line therefore
buys you nothing but two tombstones, and this is not a Docker bug to be
fixed; it is the direct price of the property that makes images shareable
at all. The fix is always to arrange for the bytes never to be committed
to a layer in the first place — one `RUN`, a multi-stage build, or a
build mount.

Copy-up is the same trade in the other direction. Making an image layer
writable per-container would mean copying it per-container, which would
undo the sharing. So the layer stays read-only and the copy happens
lazily, per file, on first write — near-zero cost for a container that
mostly reads, and a full file copy the instant it doesn't. `17B (virtual
61.3MB)` and `4.6MB for two bytes` are the two ends of that same
mechanism.

**Where the effect vanishes.** All of this is a property of the *union*,
not of containers. Mount a volume or a bind mount over a path and that
subtree is no longer part of the overlay: no copy-up, no whiteouts, no
contribution to `docker diff`, and no growth in the writable layer no
matter how much you write. That is why the copy-up surprise never bites
a database that keeps its data directory on a volume — and always bites
one that doesn't.

## Go deeper

- Rebuild `:leaky` as a multi-stage build — do the `dd` and the
  credentials work in a `FROM alpine AS build` stage, then
  `FROM alpine` and `COPY --from=build` only what you want. Check
  `RootFS.Layers`: the discarded stage contributes no layers at all,
  because it was never part of the final image's stack. This is the
  actual answer to Part 2's problem.
- Try `RUN --mount=type=secret,id=creds cat /run/secrets/creds` with
  `docker build --secret id=creds,src=./creds`. The file is present during
  the step and absent from the layer — verify with the Part 2 tarball
  scan and confirm nothing matches.
- Add `WORKDIR`, `ENV`, `LABEL` and `EXPOSE` lines and re-read
  `docker history`. All `0B`: they're recorded in the image *config*
  (`docker inspect --format '{{json .Config}}'`), which is itself a blob
  in the tarball. The config plus the ordered layer list is the entire
  image — there is no other file.
- Write a file to a `-v` volume from inside `layers-box` and re-run
  `docker diff`. It doesn't appear, because a volume is a bind mount over
  the union, not part of it — which is the whole reason volume data
  survives `docker rm`.
- Get the shared/unique split in exact bytes rather than the rounded
  column, still scoped to your own images:

  ```bash
  sock=$(docker context inspect --format '{{.Endpoints.docker.Host}}' | sed 's|unix://||')
  curl -s --unix-socket "$sock" http://localhost/v1.45/system/df | python3 -c '
  import json, sys
  for i in json.load(sys.stdin)["Images"]:
      for t in i.get("RepoTags") or []:
          if t.startswith("layers-demo"):
              print(t, i["Size"], i["SharedSize"], i["Size"] - i["SharedSize"])'
  ```

- Read the kernel's own description of whiteouts and copy-up:
  https://docs.kernel.org/filesystems/overlayfs.html — the "whiteouts and
  opaque directories" section defines the `0,0` character device you
  found, and the `.wh.` convention is Docker's tar-level equivalent,
  specified in the OCI image layer spec.

## Cleanup

Remove **only** what this exercise created — by exact name and tag, never
with a prune:

```bash
docker rm -f layers-box layers-box2
docker rmi layers-demo:leaky layers-demo:squashed layers-demo:rebuilt layers-demo:nocache
rm -rf /tmp/layers
```

The build cache outlives the images. Drop just this exercise's entries —
the `description` filter keeps everyone else's cache intact:

```bash
for pat in build-id build-cache aws-credentials app-release; do
  docker buildx prune -a --force --filter "description~=$pat"
done
```

Run it twice, for the same reason as in Setup, then confirm nothing of
yours is left and everyone else's cache still is:

```bash
docker buildx du --verbose --filter 'description~=build-cache' | grep -c '^ID:'   # → 0
docker buildx du | tail -2
```