6.1. Containers

Why the agent has to become a container image

A container image is a filesystem plus the metadata that says how to start it. It holds everything the program needs, frozen into one artifact you can address, copy, and scan. Two things make it the unit of work from here on. A cluster cannot run mise run a2a — it starts images, so nothing in Chapter 6 happens until the agent is one. And whatever you put inside the image becomes the surface an intruder inherits, because every shell, package manager, and compiler in there runs with the agent’s identity the moment someone reaches code execution.

That second point is why this image contains almost nothing. There is no shell in it — no sh, no package manager, no curl, no compiler, no module cache. The whole runtime is one static binary, a read-only dataset, and an empty writable directory, running as UID 10001.

That is a measurable property rather than an adjective, and checking it takes about thirty seconds. Build the image and interrogate it:

mise run build:agent-image
docker image inspect agentops-agent:dev \
  --format '{{.Size}} {{json .Config.Entrypoint}} {{json .Config.Cmd}} {{.Config.User}}'

18573965 ["/app/agent"] ["a2a"] 10001:10001

Under 19 MB, one entrypoint, one default argument, and a non-root user baked into the image rather than requested at run time. Treat the size as a measurement rather than a budget — remeasure it after a dependency or base-image change instead of defending a number.

An image is only useful as evidence if you can tie the bytes running in a cluster back to the source they were built from. This binary carries that link and will print it:

docker run --rm --read-only agentops-agent:dev version | jq

{
  "build_timestamp": "2026-08-10T17:37:25+02:00",
  "mode": "development",
  "version": "development",
  "source_identity": "unknown+dirty.313c8b715b68",
  "tree_digest": "sha256:313c8b715b680825efcba36fb7a5c9d90d83ed8c5208d778916776b9e92c398f",
  "dirty": true
}

Read source_identity carefully, because it is the whole point of that field. The tree it was built from had uncommitted edits, so the build refuses to name a commit and reports unknown+dirty.<digest> instead. Your digest will differ from the one above — it is a hash of your working tree, not of mine. A clean release build names its revision; a dirty one never borrows the credibility of HEAD.

You just built a deployable artifact that cannot lie about where it came from.

How a two-stage build keeps the toolchain out of the runtime

A multi-stage build compiles in one image and ships from another. The compiler, the module cache, and the source never reach the artifact you deploy. The build stage carries the pinned Go toolchain and module cache, and compiles exactly one executable:

# CGO_ENABLED=0 is a course invariant, not an optimization: it is what lets the binary ship
# on a base with no libc and keeps exactly one SQLite implementation in the process.
# -trimpath removes the builder's absolute paths so two machines produce the same binary;
# -s -w drop the symbol table and DWARF, which debug.ReadBuildInfo does not need.
ENV CGO_ENABLED=0 GOOS=linux
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build -trimpath -ldflags="-s -w \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.buildMode=${AGENT_BUILD_MODE} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.version=${OCI_VERSION} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.sourceIdentity=${AGENT_SOURCE_COMMIT} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.revision=${AGENT_SOURCE_REVISION} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.treeDigest=${AGENT_SOURCE_TREE_DIGEST} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.buildTimestamp=${OCI_CREATED} \
      -X github.com/MLOps-Courses/agentops-open-course/agents/go/buildinfo.dirty=${AGENT_SOURCE_DIRTY}" \
      -o /out/agent ./cmd/agent && \
    /out/agent version >/dev/null

CGO_ENABLED=0 is a course invariant rather than an optimization. Without cgo the binary needs no libc, which is what lets it run on a base that has none, and it keeps exactly one SQLite implementation inside the process instead of a Go one and a C one disagreeing about the same file. Reintroducing cgo would change the runtime, the scanner findings, and the portability contract in one move. -trimpath strips the builder’s absolute paths so two machines produce the same bytes; -s -w drop the symbol table and DWARF, which the runtime does not need. The build then runs /out/agent version before anything is copied out, so a binary whose linked identity is malformed fails inside the builder rather than in your cluster.

The runtime stage then starts over from a base that carries nothing at all. That base is one FROM line, and this is the whole of it:

FROM cgr.dev/chainguard/static@sha256:24dd7ff8788fdfadda39eeeaefefb6d1cec6002a545935a5f7e017484053734f AS runtime

Everything in the final image arrives after that line by explicit COPY: the compiled binary, the committed dataset, and a state directory created back in the builder with its ownership already set — because a base with no shell cannot run mkdir.

Look at how that base is addressed, because both stages do the same thing and it is a different promise from a tag. A tag is a mutable pointer: golang:1.26.6-alpine can be re-pushed tomorrow to mean different bytes, and your “reproducible” build silently becomes a different build. A digest names content, so if the bytes change the digest no longer resolves and the build fails loudly instead of succeeding differently. The builder’s base carries both, tag and digest together, so the readable version survives beside the immutable one:

FROM golang:1.26.6-alpine@sha256:af8d6740070b8906d12eae1c3e3ea0957fb63f492051ea05e354c38ef9fe88df AS build

One Go version has to hold in five places at once: the root mise.toml, the three go.mod files under agents/go, evals, and tools, and the tag on that FROM line. mise run check:docs reads all five and fails if any one of them drifts, because the comparison lives in the authoring checker rather than in the infrastructure script — scripts/check-infra.sh never opens a go.mod, so bumping the toolchain and running only check:infra proves nothing about the other four. Update them together, then rerun compile, test, scan, and image checks.

The identity you read out of version is enforced here, before the distroless stage exists:

RUN case "${AGENT_BUILD_MODE}" in development | release) ;; *) exit 1 ;; esac && \
    case "${AGENT_SOURCE_COMMIT}" in "" | "<no value>" | *'{{'*) exit 1 ;; esac && \
    case "${AGENT_SOURCE_TREE_DIGEST}" in sha256:????????????????????????????????????????????????????????????????) ;; *) exit 1 ;; esac && \
    case "${AGENT_SOURCE_DIRTY}" in true | false) ;; *) exit 1 ;; esac && \
    case "${OCI_CREATED}" in "" | "<no value>") exit 1 ;; esac && \
    case "${OCI_VERSION}" in "" | "<no value>" | *'{{'*) exit 1 ;; esac

Every case arm rejects one specific way a build pipeline lies. An empty value, a <no value> string, a half-expanded {{ template, a tree digest that is not 64 hex characters, a dirty flag that is neither true nor false — each of those exits non-zero. That guard exists because Skaffold renders an unset template as the literal text <no value> and would otherwise build on happily, shipping an image that claims a revision nobody can check. It is the second line of defence; the exercise below meets the first one.

One artifact serves three roles, because ENTRYPOINT ["/app/agent"] fixes the executable and CMD ["a2a"] only selects a default surface. A manifest overrides args and gets a different program without repeating a path that could drift:

Inside the image, /app/data holds the committed seed and /app/state is owned by UID/GID 10001 and is the only persistent write target. Run with a read-only root filesystem, mount only the state path, and give /tmp a small tmpfs — which is exactly the shape 6.3. Platform Agents declares in the cluster.

Scan the built image, then record its digest, not its tag

Before you run this, predict a number. How many separate things does a vulnerability scanner find to analyse inside an image that contains one binary and no package manager?

trivy image --severity HIGH,CRITICAL --exit-code 1 agentops-agent:dev

Trivy narrates its progress on stderr. This is everything it wrote to stdout, from the Report Summary heading down:

Report Summary

┌─────────────────────────────────────┬──────────┬──────────┬───────────────────┬─────────┬─────────────────┐
│               Target                │   Type   │ Licenses │ Misconfigurations │ Secrets │ Vulnerabilities │
├─────────────────────────────────────┼──────────┼──────────┼───────────────────┼─────────┼─────────────────┤
│ agentops-agent:dev (wolfi 20230201) │  wolfi   │    -     │         -         │    -    │        0        │
├─────────────────────────────────────┼──────────┼──────────┼───────────────────┼─────────┼─────────────────┤
│ app/agent                           │ gobinary │    -     │         -         │    -    │        0        │
├─────────────────────────────────────┼──────────┼──────────┼───────────────────┼─────────┼─────────────────┤
│ OS Packages                         │    -     │    0     │         -         │    -    │        -        │
└─────────────────────────────────────┴──────────┴──────────┴───────────────────┴─────────┴─────────────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

Three rows, and only the first two carry a vulnerability count: the Wolfi base’s package database, and the Go binary’s own module graph. Both are zero. The third row is the licence scanner reporting on the same operating system — which, in this image, is three packages in total. Trivy says so in passing on stderr, [wolfi] Detecting vulnerabilities... pkg_num=3, and will name them if you ask:

trivy image --format json --list-all-pkgs agentops-agent:dev \
  | jq -r '.Results[] | select(.Type == "wolfi") | .Packages[].Name'

ca-certificates-bundle
tzdata
wolfi-baselayout

A certificate bundle, a timezone database, and the file layout the two sit in. That is the small attack surface stated as a scanner’s arithmetic rather than as an adjective, and --exit-code 1 turns it into a gate you can put in front of a deploy.

One habit makes that gate worth having: record the image digest after the scan, not the tag. A tag rebuilt tomorrow can name different content, so release evidence that says “we scanned agentops-agent:dev” says almost nothing a week later. An SBOM inventories what is inside; a signature binds an identity to a digest. Neither says the agent behaves correctly — that is what 6.7. Promotion and Rollback is for, and what 0.2. Evidence means by not letting one green tick imply another.

Verifying a published signature and its attestation belongs to a release pipeline this course does not run, so this page stops where you can check every claim yourself: the artifact on your own machine.

Your turn: ask for a release build on a tree that has moved

The version output at the top of this page said unknown+dirty because the tree had uncommitted edits. A development build was happy to record that. Now ask the same tree for a release build. Predict first: does it build anyway and label the result dirty, or does it refuse — and which of the two would you want the night someone asks which commit is in production?

• Mode: temporary experiment.
• Goal: watch the release path refuse to name a revision it cannot stand behind, over a change as small as one untracked file.
• Files to touch: one throwaway file you create at the repository root, scratch.txt. No tracked file is edited, and no image is overwritten.
• Preflight: confirm nothing is there already with test ! -e scratch.txt.
• Steps: from the repository root, run printf 'scratch\n' > scratch.txt, then AGENT_BUILD_MODE=release mise run build:agent-image.
• Gate that proves completion: the task exits non-zero before docker build is reached, naming the tree rather than the file.

[build:agent-image] $ build_mode="${AGENT_BUILD_MODE:-development}"
source-identity: release source is dirty; tracked and untracked inputs must match HEAD
exit status 1
[build:agent-image] ERROR task failed

• Final state: run rm -- scratch.txt and confirm the test ! -e preflight passes again. Development builds were never affected either way — mise run build:agent-image still rebuilds the same image you inspected at the top of this page.

That refusal happened in mise, before Docker was invoked at all, because source-identity resolves the working tree first and will not hand a release build a revision. Untracked counts, which surprises people — a stray scratch file is exactly the kind of thing that ends up baked into an image nobody can reproduce later.

You just proved that this build system would rather fail than mislabel an artifact.

What you can do now

• docker image inspect reports /app/agent, default argument a2a, and user 10001:10001 for the image you just built.
• docker run --rm agentops-agent:dev version prints a source identity that matches your tree’s cleanliness rather than claiming HEAD.
• The high/critical scan exits zero over the two vulnerability targets a distroless image contains.
• One untracked file stopped a release build, and removing it let the preflight pass again.

An image is not something you trust because the build succeeded. You can open this one, read what it claims about its own origin, check that claim against the tree on your disk, and say why cgo is off, why both bases carry digests, and why one binary serves three roles.

Continue to 6.2. Platform Install once the exact image you intend to deploy has been inspected and scanned.