6.6. Platform Delivery

What the state volume holds, and what losing it costs

Every durable record this platform produces lives on one PersistentVolumeClaim, a disk that outlives the pod mounting it. It is agentops-agent-state, mounted at /app/state, the cluster equivalent of agents/go/.state/ on your laptop:

1. runtime.db — ADK’s sessions and the ordered events inside them, exactly as 2.4. Sessions read them on your laptop.
2. tasks.db — the A2A task store: every protocol-visible unit of work, each carrying the context_id that joins it back to a session in the other file.
3. incidents.db — the writable dataset copy and its append-only audit_log (Chapter 3.1). Authenticated host actions and explicit drill fixtures populate it; the shipped k3d and GKE pods freeze guarded operational writes.
4. vectors.db — the derived retrieval index, and the only file here you could rebuild from the committed seed rather than recover.

Losing that volume loses no course data — the seed is safe in Git and in the image. It loses the record: every session, every task, and every audit_log row naming who approved an action and why, the row someone asks for three weeks after an approved restart. Without a backup, a volume failure or a hasty skaffold delete erases that trail with no error message, because nothing was broken.

A backup you have never restored is a hope, so this page restores one before it explains anything, then breaks a snapshot of your own state and ends on teardown. The drill goes first: it uses a throwaway temporary directory, never your real state, and needs no cluster and no model:

mise run state:drill

msg="backed up state file" file=incidents.db target=/tmp/tmp.2FDWKnM45n/backups/.incomplete-20990101T000001Z.1007982494/incidents.db
msg="backed up state file" file=runtime.db target=/tmp/tmp.2FDWKnM45n/backups/.incomplete-20990101T000001Z.1007982494/runtime.db
msg="snapshot complete" snapshot=/tmp/tmp.2FDWKnM45n/backups/20990101T000001Z
msg="restored state file" file=incidents.db target=/tmp/tmp.2FDWKnM45n/rollback-target/incidents.db
msg="restored state file" file=runtime.db target=/tmp/tmp.2FDWKnM45n/rollback-target/runtime.db
msg="restore complete" state_dir=/tmp/tmp.2FDWKnM45n/rollback-target
drill passed: versioned manifest, exact inventory, rollback, hashes, and schema compatibility

That is trimmed: the real run takes two snapshots and prints each file twice. The directory names carry the whole design — a snapshot is written under .incomplete-<stamp> and renamed to the visible <stamp> only after every file has been copied and hashed.

How backup and restore make a half-copied snapshot unusable

Never cp a live SQLite file. A copy taken mid-transaction can be torn, and a torn database is worse than no backup because it looks like one. Safe hot copies go through SQLite itself, which holds the page-level locks for you: mise run state:backup runs infra/scripts/backup-state.sh, which invokes the shared agent state CLI to snapshot every database with VACUUM INTO — an online-safe copy taken inside a read transaction, so a concurrent writer neither blocks it nor tears it.

Backup and restore share one contract, so an interrupted attempt can never masquerade as the newest good one:

    flowchart TD
    subgraph Backup["agent state backup — staged publication"]
        B1["stage in hidden<br/>.incomplete-STAMP directory"] --> B2["SQLite backup API<br/>copies every *.db"]
        B2 --> B3["inspect integrity,<br/>schema identity, size, SHA-256"]
        B3 --> B4["write manifest.json<br/>with source + exact inventory"]
        B4 --> B5["write JSON .complete<br/>with manifest SHA-256"]
        B5 --> B6["atomic rename →<br/>visible STAMP/ directory"]
    end
    subgraph Restore["agent state restore — generation replacement"]
        R1{"published directory<br/>with both metadata files?"} -->|no| RX(["refuse before mutation"])
        R1 -->|yes| R2["validate format, manifest hash,<br/>inventory, file hashes, schemas"]
        R2 --> R3["stage the complete<br/>manifest generation"]
        R3 --> R4["quarantine current databases<br/>and SQLite sidecars"]
        R4 --> R5{"install every<br/>staged database"}
        R5 -->|failure| R6["remove partial generation<br/>restore quarantine"]
        R5 -->|success| R7["remove quarantine<br/>fsync state directory"]
    end
    B6 -. "only a complete, manifest-bound generation is restorable" .-> R1
  

Diagram in words: Backup copies and inspects every database, records an exact hashed manifest, binds the completion marker to that manifest, and publishes once with an atomic rename. Restore validates the complete generation before touching live state, quarantines the current one, then installs every declared database or recovers the byte-identical prior generation.

Two files make that work. manifest.json records, for each copy, the filename, byte size, SHA-256 digest, and SQLite schema identity, plus a source block carrying build mode, version, timestamp, source identity, revision, tree digest, and dirty state from the same validated build authority as the image in 6.1. Containers. The JSON .complete marker carries format_version and manifest_sha256, so changing either the manifest or one declared database invalidates the whole generation rather than one file.

A multi-file restore cannot switch every SQLite file in one rename, so the shared CLI takes an exclusive process lock and fsyncs a three-phase journal recording the phase the restore reached and the hashed inventory of both the old and the new generation before replacing anything. On the next backup, restore, or A2A startup, an interrupted uncommitted transaction reconstructs the exact old generation, and a durable committed one validates and keeps the complete new generation. Missing or unexplained recovery evidence fails closed instead of guessing — the same journal you attacked by hand in 2.4. Sessions. Writers must still be stopped first.

In Kubernetes a CronJob does the same thing unattended. agentops-state-backup runs agent state backup from the course image — the same binary the agent is, under different arguments — writing completed timestamped snapshots from the state volume into a second agentops-state-backups volume. Host and cluster snapshots therefore share one format, and no copied program or extra image pin enters the supply chain.

infra/k8s/base/state-backup.yaml holds that CronJob, its ServiceAccount, and the backup claim, and nothing else: this course ships no restore Job. The shape one would take is already decided by the CronJob beside it — the same image with args: ["state", "restore", "<snapshot>", "--state-dir", "/app/state"], run against stopped workloads rather than as a file copy taken behind their backs.

The one in-cluster restore that does run here is deliberately a different program. agent state moves generations and asserts nothing about their contents, which is all a scheduled backup or a host restore needs; agent platform-backup plants identities and verifies them after the restore, evidence an ordinary restore has no reason to produce. infra/scripts/platform-backup-drill.sh generates the Job that calls it, against the live workloads of a real cluster.

This is single-node, lab-grade backup, not disaster recovery. The backup volume sits in the same cluster on the same disk, there is no point-in-time recovery between snapshots, and skaffold delete removes both volumes together. Production ships snapshots off-cluster and restores them on separate infrastructure — the simulation-is-not-production line 0.2. Evidence draws.

The drill you ran at the top of this page is not a smoke test. It is six deliberate attacks on the restore path, and it seeds throwaway session and incident state plus a mock approved-action audit row so there is something real to lose. A corrupt input publishes neither a completed snapshot nor leftover staging state; a changed database hash is rejected before one byte of live state changes; an injected second-file failure restores the complete previous generation; a successful restore removes a live database absent from the manifest instead of mixing generations; a future audit schema is rejected without mutating its input; and the valid generation restores the original session and audit evidence.

The Go state suite adds the process boundary a shell drill cannot fake safely: a child exits without cleanup after the first old-file replacement, and again after the durable commit marker. The next state command must recover the old generation in the first case and preserve the new one in the second.

To restore real host state, stop every writer first — the A2A server, the MCP server, any agent console or agent web — then select one visible directory carrying both metadata files and point the wrapper at it:

snapshot=""
for candidate in .state-backups/*; do
  [[ -f "${candidate}/manifest.json" && -f "${candidate}/.complete" ]] && snapshot="${candidate}"
done
test -n "${snapshot}"
mise run state:restore -- "${snapshot}"

The in-cluster drill uses stronger evidence than a row count. platform-backup seed records a session, a workflow task, a memory note, and an idempotent audit invocation as JSON; platform-backup restore-drill restores the selected snapshot and verifies all four identities plus the exact snapshot source identity. That seed writes through a typed maintainer tool and deliberately bypasses the A2A network path, so a green drill can never be cited as proof that Kubernetes callers are authenticated or that operational writes are enabled.

source_revision="$(
  go -C tools run ./cmd/source-identity \
    --root .. --mode release --field revision
)"

./infra/scripts/platform-backup-drill.sh \
  --context k3d-local \
  --work-dir "${work_dir}" \
  --source-revision "${source_revision}" \
  --reader-image \
  curlimages/curl:8.21.0@sha256:7c12af72ceb38b7432ab85e1a265cff6ae58e06f95539d539b654f2cfa64bb13 \
  --agent-card-url http://127.0.0.1:8080/.well-known/agent-card.json \
  --docker-config "${docker_config}" \
  --evidence-marker platform-backup-canary

Your turn: unpublish a snapshot and try to restore it

Now attack a snapshot of your own state, taken from the real databases on your disk. Delete one zero-byte file, the .complete marker, and ask for a restore. Predict which of three things happens. The restore proceeds because every database is present and hashed, it proceeds but warns, or it refuses. Then predict the second half: if it refuses, does it refuse before or after it has started writing the target?

• Mode: temporary experiment.
• Goal: watch an unpublished generation get refused, and see that the refusal costs the target nothing.
• Files to touch: only what you create under /tmp/agentops-restore-lab. Your live agents/go/.state/ is read, never written: both commands below name a throwaway backup root and target.
• Preflight: confirm the lab is not already there with test ! -e /tmp/agentops-restore-lab, then create it with mkdir -p /tmp/agentops-restore-lab.
• Steps: take a snapshot into the lab, delete its completion marker, then ask for a restore into a target directory that does not exist yet.

mise run state:backup -- agents/go/.state /tmp/agentops-restore-lab/backups
snapshot="$(ls -d /tmp/agentops-restore-lab/backups/* | tail -1)"
rm -- "${snapshot}/.complete"
mise run state:restore -- "${snapshot}" /tmp/agentops-restore-lab/target

• Gate that proves completion: the backup names every database it copied, and the restore exits non-zero naming the snapshot directory rather than any file inside it.

[state:backup] $ ./infra/scripts/backup-state.sh agents/go/.state /tmp/agentops…
msg="backed up state file" file=incidents.db target=/tmp/agentops-restore-lab/backups/.incomplete-20260810T211820Z.1385720088/incidents.db
msg="backed up state file" file=runtime.db target=/tmp/agentops-restore-lab/backups/.incomplete-20260810T211820Z.1385720088/runtime.db
msg="backed up state file" file=tasks.db target=/tmp/agentops-restore-lab/backups/.incomplete-20260810T211820Z.1385720088/tasks.db
msg="backed up state file" file=vectors.db target=/tmp/agentops-restore-lab/backups/.incomplete-20260810T211820Z.1385720088/vectors.db
msg="snapshot complete" snapshot=/tmp/agentops-restore-lab/backups/20260810T211820Z

[state:restore] $ ./infra/scripts/restore-state.sh /tmp/agentops-restore-lab/ba…
error: state snapshot failed: Snapshot is incomplete: expected .complete and manifest.json under /tmp/agentops-restore-lab/backups/20260810T211820Z
exit status 1
[state:restore] ERROR task failed

Your timestamps will differ, and a fifth line appears if the agent ever saved a memory note, but the four above are the inventory from the top of this page, as output rather than a claim. Now the second prediction: run ls /tmp/agentops-restore-lab/target. There is no such directory. The restore refused while validating, before it created the target, staged a generation, or quarantined anything.

• Final state: run rm -rf -- /tmp/agentops-restore-lab, then confirm test ! -e /tmp/agentops-restore-lab passes again. Nothing inside the repository was touched, and agents/go/.state/ still holds exactly what it held before.

A snapshot is not a backup until something has refused to trust a broken one.

Tear down the lab, and plan the optional GKE substrate

skaffold delete removes the course workloads, including their volumes and data, without touching the shared cluster or the kagent control plane:

# Choose exactly one profile after reviewing the context.
(
  cd infra
  skaffold delete --filename skaffold.yaml --profile local
)

The controller and the cluster are shared, so answer the ownership question with commands, not memory. If kubectl get agents.kagent.dev -A returns only the agents you deployed and kubectl get namespaces --show-labels shows no non-built-in namespace besides agentops and kagent, this cluster carries nothing but this course and removing the controller costs no one anything. Anything else on either list belongs to another project: leave the controller installed and stop after skaffold delete. To keep the cluster for a later session instead, 6.8. Platform Operations frees the memory without losing a volume. When the lab is genuinely over, helmfile --file infra/helmfile.yaml destroy and then k3d cluster delete local finish the job — in that order, because deleting the CRDs first leaves a BYO Deployment nothing owns.

The optional Google Kubernetes Engine lab creates nothing here, and the course checkpoint is the local profile alone — skip the rest and lose nothing. Its substrate is declared for OpenTofu, the open-source fork of Terraform, whose plan prints what an apply would create and changes nothing in the cloud:

tofu -chdir=infra/gcp init
tofu -chdir=infra/gcp validate
tofu -chdir=infra/gcp plan -out=tfplan

Three things come first, in this order:

1. Authenticate Application Default Credentials, because the provider reads Google credentials from the environment and cannot plan without them.
2. Copy terraform.tfvars.example to a gitignored terraform.tfvars and set the project ID there, so no project identifier is committed.
3. Set your own public /32 in the same file, so the control plane answers your address and not the internet.

Then review every created, changed, and destroyed resource in the plan, and read the monthly figure off 7.3. Costs, which owns the estimate, its date, and what it excludes. Planning is not permission to apply, and this course task executes no cloud commands.

Tear an approved GKE lab down, in the order that leaves nothing billing

A lab nobody destroys keeps charging. Teardown is the mirror image of the apply, every step is context-checked, and the order matters because each stage removes the thing the next one would otherwise strand:

1. Record the project-level state OpenTofu deliberately does not restore — enabled services, service accounts, IAM policy — and every PersistentVolume, with infra/scripts/gcp-lab-audit.sh record. Those disks are the whole data inventory, so this runs before either deletion, not after.
2. Run skaffold delete --profile gke against the exact named context, and wait for the namespace to disappear before continuing.
3. Destroy the Helm releases with helmfile --file infra/helmfile.yaml destroy, then delete the kagent namespace.
4. Review one saved tofu plan -destroy, then apply that saved plan rather than re-planning at apply time.

You are done when every recorded PV name and CSI disk handle is gone and tofu state list prints nothing. Anything left in either place is still billable.

Deployment does not authorise application writes, on any environment. Enabling them needs a separate owner decision and an implemented caller-identity substrate — issuer, audience, JWKS lifecycle, gateway subject transformation, and rotation and revocation operations — followed by authenticated action evidence on the exact deployed revision. Flipping the kill switch alone is unsafe.

What you can do now

• mise run state:drill ends with drill passed: versioned manifest, exact inventory, rollback, hashes, and schema compatibility, having attacked its own restore path six ways.
• You can predict what a missing .complete costs a restore: it refuses naming the snapshot directory, before it creates a single file.
• You can explain why a snapshot directory without the .incomplete- prefix is the only kind a restore will trust.
• You can say what skaffold delete removes, what it leaves behind, and the two kubectl questions you answer before running it. Nothing was created in GCP, whether you read the optional plan or skipped it.

Continue to 6.7. Promotion and Rollback when you have restored state from a backup at least once.