7.6. Governance

Why every consequential write leaves an audit row

An audit row is the record a consequential write leaves behind. It names which agent executed it, which principal approved it and why, what the state looked like at execution time, and the session and invocation that produced it. It exists because the thing that ran the command is an agent, and “the agent restarted it” is an answer nobody accepts — a service that came back with no attributable record cannot be reviewed, apportioned, or defended. The questions that arrive afterwards are who approved this, what did they know, and show me.

This page reads one such row and the schema behind it, finds where “append-only” stops being a security claim, and names the retention owner for every store. Every consequential write in this repository leaves that row, written for someone who was not there — here, a restart of inventory:

sqlite3 -header -column agents/go/.state/incidents.db \
  'SELECT schema_version, actor, approved_by, action, target FROM audit_log ORDER BY id DESC LIMIT 5;'

schema_version      actor       approved_by      action        target
--------------  --------------  -----------  ---------------  ---------
             1  agentops-agent  ana          restart_service  inventory

Without -header -column you get the same row as 1|agentops-agent|ana|restart_service|inventory, fine for a script and hard for a person to read. On a fresh checkout that command returns nothing at all, because nothing has been approved yet. The row above is the same query run against the throwaway copy the exercise below builds — the only audit row this repository can show you before you make one, and named as a substitution because a capture that quietly came from a different file is exactly the drift this chapter is about. Run an action through mise run run, approve it when ADK asks, and your own row appears under agents/go/.state. Query that, never the committed seed at agents/data/incidents.db: host writes belong to runtime state.

Two columns in that output carry the weight. actor is the executing agent; approved_by is the principal that confirmed the action. They are separate fields because they are separate claims: conflating them is how an automated action quietly acquires a human’s authority in a report.

What one approved write records, column by column

Governance here means something narrow and checkable: every consequential write has a named authority, a bounded surface, and a record another person can inspect. Four boundaries implement it:

1. Only restart_service and resolve_incident can change operational state.
2. ADK confirmation — the runtime pausing a tool call until a person approves it — must clear on the exact action and target before either handler runs.
3. The state change and its audit row commit in one SQLite transaction, so there is no window where the world changed and the record did not.
4. The model-backed evidence a release decision reads is sanitized by construction: one dispatch-only workflow runs the evaluation and uploads results.json, whose shape cannot carry a prompt, an answer, a tool body, or a judge’s rationale, so the artifact is safe to keep and pass on.

The schema is where that design becomes legible:

CREATE TABLE audit_log (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    schema_version  INTEGER NOT NULL DEFAULT 1 CHECK (schema_version >= 1),
    ts              TEXT NOT NULL,                         -- ISO-8601 UTC of the action
    actor           TEXT NOT NULL,                         -- executing agent, e.g. "agentops-agent"
    approved_by     TEXT NOT NULL,                         -- ADK user id; synthetic on unauthenticated A2A
    rationale       TEXT NOT NULL,                         -- the approver's stated reason (HITL, Ch. 4.5)
    context_summary TEXT NOT NULL,                         -- current decision context reconstructed at execution
    session_id      TEXT NOT NULL,
    invocation_id   TEXT NOT NULL,
    action          TEXT NOT NULL,                         -- e.g. "restart_service", "resolve_incident"
    target          TEXT NOT NULL,                         -- service name or incident id
    detail          TEXT NOT NULL
);

CREATE UNIQUE INDEX uq_audit_log_idempotency
ON audit_log (invocation_id, action, target);

Each column answers a separate question. rationale records why the approver accepted the action, after deterministic redaction and length validation. context_summary captures the service and incident state as it was at execution time, which makes the decision reviewable rather than merely recorded. session_id and invocation_id join the row to the exact decision, and the unique index over (invocation_id, action, target) at the bottom makes replay idempotent: re-running the same invocation, action, and target returns the original row without mutating state again, while changing any member of that triple is a genuinely different request. schema_version lets backup and restore refuse a database that newer code wrote and older code cannot interpret, so a restore of the wrong snapshot fails loudly instead of misreading columns.

What the row does not contain is equally deliberate: no prompt, no model response, no raw tool body, no credential, no endpoint URL, no trace body.

Approval is checked against the work rather than stored as a bare flag: the handler validates the invocation id, the action, the target, the rationale, and the current decision context together. That is why the conversational entrypoint keeps write tools in-process: the remote MCP surface publishes six reads and cannot manufacture an ADK confirmation context, no matter what tools it advertises.

Who may authorize anything depends on how the agent is reached, and the three profiles differ:

• The direct ADK surface carries its local user id into confirmation, so approved_by names a real local principal.
• A2A without gateway identity gets a synthetic id that scopes sessions and memory but cannot authorize a guarded write. A trusted gateway must validate the caller and overwrite the configured identity header — the request header carrying a caller identity, which any client can forge. The typed subject parsed from that header must then match the invocation owner, the principal the session was opened under; otherwise one caller acts inside another’s session. The secured host profile makes that chain executable with lab JWTs.
• The shipped k3d and GKE profiles configure no caller identity provider at all and therefore set AGENT_WRITES_DISABLED=true, so guarded actions refuse regardless of confirmation. That is not an oversight; it is the only correct default for a surface with no verified caller.

On the account-free route, task and context identifiers are single-user capability handles: the identifier is the whole authorization, so anyone who learns one has that access. They are not tenant credentials and support no multi-user confidentiality claim — keep that route private, or add gateway authentication before sharing it.

Append-only triggers stop application writes, not an administrator

SQLite triggers reject UPDATE and DELETE against audit_log. Try it on a throwaway copy and the database refuses:

sqlite3 /tmp/audit-demo.db "UPDATE audit_log SET rationale = 'looked fine to me' WHERE id = 1;"

Error in 2nd command line argument: audit_log is append-only

DELETE answers identically, and the tests show the action and its row rolling back together when a write fails. So the precise claim is: ordinary application writes cannot rewrite or delete an existing audit row. The claim you must not make is immutability. An administrator who can replace the database file, alter the schema, drop the triggers, or restore an older snapshot changes history without touching a single application code path. Production-grade immutability needs a separately administered append-only or signed destination with independent access control and retention, which is a different system rather than a stricter trigger.

Authority is layered rather than absolute: an authenticated host agent appends rows but cannot update or delete them. The shipped Kubernetes agent freezes guarded writes at startup. The read-only MCP pod cannot write the shared state at all. The evaluation harness cannot import the agent module and exports only through its own explicit OTLP endpoint. The collector routes telemetry but receives no prompts or answers by default. A cluster or storage administrator can replace state and telemetry outright, so a real audit trail needs an administrative separation this lab does not have. On the optional GKE path, Workload Identity Federation authenticates the gateway to Google Cloud, not a human or an A2A client to the gateway; it cannot establish task tenancy or authorize an operational write.

A record you cannot produce months later is not evidence, so retention is a governance question. It follows the store that owns each signal, and the course configures no universal policy:

Telemetry and generated eval artifacts are content-free by default, but metadata is not automatically harmless: a session id, a source identity, a model name, or an incident target still reveals operational context.

A release decision reads the same way: one clean source identity, plus results.json with its per-case scores and required-case verdict, plus judge-calibration-results.json read as a measurement rather than a threshold, plus the image digest, scan, smoke result, and attestation for the exact artifact. Prompts, answers, tool arguments and responses, judge rationales, provider errors, endpoints, and credentials are forbidden from all of it. A green offline suite covers the harness and the assets; it says nothing about a model-backed run, a deployed image, or a published release — the distinction lives in 0.2. Evidence.

Your turn: try to rewrite an audit row

Predict before you run it. You will append a row to a copy of the database and then edit that same row as its owner, with full filesystem permissions and no application in the way. Does SQLite let you?

• Mode: temporary experiment.
• Goal: append one audit row to a throwaway copy, fail to change it, fail to delete it, and then name the one move that would defeat both triggers.
• Files to touch: only /tmp/audit-demo.db, which you create and delete. The committed seed and your runtime state are never opened for writing.
• Preflight: from the repository root, confirm the copy does not exist with test ! -e /tmp/audit-demo.db, then create it with cp agents/data/incidents.db /tmp/audit-demo.db.
• Steps: insert one row with sqlite3 /tmp/audit-demo.db "INSERT INTO audit_log (ts, actor, approved_by, rationale, context_summary, session_id, invocation_id, action, target, detail) VALUES ('2026-08-10T22:40:00Z', 'agentops-agent', 'ana', 'crash loop confirmed in the last 200 log lines', 'inventory: down; INC-002 open', 's-1', 'e-1', 'restart_service', 'inventory', 'restart requested');". Read it back with the SELECT from the top of this page, pointed at /tmp/audit-demo.db and keeping the -header -column flags. Then try the UPDATE above, and a DELETE of the same row.
• Gate that proves completion: the insert succeeds and the row reads back; the UPDATE and the DELETE both exit non-zero with audit_log is append-only; and you can state in one sentence the move that would erase the row anyway.
• Final state: run rm -- /tmp/audit-demo.db. Nothing in the repository changed, and git status --short shows no new file.

The answer to that last clause: replace the file, or drop the triggers and then update. Both are one command for anyone with write access to the directory.

None of this needs a model; the guarded-write path and the state machinery around it are decidable offline:

cd agents/go
go test -race ./tools ./state

ok  	github.com/MLOps-Courses/agentops-open-course/agents/go/tools	1.625s
ok  	github.com/MLOps-Courses/agentops-open-course/agents/go/state	6.219s

What you can do now

• You can join an approved action to its actor, approver, rationale, session, invocation, and target — and say why actor and approved_by are separate columns.
• You can say what audit_log is append-only proves, what it does not, and the move that defeats it.
• You can say which of the three deployment profiles may authorize a guarded write, and why the other two refuse.
• You can name the retention owner and the erasure limitation for each store, without promising deletion you cannot perform.

Continue to 7.7. Incident Response, which reads every signal in this chapter together for the case where the agent is itself the failing component.