7.6. Governance

What does governance mean for this agent?

Six months from now, someone asks: who approved restarting the checkout service, and why? Governance is being able to answer that, with a record rather than a memory.

Answering it after the fact, to someone who was not in the room, means holding four things:

• who could do what, under which policy;
• which model, prompt, and data were used;
• which runtime identity confirmed a state change, and what changed;
• how long the evidence survives to be inspected.

For a deterministic service a commit hash and an access log almost cover it. An agent adds three problems a CRUD app does not have:

1. The actor proposing the change is a sampling model.
2. The target of the change is decided at runtime from data the model read.
3. The evidence is scattered across a trace store, a metrics store, a log store, and an application database that have different owners and different lifetimes.

This repository implements governance as identities, code, transactions, storage, traces, and review — not a policy paragraph. This page walks that chain end to end for the one thing the agent can actually change: a guarded action, a mock write that runs only after a human approves it. It is deliberately honest about where the chain breaks.

Who authorized this change, and can you prove it?

An audit row is only as strong as the identity in its approved_by column. The hard part of agent governance is carrying a verified human identity from the network edge all the way to the row that records a mutation. The course ships the whole chain and one intentional break in it:

flowchart LR
    Person([Real operator]) --> Edge["agentgateway edge<br/>JWT validated, sets<br/>trusted identity header"]
    Edge -->|"AGENT_TRUSTED_IDENTITY_HEADER set"| Verified["verified subject<br/>threaded into user_id"]
    Edge -. "header unset (default):<br/>no subject to carry" .-> Synth["ADK synthetic user<br/>A2A_USER_context-id"]
    Verified --> Ctx
    Synth --> Ctx["ToolContext<br/>user_id, session_id, invocation_id"]
    Ctx --> Row["audit_log row<br/>approved_by, rationale, context_summary"]
    Row --> Trace[("MLflow trace<br/>same session and invocation ids")]

On the default course path the A2A listener is unauthenticated (4.6. Security), so there is no real subject to carry. ADK derives a synthetic per-conversation user A2A_USER_<context-id>, and that value lands in approved_by.

That synthetic id still links the pause, the resume, the mutation, the session, the invocation, and the MLflow trace into one correlatable thread. What it does not do is name a person. The 5.3. A2A Gateway integration test asserts exactly this default shape (approved_by == f"A2A_USER_{context_id}").

Closing that gap is opt-in and lives at the server boundary. A trusted gateway validates the caller's JWT — a signed token proving who the caller is — and sets a caller-identity header. AGENT_TRUSTED_IDENTITY_HEADER names that header. A small ASGI middleware in server.py binds the verified subject for the request, and the A2A request converter makes it the session user_id, so the same approved_by column now names the real operator.

The trust boundary is explicit and one-directional:

• The value is honored only when the variable is set.
• An operator sets it only behind a gateway that validates the JWT and overwrites any client-supplied copy of the header, because a raw client could otherwise forge it.
• Unset, the synthetic id stands: the audit thread is still correlatable, just not attributable to a person.

Owned by 5.5. Gateway Security, which shows the header contract.

Prove the approval and transaction boundary offline before inspecting a runtime row:

cd agents/python
uv run pytest tests/test_actions.py -q

How do you inspect action evidence?

Read the rows the agent wrote. One has to exist first: approve a guarded action through the confirmation flow of 3.1. Tools before you run either command below.

On the Kubernetes path this needs the local platform from 6.2. Platform Install, with the agent deployed as in 6.6. Platform Delivery. Read the table inside the pod:

kubectl -n agentops exec deploy/agentops-agent -- \
  python -c 'import sqlite3; db=sqlite3.connect("/app/state/incidents.db"); print(db.execute("SELECT ts, actor, approved_by, action, target FROM audit_log ORDER BY id DESC LIMIT 10").fetchall())'

If you are still on the host path used earlier in this chapter, the same table sits in the agent's own state directory:

cd agents/python
uv run python -c 'import sqlite3; db=sqlite3.connect(".state/incidents.db"); print(db.execute("SELECT ts, actor, approved_by, action, target FROM audit_log ORDER BY id DESC LIMIT 10").fetchall())'

Both print the ten most recent rows, newest first.

Add session_id, invocation_id to the SELECT list and compare those values with the corresponding MLflow trace: this is the right-hand join in the authority diagram above, made concrete. Treat the synthetic A2A_USER_<context-id> as correlation evidence, not authenticated identity. Do not expose audit rows through an unrestricted agent tool — a read tool the model can call turns the append-only log into model-reachable data.

What does an audit row contain?

Twelve columns, eleven of them NOT NULL, so a row cannot be half-recorded. Here is the schema, then one row filled in.

CREATE TABLE audit_log (
    id              INTEGER PRIMARY KEY AUTOINCREMENT,
    schema_version  INTEGER NOT NULL DEFAULT 1 CHECK (schema_version >= 1),
    ts              TEXT NOT NULL,                         -- ISO-8601 UTC of the action
    actor           TEXT NOT NULL,                         -- executing agent, e.g. "agentops-agent"
    approved_by     TEXT NOT NULL,                         -- ADK user id; synthetic on unauthenticated A2A
    rationale       TEXT NOT NULL,                         -- the approver's stated reason (HITL, Ch. 4.5)
    context_summary TEXT NOT NULL,                         -- current decision context reconstructed at execution
    session_id      TEXT NOT NULL,
    invocation_id   TEXT NOT NULL,
    action          TEXT NOT NULL,                         -- e.g. "restart_service", "resolve_incident"
    target          TEXT NOT NULL,                         -- service name or incident id
    detail          TEXT NOT NULL
);

CREATE UNIQUE INDEX uq_audit_log_idempotency
ON audit_log (invocation_id, action, target);

Approving restart_service on inventory records this (identifiers shortened for reading):

Three of those columns are what make the row evidence rather than a log line: approved_by (who), rationale (why), and context_summary (what the agent saw at that moment).

schema_version makes the append-only row self-describing. The writer stamps CURRENT_AUDIT_SCHEMA_VERSION; the database also defaults the column to 1, so every current row declares the contract a future reader must use.

Every guarded action writes exactly one row inside the same transaction that performs the mutation. The (invocation_id, action, target) tuple is also a unique idempotency key: replaying one approved write returns its original row without mutating state again. Why that atomicity is non-negotiable is 3.1. Tools's lane.

The row is not free-form. _validated_approval in actions.py requires three things before any write:

1. A confirmed approval carrying user_id, session id, invocation id, and a non-empty rationale.
2. A rationale that, once trimmed, stays within MAX_AUDIT_RATIONALE_LENGTH (500 characters); a longer one is rejected before any write.
3. A pass of redact_persisted_text over the rationale before persistence.

That last pass scrubs concrete PII (personally identifiable information) and obvious credential/token patterns on the way into SQLite, while domain identifiers such as INC-002 survive. The persisted-text policy deliberately excludes Presidio's broad ORGANIZATION recognizer so incident and service ids stay legible in evidence. A rejected rationale produces neither a mutation nor a row.

The context_summary column is the one field a caller cannot supply. It is reconstructed server-side at execution while the write lock is held (_restart_context and _resolution_context in data.py). So the recorded "why now" is the agent's actual view of state — the service status and open incidents at the moment of the change — not a claim the caller typed.

One honesty note carried from 7.4. Feedback: redact_persisted_text guards this rationale, but the parallel free-text field a human types into an MLflow assessment is not redacted. Keep reviewer notes about behavior and off the record for specifics.

Is the audit log immutable?

No — append-only, not immutable, and the difference is the whole point. The schema installs two triggers immediately after the table:

CREATE TRIGGER audit_log_no_update
BEFORE UPDATE ON audit_log
BEGIN
    SELECT RAISE(ABORT, 'audit_log is append-only');
END;

CREATE TRIGGER audit_log_no_delete
BEFORE DELETE ON audit_log
BEGIN
    SELECT RAISE(ABORT, 'audit_log is append-only');
END;

These abort any UPDATE or DELETE against existing rows, so INSERT is the only write the application can make through the schema. That is genuinely useful: a compromised agent process or a buggy tool cannot quietly rewrite what it already recorded.

It is not tamper-proof. Anyone who reaches the database file rather than the schema — replace the file, DROP TRIGGER, or re-create the table on a fresh connection — can still rewrite the log, and nothing in the course records that they did. Two later sections make that boundary concrete: how long the evidence lives, and who can still touch it.

How long does each kind of evidence survive?

The evidence behind one action does not have a single lifetime. It has several, each set independently by whichever store holds that pillar, and they do not agree. That mismatch is a governance fact you must state before promising "we can reconstruct any action for N days":

Four rows use the glossary's PVC and RWO storage contracts.

Those lifetimes are set in different places:

1. Prometheus metrics age out at 2 days in the Kubernetes overlay (7 days on the host Compose profile) and Loki logs at 7 days by explicit configuration, so the metrics and log lines that gave an action its context are gone within a week — that retention is 7.2. Monitoring's lane.
2. MLflow ships no retention or garbage-collection job, so its traces and its human and judge assessments persist for the life of the volume (the mlflow-data volume on the host, the 5 Gi PVC in Kubernetes, GCS artifacts on GKE) — 7.4. Feedback's lane.
3. Runtime sessions, A2A tasks, and audit rows live for the life of the agentops-agent-state RWO PVC. The agent and MCP share that claim, so state stays coherent across pod replacement.

One nightly snapshot is the only copy that ever leaves that state PVC, and it is crash and mistake recovery rather than disaster recovery.

So one action's evidence has heterogeneous lifetimes — 2 days of metrics, 7 days of logs, PVC-lifetime traces, sessions, tasks, and audit rows, plus seven daily state snapshots. Losing the cluster or both PVCs loses all of it at once.

How do you answer a data-subject access or erasure request?

Someone emails and asks what personal data you hold about them, or asks you to delete it. What can you actually do?

Privacy laws can grant access and erasure rights, but their scope and exceptions depend on the jurisdiction, controller, purpose, and facts. A DSAR is a data-subject access request. Retention says how long evidence lives; this question asks what an operator can find and remove.

Start with the real data inventory. The only shipped per-user erasure command is forget_user_memory, which deletes long-term incident_notes keyed by user_id:

cd agents/python
uv run python -m agent.longterm "alice@example.com"   # erase one user's long-term notes

Do not infer runtime storage from telemetry policy. The two content-capture flags keep message bodies out of exported OpenTelemetry and MLflow spans by default (7.5. Online Evaluation). They do not change what ADK or the A2A server writes to runtime.db.

ADK stores complete event JSON in events.event_data. It can include user messages, model responses, tool calls, tool responses, and state changes. The A2A task store writes status, artifacts, and history as JSON, so request and response content can appear there too. The sessions table also holds the user id and persistent session state.

The repository does not ship a subject-wide runtime erasure workflow. The pinned SDKs expose low-level delete_session(...) and task-store delete(...) methods. The server exposes no protected operator command that maps a subject to every session and task, removes both, and verifies the result. Until that workflow exists, session and task erasure is an operator gap.

The audit log has no erasure path either: its triggers reject application DELETE operations (above). That technical choice does not decide whether retaining a row is lawful. The controller must decide how access, retention, exceptions, and erasure apply under its jurisdiction and purpose.

Three honest limits:

• The operator, not the agent, runs the only shipped erasure command. The model cannot delete long-term memory itself.
• SDK deletion primitives are not a subject-wide operator procedure. The repository still needs identity mapping, authorization, orchestration, and verification before it can claim runtime erasure.
• This is an engineering inventory, not legal advice. The controller decides what rights apply and whether any record may or must be retained.

Who can still alter the evidence, and what stops them?

The schema stops the application from rewriting history. It does not stop anyone who can reach the database file.

A control is governance evidence only when you can name its owner and its bypass. Here is the honest ledger for the audit trail.

Defends it today:

• Append-only triggers reject UPDATE and DELETE at the schema, as shown above.
• The mutation and its audit row commit in one BEGIN IMMEDIATE transaction in data.py, so there is no window with a state change and no record — the atomicity 3.1. Tools owns.
• Every workload ServiceAccount sets automountServiceAccountToken: false (including the backup job), so a compromised pod holds no Kubernetes API token to reach the PVC, secrets, or other workloads through the API server.
• MCP and the backup job mount the state PVC readOnly: true; only the agent mounts it read-write, so the write surface is exactly one workload. Agent startup creates /app/state/.state-restore.lock; backup, restore, and crash recovery all take flock(2) on that one PVC inode. A backup fails closed if startup has not initialized the lock.
• Telemetry content capture stays off, model and tool callbacks redact their controlled paths, and readOnlyRootFilesystem with dropped capabilities hardens each pod.

Does not defend it:

• No WORM store (write once, read many: storage that refuses overwrites) and no external append-only store, so a storage or database administrator who reaches the file rather than the schema can replace or re-create it.
• No integrity hashing or signing of rows, so a rewrite leaves no detectable gap.
• No independent access log — nothing records who read or replaced the SQLite file.
• Runtime session ingestion happens before model-request redaction. The content-capture flags do not stop runtime.db from storing raw messages and A2A content.
• The backup is single-cluster crash recovery restored by restore-state.sh, not tamper-evident or off-cluster archival.

Production evidence needs five things:

1. restricted database administration;
2. external append-only/WORM storage where required;
3. row integrity verification — hash chaining (each row carries a hash of the one before, so a removal shows a gap) or signing;
4. independent access logging;
5. a tested restore path.

Name which of these you have before you call the log "audit-grade".

How is cloud authority separated?

On the optional GKE path, each workload carries its own identity and no long-lived JSON key. One compromised workload's blast radius — how far the damage reaches — therefore stops at that workload.

What evidence belongs in a release decision?

A release decision is itself a governance act: someone accepts residual risk on the record.

The full reproducibility tuple — code, dependencies, image, model path, prompt, data, tool contract, runtime, evaluation — is 7.0. Reproducibility's lane, and its one-pass commands capture it. Do not re-list it here. This page's addition is the authority-and-retention layer on top of that tuple:

• Who approved the release and which residual risks they accepted, recorded rather than implied.
• The audit-and-trace correlation for any guarded action taken during validation — session and invocation ids agree across the row and the MLflow trace.
• The retention and backup state of that evidence: which pillars age out in 7 days, which persist for PVC life, and when the last snapshot ran, so an approver knows how long the decision stays reconstructible.
• The rollback and teardown path, including that skaffold delete removes both PVCs and that restore-state.sh restores a snapshot once every writer is stopped, with no cluster-level disaster-recovery runbook.

What proves this page worked?

Approve one mock action, then verify that the action result, the audit row, and the MLflow trace agree on the same session and invocation ids. Attempt to rewrite an existing row and confirm the trigger aborts it — an identity-only update needs no string literal and still fires BEFORE UPDATE:

kubectl -n agentops exec deploy/agentops-agent -- \
  python -c 'import sqlite3; db=sqlite3.connect("/app/state/incidents.db"); db.execute("UPDATE audit_log SET id = id WHERE id = 1")'

That raises audit_log is append-only instead of succeeding. Finally, document who can still alter the SQLite file or the PVC, and how long each pillar of the evidence survives — the retention table and the ledger above are the answers to write down. A control is governance evidence only when its bypass, its owner, and its retention are all understood.

You are done when:

• A SELECT on audit_log returns a row whose approved_by, rationale, and context_summary are all filled in.
• That row's session_id and invocation_id match the ids on the corresponding MLflow trace.
• The UPDATE above fails with audit_log is append-only rather than succeeding.
• You can name one control that defends the audit trail today and one bypass that nothing in the course covers.
• You can say, per pillar, how long the evidence survives: metrics, logs, traces, runtime sessions and tasks, audit rows, snapshots.

Continue to 7.7. Incident Response when you can point at the row for a change you approved, say who authorized it, and admit who could still rewrite it.