7.7. Incident Response

The agent is a workload with incidents of its own

An agent-platform incident is one where the failing workload is the agent itself rather than the service it was investigating. The operational answer to it is a fixed loop: detect → triage → mitigate → review → prevent. The agent has needed that loop since Chapter 5, when it became a workload of its own — a host process behind the gateway, or a Kubernetes deployment after Chapter 6. The signals the previous pages built make the agent’s own outages legible, including the one where it answers perfectly and your ability to see it is gone.

This page runs that loop once: cause two failures deliberately, watch one refuse itself and one get detected, walk the four triage signals, pull one mitigation lever, then turn the incident into a check that catches the next one without you.

Cause one now; the walk below reads better against a live symptom. Stop the collector for three and a half minutes:

CHAOS_HOLD_SECONDS=200 mise run chaos:collector

[chaos:collector] $ ./scripts/chaos-drill.sh collector
 Container agentops-observability-otel-collector-1 Stopping
 Container agentops-observability-otel-collector-1 Stopped
otel-collector stopped for 200s; watch ObservabilityCollectorDown move pending -> firing

That terminal now holds. Two minutes later, ask Alertmanager what it has:

curl -s 'http://localhost:9093/api/v2/alerts' \
  | jq -r '.[] | "\(.labels.alertname) \(.labels.severity) \(.status.state)"'

ObservabilityCollectorDown page active

Keep sending the agent questions while that holds and it answers every one. Nothing is wrong with the agent; what is wrong is that you have stopped being able to see it. The severity on that line is page, the label reserved for symptoms that interrupt whoever is on call instead of filing a ticket. The workload you were using to run the loop has moved inside the loop.

Nine shipped rules watch for symptoms that mean the agent itself is misbehaving rather than the platform it watches. Each previous page owns one of those signals; this page joins them into the loop.

    flowchart LR
    Detect["Detect<br/>alert fires (7.2)"] --> Triage["Triage<br/>trace + metrics + logs + audit"]
    Triage --> Mitigate["Mitigate<br/>config rollout or prior image"]
    Mitigate --> Review["Review<br/>blameless postmortem"]
    Review --> Prevent["Prevent<br/>add test, eval, alert, or baseline"]
    Prevent -.->|"automated evidence catches recurrence"| Detect
  

Diagram in words: An alert starts the loop; triage joins the trace, metrics, logs, and audit row; mitigation is a configuration rollout or a prior image; review produces a blameless postmortem, one that accounts for what the system did rather than assigning fault to whoever shipped the change; prevention adds the check that closes the loop by catching the next occurrence automatically.

The last step is the one people skip. An incident is not closed when service is restored. It closes when the cheapest appropriate layer would expose it again — CI for deterministic behavior, a scheduled model run for generative behavior, a runtime drill for an alert. That is what keeps the AgentOps loop from leaking.

Every one of those rules maps to a first signal and the page that owns it:

Only two of the nine carry severity: page. The three cost rules file tickets on purpose, because a bill is a working-hours decision rather than an outage. One class has no rule at all: a quality incident — a cluster of thumbs-down feedback or judge disagreement on answers that are wrong rather than merely terse — is a judgement no threshold expresses. Treat it as a first-class incident anyway, because nothing will raise it for you.

Your turn: induce two failures and time the recovery

You have already run half of this. Do it again with a clock, because the times you write down become the postmortem’s timeline. The first failure is fail-closed: a limit that refuses the call rather than letting it through once it cannot be satisfied, enforced by the agent on itself. The second is an outage a shipped detector finds without you. You will send two turns under that limit; predict which one it refuses.

• Mode: inspect.
• Goal: observe one fail-closed application limit and one real alert lifecycle, then return every process and setting to its starting state.
• Files to touch: none. The token limit is scoped to one foreground command; do not edit the root .env.
• Preflight: from the repository root, require docker compose -f infra/observability/compose.yaml ps --status running --services | grep -Fx otel-collector to succeed before stopping anything.
• Steps: induce the application symptom first, noting the wall-clock time of the refusal, then induce the pipeline outage and note when the alert changes state. The two paragraphs below walk each one; the next section reads the signals they produce, and the one after it turns your two times into a postmortem timeline.
• Gate that proves completion: the second agent turn returns TOKEN_BUDGET_EXHAUSTED, and ObservabilityCollectorDown moves through pending, firing, and resolved.
• Final state: stop the foreground agent, verify otel-collector is running again, then run unset ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS OTEL_TRACES_SAMPLER to close the trace switch you accepted back in 7.1. Tracing, and use the page teardown to stop the stack. No environment or tracked-file change remains.

The application symptom. Scope AGENT_MAX_TOKENS_PER_SESSION=1 to one foreground process, then start a session and send two turns:

cd agents/go
AGENT_MAX_TOKENS_PER_SESSION=1 mise run run

The first turn still calls the model and answers, because a fresh session has spent nothing yet. The second is refused before any model call, with ErrorCode: "TOKEN_BUDGET_EXHAUSTED" and a message naming the variable to raise. That refusal is your induced capacity incident, exactly as 7.3. Costs describes it.

The detected outage. Nothing pages you for that refusal: a budget refusal is an application decision, not an outage, and the three shipped cost rules watch the gateway’s spend rather than one session’s cap — they file tickets when they fire, as 7.3b. Cost Governance sets out. An incident you notice only because you happened to be watching is not a detection story, so run the collector drill again and watch the whole arc. Open http://localhost:9090/alerts and note the wall-clock minute the rule crosses from pending to firing. When the hold expires the task restarts the collector, interrupting it runs the same restore trap, and the alert clears by itself a scrape or two later — note that minute too. Traces, metrics, and logs for the window in between are gone permanently, because nothing backfills them.

You broke your own platform on purpose, watched a shipped detector find it, and timed the recovery.

The other chaos drills isolate different seams:

mise run chaos:database # corrupt and reject a temporary SQLite copy
mise run chaos:runbook  # poison and neutralize a temporary runbook copy
CHAOS_HOLD_SECONDS=30 mise run chaos:mcp # k3d-local only; restores the exact replica count

The database and runbook modes never alter committed seed or runtime state. The MCP mode refuses every context except k3d-local, records the existing replica count, and restores it on exit; it exercises the gateway’s fail-closed tool path and claims no shipped MCP-down alert.

Triage across the four signals, then pull one lever

Triage is a fixed walk across four signals — the three pillars plus the audit trail — not a hunt. They join on the trace id, so each step narrows the next. Start with the metric to confirm the alert is real and scope it: is the error ratio or p95 rising for every turn or one tool? Then open one failing turn’s trace in Grafana’s Tempo view and read the span tree — which model or tool span failed, retried, or ran long, and what were its token counts? Follow that span’s logs link into Loki for the error text the span only summarizes; under pressure, a provisioned link beats a copied identifier. Finally, if a guarded remediation changed state, read the append-only audit row: who approved which action, with what rationale, against which incident.

While you walk, the blast radius is already bounded. Every guarded remediation required human approval. Durable memory notes are a separate PII-redacted write that the kill switch also freezes, and they do not mutate the mock service or incident tables. That limit is owned by 4.5. Guardrails.

Most agent incidents are then mitigated with existing configuration or a prior image. None of these flags hot-reloads: config.Load() reads the environment once at startup, so changing one means restarting the host process or rolling out the workload configuration.

Freezing every write is the fastest application-level blast-radius cut, which is why it is first. The last row is the exception: its Setting names a deployment you perform, not a flag you set. Record the configuration change and the moment the replacement process became ready, because those two timestamps start the postmortem. Note what is not in the table: there is no mutable prompt-registry seam, so a prompt rollback means redeploying the prior evaluated image digest and its matching Git revision, which is the instruction authority.

Write the postmortem, then add the check that catches it

Keep the postmortem short, factual, and blameless. Five sections carry it: impact (what degraded, for how long, for whom), timeline (the alert, the trace, the lever, each with a time), root cause (the one change or condition, tied to a trace id or an audit row), what caught it and what didn’t, and actions (one owner and one check per line).

The fourth section is the one that matters. An incident a cheaper layer should have caught is really a gap in that layer, and naming it is how the layer improves.

# Postmortem: <short title> (<date>)

## Impact

What degraded, for how long, and for whom.

## Timeline

- HH:MM AgentTriageSchemaFailures fired
- HH:MM Trace <id> showed the model omitting a required field
- HH:MM Redeployed the previous known-good image digest; alert cleared

## Root cause

The one change or condition that produced the impact, tied to something checkable: trace ids, the audit row, the fired alert, the release version.

## What caught it, and what didn't

Which signal surfaced it, and which check should have exposed it earlier but did not.

## Actions (owner, check)

- [ ] New eval case reproducing the failure on seed data — <owner>
- [ ] Alert / test / token-drift observation that would catch a recurrence — <owner>

Then pick the cheapest layer that would have caught the failure, reusing machinery from earlier chapters:

    flowchart TD
    Incident["Resolved incident"] --> Which{"What would catch it again?"}
    Which -->|"wrong/unsafe answer"| Case["New sanitized eval case<br/>scheduled model evidence"]
    Which -->|"missing health signal"| Alert["New Prometheus rule<br/>CI syntax + runtime drill"]
    Which -->|"token/cost blowout"| Cost["Fix or approve cost change<br/>then review baseline evidence"]
    Which -->|"deterministic boundary failed"| Test["New redteam/unit case<br/>pull-request gate"]
    Case --> Review["Exact-head release review"]
    Alert --> Review
    Cost --> Review
    Test --> Review
  

Diagram in words: A resolved incident routes to one of four preventions by class — a sanitized eval case for a bad answer, a Prometheus rule for a missing health signal, an explained cost change for a token blowout, and a red-team or unit case for a deterministic boundary — and each one feeds an exact-head release review, which counts a check only when it is green on the exact commit being shipped.

A quality incident becomes a sanitized eval case: CI validates its structure, and a scheduled model run supplies the behavioral half. A missing health signal becomes a Prometheus rule: CI validates the manifest, and a runtime drill shows the rule firing. A cost incident is explained before it is accepted — an eval run compares its token total with the previous run of the same evalset and model and only warns when drift passes 25%, so the fix is an account of what changed rather than a quiet re-run that makes the warning stop. A deterministic failure becomes a unit or red-team case that blocks pull requests.

Reproduce the behavior on the committed seed with cd agents/go && mise run data:reset, an agent-module task that errors from the repository root. Never paste a real production conversation into the repository; transcribe the behavior onto sanitized seed data, exactly as the feedback loop prescribes. Close the incident only when the relevant check is green on the exact main commit, and do not call a scheduled eval or a runtime alert a pull-request check when CI only validates its source.

Stop any foreground agent with Ctrl-C, put the ADK trace switch back where the repository ships it with unset ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS OTEL_TRACES_SAMPLER, then stop the stack without deleting its volumes:

mise run observability:down

What you can do now

• You can tell a refusal from a detection: TOKEN_BUDGET_EXHAUSTED is the agent policing itself, ObservabilityCollectorDown is a detector finding an outage without you.
• You can walk metric → trace → log → audit row for any of the nine shipped alerts, and pull one configuration lever without writing code.
• You can write a five-section postmortem, and read its fourth section as a gap in a cheaper layer.
• You can route a resolved incident to its cheapest check, and close it only when that check is green on exact main.

That last bullet is the chapter’s point: a check that never gets tired is the difference between running an agent and operating one.

Continue to 8.7. Capstone, which replaces the reference domain with one of your own, one vertical slice at a time, so this loop runs on a system you operate.