3.9. Incident Run
In one glance
- You will: Put the agent you built through one incident, then read the turn back call by call and say, for each, whether the model decided it or the code did.
- You need: A model served and configured in the root
.env, andcd agents/go && mise run testgreen. If you did the 3.1. Tools and 3.2. Skills exercises, your own tool and skill ride along here; if you skipped them, the six shipped reads and the two shipped skills carry all five requests. - Time: about 45 minutes, hands-on — the reading is fifteen, and five grounded turns are the rest.
What one live session shows that the offline gates cannot
The default composition is what the agent entrypoint builds: one agent holding the read and action tools, the memory tools, and the skill toolset. mise run web starts it with AGENT_ENTRYPOINT=agent, so this session reaches 3.1. Tools, 3.2. Skills and 3.4. Memory, and no further.
Every gate in this chapter so far was decidable without a model, and offline is where those belong. But no offline check shows which tools a model reaches for. A call the model chose and a constraint the code imposed read identically in the answer; in the event log they are separate rows.
So this page runs five requests, reads the event log call by call, stops at the refusal this surface returns instead of a write, and ends with nothing in the audit log. The run is INC-002, open since 2.1. First Agent — the inventory service crash-looping, and back then an incident the agent could only read.
The rest of the chapter lives behind other entrypoints and switches, each one command away: mise run workflow builds the bounded investigation from 3.5. Workflows, mise run coordinator builds the delegating specialists from 3.7. Multi-Agent, mise run a2a serves the network agent from 3.6. A2A and carries the token stream from 3.8. Streaming when AGENT_A2A_STREAMING=true, and setting AGENT_MCP_URL moves the six reads to the server from 3.3. MCP. Run the single agent first: an undelegated trace reads easiest.
Start the developer UI. The console prints only the final answer; the UI shows each tool call as a row:
cd agents/go
mise run webOpen http://localhost:8002, select agentops_agent, and ask one request at a time, not one paragraph holding all five, so each gets its own turn to read. Each elicits a different kind of call:
- Ask what is open right now — a list read.
- Ask what is wrong with the inventory service specifically — one exact record read.
- Ask what has already been tried — a memory recall, not a database read.
- Ask what to do about it — a runbook search, then a skill load: your own “Use when…” clause if you wrote one, otherwise the shipped
remediationskill, whose clause covers asking how to fix a known incident. - Ask it to do that thing — a write proposal, which is not a write.
Give each turn room. On a CPU-only host a grounded turn runs several multiples of the inference line mise run doctor:model printed, and 1.4. Providers owns the budget — AGENT_MODEL_TIMEOUT_S, which names itself in the failure when a turn never returns.
How the Events view separates model choices from code constraints
The chat pane shows prose. The Events pane shows the turn as rows: a functionCall is the model’s request for a named tool, a functionResponse is what the Go handler returned, and the final model message is built from both. Telling the two call kinds apart is the skill worth leaving the chapter with:
- A call the model chose.
list_incidentswith a status filter,get_incidentwith an identifier,search_runbookswith words from the symptom,load_skillwith a name — each is the model reading the situation and picking. Nothing in the code says to call them, and a different model would pick differently. - A call the code forced, or refused. The refusal of
restart_serviceis no decision of the model’s, and not one it can skip: confirmation, a named approver, and a bounded rationale are structural preconditions of the write. The model can ask for the write. It cannot perform one.
The page stops here, one step short of the write — and on this surface the code stops one step short of the pause. mise run web starts ADK’s launcher family, which agents/go/cmd/agent/launcher_plan.go runs with writes frozen no matter what your .env says, so the policy plugin in agents/go/policy/guardrails.go refuses the guarded call in its before-tool hook, ahead of the tool body that would have requested a confirmation. No card is built and there is nothing to approve, which is the point of a kill-switch: it must not page a human for an approval it would reject anyway.
So the two rows to read are the functionCall and the functionResponse under it. The call names restart_service and the exact target the model proposed — the entire thing an approver would be signing on a surface where approving is possible. The response is one sentence: Writes are frozen by the AGENT_WRITES_DISABLED kill-switch; reads still work. Clear the flag once the incident is contained to resume approvals. The model then relays some version of it in prose, which is exactly why the Events pane is the thing you read.
The approval would be unavailable here for a second reason, one that outlives the freeze. 3.1. Tools requires a rationale from the approver, bounded and redacted, before the guard will run anything, and the dev UI’s confirmation dialog answers with {confirmed, payload} — the payload being the arguments the model already sent, with no field to add to them. A click is not attribution. The one shipped surface with a field for a rationale is the browser client in clients/web/index.html, which 5.3. A2A Gateway drives against mise run a2a, where writes are not frozen and an approval can actually be attempted.
Check the other half of the contract — that the request alone wrote nothing:
ls .stateA write is the only thing that publishes the runtime copy of the incident database into .state, on the first one and never before, which agents/go/data/store.go owns; every read answers from the committed seed until that happens. So a session that never got past the freeze leaves no .state/incidents.db at all, and its absence is the receipt: the audit log has no runtime copy to grow a row in. What .state does hold is runtime.db, the session store this run created for the conversation.
If an earlier page already published that copy — mise run a2a prepares it during startup, so 3.6. A2A leaves one behind — query it and read the same answer in rows:
sqlite3 -header -column .state/incidents.db \
'SELECT schema_version, actor, approved_by, action, target FROM audit_log ORDER BY id DESC LIMIT 5;'Those five columns are what a completed write leaves behind: row format, executing agent, approver identity, action, and target. No rows is the correct result: the agent asked for a restart, the surface refused before anyone could approve, and the table records exactly the writes that happened. The approval 5.3 sends with a rationale still fails closed one precondition later — a guarded write over a network boundary also needs a caller that some boundary authenticated, which the plain gateway profile does not supply. 5.5. Gateway Security is where that piece arrives — a JWT the gateway validates and rewrites into the header the agent accepts as an approver — and mise run eval is where the whole approval loop, audit row included, runs today.
Your turn: run INC-002 end to end and name every call
Predict first, before you open the browser. Across those five requests, how many Events rows will be calls the model chose, and how many things the code required? Write both down, then count.
- Mode:
inspect— the session writes runtime state underagents/go/.state, which is already gitignored, and touches no tracked file. - Goal: drive one incident through the default composition, and account for every call in the event log by who decided it.
- Files to touch: none.
- Preflight:
cd agents/go && mise run testgreen, andmise run doctor:modelreporting a measured inference line. - Steps: start
mise run web, work the five requests above in order, then open the Events pane and list everyfunctionCallin the order it fired. Beside each one write model or code, and for the ones you marked model, name the words in your question that made it pick that tool. The fifth request ends at a refusal rather than at an approval, so read the row instead of looking for a button. - Gate that proves completion: your list accounts for every
functionCallrow;load_skillappears on the fourth request, and you can point at the “Use when…” clause explaining the name it carried — your own if you wrote one, otherwise the shippedremediationdescription; thefunctionCallforrestart_servicenames the exact target the model proposed and thefunctionResponseunder it is theAGENT_WRITES_DISABLEDrefusal rather than a confirmation request; andls .stateshows noincidents.db, or thesqlite3query above returns no rows if an earlier page published one. - Final state: stop the server. Runtime state stays under
agents/go/.state;git status --shortreports nothing.mise run data:resetdrops that state back to the committed seed for a clean run.
Where your prediction and your count disagree, the gap usually runs one way: people over-count what the model decides, because the answer reads like one act of reasoning, and under-count the constraints, because a working constraint is invisible.
What you can do now
- You can list what
AGENT_ENTRYPOINT=agentcomposes — read and action tools, memory tools, the skill toolset — and which request reaches each. - You can name the five capabilities that session never touched — MCP, the workflow, A2A, streaming, and delegation — and the command or variable that reaches each one.
- For every call in the event log, you can say whether the model chose it or the code required it, and point at the file that decides.
- You can predict an empty
audit_logafter a write request, because the developer UI runs with writes frozen and refuses the call before any approval exists — asking and performing stay different events.
Continue to 4. Quality, where the capabilities you have been adding start having to prove they behave.