4. Quality

Three failure modes no single gate catches

Agent quality is several unrelated properties, not one. Values crossing a boundary hold their shape, the code means what it says, behavior survives hostile input, and a reported number means what you take it to mean. No compiler, test suite, or dashboard proves more than one.

Chapters 2 and 3 gave the agent a conversation and capabilities. Neither establishes that it behaves when its inputs are hostile, nor that a green result means what it appears to mean. So this chapter is seven layers ordered as a cost curve, each catching failures the cheaper layers cannot see, plus three offline commands covering all seven.

Three examples make that concrete. A value that compiles: a string holding INC-2; DROP TABLE incidents. Neither the compiler nor a test that never thought to try it objects. A value that lies: a confident paragraph with nothing underneath it, indistinguishable, from the outside, from the grounded one in 2.1. First Agent. A run green for the wrong reason: a suite that passes because its interesting case was never written, or a pass rate that clears its floor while the safety case fails.

Seven layers, and the failure each one catches

The first three layers are free, offline, and instant. Only 4.3 and 4.4 buy answers a running model alone gives; the last two return offline, apart from an optional gateway run and the scanners’ network.

• 4.0. Type Safety (concept · offline): a hostile tool argument dies at the boundary that owns it, instead of reaching the database.
• 4.1. Linting (hands-on · offline): an ignored error in a transaction — a bug that compiles perfectly — is caught by a machine.
• 4.2. Testing (hands-on · offline): the model is replaced and everything else stays real, so most of the agent is decided in seconds.
• 4.3. Metrics (reference · needs a model for the live rows): a scorecard where every number names its command, candidate, and proof class.
• 4.4. Evaluations (hands-on · offline validation, model-backed runs): black-box turns over the wire, scored by structure first and by a judge only for the residue.
• 4.5. Guardrails (hands-on · offline, except the optional gateway run): injected text is defused, personal data is masked twice, and no write lands without confirmation and one transaction.
• 4.6. Security (hands-on · offline; scanners may use the network): authority the agent never holds cannot be misused, and a suite proves nobody quietly added it back.

Every Your turn here is optional and self-contained: no later page’s prerequisites name what one leaves behind, so read the page, take the code, and skip the exercise if you came for the argument. Three break something on purpose and restore it — 4.1. Linting, 4.5. Guardrails, 4.6. Security — and three leave a case you keep: 4.2. Testing, 4.4. Evaluations, and 4.5 again.

One distinction runs through all seven pages: a gate is deterministic, same input, same verdict, every time; an observation measures something that varies, such as a pass rate. 0.2. Evidence owns it; each page links there rather than re-deriving it.

Three offline commands cover all seven layers

Three commands, run from the repository root, none needing a model, provider key, or network:

mise run test
mise run redteam
mise run eval:validate

mise run test is the widest: it fans out to all three Go modules — the agent in agents/go, the harness in evals, the maintainer tooling in tools — and two of them end by enforcing a coverage floor. mise run redteam runs the deterministic adversarial regressions in the agent’s policy package; mise run eval:validate proves the committed evalsets against the seed, the committed incident dataset.

[test:tools] DONE 309 tests in 24.858s
[test:go] DONE 1815 tests, 1 skipped in 21.280s
[test:go] agents/go meets the 80% per-package coverage floor
[test:evals] DONE 395 tests in 27.561s
[test:evals] evals meets the 80% per-package coverage floor
Finished in 40.52s

Six lines from that run, in the order they arrived. The three suites run concurrently, so the full log interleaves them and prints a coverage line per package; tools deliberately sits outside the floor.

What this chapter proved

• Just over twenty-five hundred tests decide the agent, the evaluator, and the repository tooling under the race detector in well under a minute, with no service to start and nothing spending a token.
• mise run redteam settles the adversarial policy cases and mise run eval:validate proves the evalsets against the seed, both offline.
• By the end you can take any green result in this chapter and say whether it is a gate or an observation — and name the command behind it.

The guardrails were already there; the agent refuses hostile input either way. These seven pages let you prove it in a second, before the agent sits behind a gateway any local process can reach.

Continue to 4.0. Type Safety; the first three pages need no model and no account.