0.2. Evidence

What a gate proves, and what an observation only suggests

A gate is deterministic: same input, same verdict, every time, on your machine and in CI. mise run check is one. A schema validator is one. So is the test showing that a mutation and its audit row commit together. When a gate is green, the thing it checks is true — not likely, not usually. True.

An observation measures something that varies: an evaluation pass rate, a judge’s agreement number, a latency figure. When an observation is good, you have learned something real about one run, on one machine, against one model, at one moment. That is worth having. It is not the same currency.

The two get confused because release decisions are made on mixed evidence: a passing suite and a good evaluation score arrive in the same summary wearing the same colour, and only one licenses “this is safe to ship”. Treating an observation as a gate is how a team ships behavior nobody proved.

Gate versus observation is the first of this page’s five distinctions; the other four are below.

What the coverage floor proves, and what it cannot

mise run test inside agents/go runs the suite under Go’s race detector, then checks the coverage profile it just wrote, package by package. The race detector fails the run on an unsynchronised read and write, instead of letting that bug surface later under different timing. Below is the tail of that run — the test results scroll past first — trimmed to six of the twenty package lines, plus the verdict:

cd agents/go
mise run test

  ok      84.2%  agents/go/a2aserver
  ok      91.7%  agents/go/compose
  ok      99.4%  agents/go/domain
  ok     100.0%  agents/go/internal/safefile
  ok      90.9%  agents/go/policy
  ok      82.7%  agents/go/state
agents/go meets the 80% per-package coverage floor

Take the 100.0% row. It entitles you to one claim: every statement in internal/safefile ran at least once while the tests executed. That is all. It says nothing about whether the assertions around those statements checked anything worth checking — a test that calls a function and confirms it did not panic raises coverage and settles nothing. Coverage measures reach; correctness is decided by what the tests assert. So the review question is “which test fails if I break this”, never “what percentage is that package at”.

The floor is per package rather than per module because a module-wide average lets a well-tested package carry an untested one.

The floor’s scope is narrower than it looks. Two Go modules carry it: agents/go, the code the course teaches from, and evals, the harness that drives the agent from outside and scores what comes back. cmd/ packages are excluded by kind in both: they are package main composition wiring — flag parsing, dependency construction, process lifecycle — that this project has chosen not to hold to the floor. Their coverage is still measured, and it simply sits below it. Notice what that exclusion is and is not: a decision recorded in scripts/check-coverage.sh, not a property of Go, and the honest way to read it is “nobody promised this number”, never “this code is covered elsewhere”. The repository’s own tooling module carries no floor, and says so in a comment.

Three ways a green result claims more than it proves

Temperature zero is not determinism. Setting temperature to 0 narrows a model’s output distribution; it does not make a run reproducible. The same prompt against the same model can still take a different route through the tools, and certainly will after a model update, a context change, or a different machine’s floating-point behavior. That is why mise run eval samples every case three times: repeats do not remove variance, they let you see it. Three that agree say the case is stable; three that disagree make the pass rate a range, and the range is what you report.

A simulation is not production, and the local stack is honest about it. The k3d cluster (Kubernetes inside Docker on your laptop) and the seeded incident database are real software doing real work, against a fixed dataset, with no noisy neighbours, no contention, and no traffic you did not send. That tells you the mechanism is correct. It cannot tell you how the mechanism behaves at your scale, with your data, under your failure modes.

Silence is not approval, and it is the subtlest of the three. A green offline suite says nothing about whether the agent answered well, because it never called a model; a passing conventions check says nothing about whether this page teaches anything. That kind of false claim is almost never stated out loud — it arrives as a green tick left standing next to a question nobody asked. So this course names the command behind anything it calls proved, and says plainly when there is none.

How later pages state a caveat once and link here

Every later page states its point plainly and, where a caveat matters, links here instead of restating it. That is a deliberate trade: repeating a warning on forty pages trains you to skip it, and a skipped warning is worse than none, because it lets everyone believe the risk was communicated.

So take a passed gate literally, notice what varied in a reported observation, and treat any claim with no command behind it as the page’s opinion — including this one.

What you can do now

• You can say what a gate proves and what an observation suggests, and what a release decision is allowed to rest on.
• You can read a coverage percentage as reach rather than correctness, and name the two modules the 80% floor actually covers.
• You can name the three ways a green tick overreaches: variance the repeats reveal, a simulation standing in for production, and silence read as approval.

A green checkmark used to be a green checkmark. You can now take any claim, here or in your release channel, and ask the one question that settles it: which command would have to fail for this to be false? When no command would, you are holding an opinion.

Continue to 0.3. AgentOps, which turns this epistemology into the six lifecycle phases the chapters are built from.