4.3. Metrics

A scorecard versus a dashboard for an agent’s quality signals

A scorecard is a record where every number names the command that produced it, the candidate it describes, and the proof class it belongs to. A dashboard names none of those. Availability, latency, and error ratio describe the pipe, not what went through it: an agent can answer HTTP 200 as fast as ever and be wrong with every service metric flat, because nothing in the service failed. In the reference agent, every triage answer since the last deploy recommends a runbook for a service missing from the seed data.

No single number summarizes agent quality honestly, because five classes of signal contribute to it:

• Deterministic software correctness — same input, same verdict.
• Model behavior over fixed cases — scored answers to cases written in advance.
• Cost and call usage — tokens and model calls per answer.
• Runtime RED signals and guardrail counters — request rate, errors, duration, refusals.
• Human review of the residue — what the other four cannot decide.

Some of those five have a command behind them:

mise run --cd agents/go coverage

github.com/MLOps-Courses/agentops-open-course/agents/go/tools/tools.go:294:			newTool				80.0%
github.com/MLOps-Courses/agentops-open-course/agents/go/tools/tools.go:327:			describeProperties		100.0%
total:												(statements)			87.3%

Two function lines out of several hundred, plus the total, and anyone holding this repository can reproduce it. A number recalled from a dashboard cannot, because nothing recorded what measured it. This page sorts the agent’s signals along that gap, then fills one row from a command.

Which signals gate a merge, and where each threshold lives

A gate is deterministic enough to block a change and to name the defect it found. Evidence informs a decision but varies between runs or needs infrastructure that may simply be absent. 0.2. Evidence owns that distinction; this page only assigns each of the agent’s signals to one side of it.

Commit-scoped evidence comes from running one named candidate, so it keeps its meaning as long as you record the commit or image it describes. Runtime evidence comes from a deployment serving traffic, so it needs a time window and is missing when no collector is running.

Two mistakes cross that line in opposite directions, and both are common. Treating stochastic output as a merge blocker teaches the team to re-run the job until it is green. Reading a missing runtime observation as a code failure sends someone to “fix” an agent that was never broken, only unobserved.

Every gate needs a threshold, and it must sit where a reviewer can find it. In this repository the rule is deliberately boring: only a threshold encoded in a task, a manifest, or an alert rule is policy. Every evaluation threshold sits on the eval task’s command line in evals/mise.toml, and mise tasks info eval prints that definition straight from the file, so its output is the policy:

cd evals
mise tasks info eval

Run:
  go run ./cmd/agentops-eval run --evalset ops.evalset.json --entrypoint agent --transport a2a --repeat 3 --min-pass-rate 0.33 --required-cases investigation-recalls-context,remediation-loads-skill,restart-needs-approval,resolve-needs-approval,restart-approval-verified --require-grounded --judge --output results.json

Trimmed to the Run: block; the command also prints the task’s description, source file, and usage spec. --evalset ops.evalset.json names the evalset, the committed file of scored cases. --repeat 3 takes three samples of each case, because a model’s answer varies between identical runs. --min-pass-rate 0.33 floors the pass rate: the share of samples accepted by a judge, a model scoring answers against written criteria. --required-cases names five cases that must pass in every single sample rather than on average, and --transport a2a decides which surface answers them, because three of those five propose a guarded write and only the deployed contract may carry one. That is the release policy for this agent’s behavior, and changing any of it is a diff someone reviews. The Go coverage floor qualifies the same way: the 80 sits on the test task’s command line in agents/go/mise.toml and evals/mise.toml, which is why mise run test fails any package under it.

Two numbers are deliberately measured and not enforced. mise run eval:judge-calibration prints how often the judge agreed with a human label and exits zero either way, and a run whose token total moves more than 25% from the previous run prints a warning without failing. A number that gates is a number someone eventually tunes until it passes.

Keep the two roles of coverage apart: the enforced 80% floor belongs in the policy column, the measured 87.3% in the scorecard with its command and date.

Which application metrics ship?

The Go agent emits a small, bounded set of custom signals — bounded because each has to stay cheap to store and query:

All five export over OTLP only when a metrics endpoint is configured, and Prometheus renames them: query agentops.tokens as agentops_tokens_token_total.

Session token totals and an estimated cost also travel as span attributes, with prices supplied by configuration and defaulting to zero for local models. Request rate, errors, and duration are derived by the collector from spans, so do not re-implement them in application code and then wonder which to believe.

Keep evaluation and application signals apart, bounded, and paired

The evaluator emits its own family — agentops.eval.score, agentops.eval.case.passed, agentops.eval.tokens, agentops.eval.model_calls, and agentops.eval.run.passed. Each is attributed by run, commit, model, evalset, transport, case, sample, score, and pass state. Keep that family separate from the application one above: one describes a service answering live traffic, the other an experiment scoring a fixed set on whatever day somebody last ran it. Merge them into one “agent quality” panel and it reads 99.4%: a service success ratio wearing a quality label, over an evalset nobody has run in days, in front of a room deciding to release.

What never becomes a metric dimension is equally deliberate: prompt text, answer text, user, session, tool arguments, and trace id. Unbounded label values multiply series, wreck query performance, and quietly turn your metrics backend into an unsanctioned copy of your users’ data. Segment on bounded operational dimensions — operation, model, error type, transport, score name, deployment — and reach for traces to investigate one specific turn.

A useful SLO needs four things and nothing else: one user-visible outcome, an indicator, a target window, and a named owner. “Successful read-only A2A requests under a stated latency objective over a stated window” is a contract; “the agent is fast” is a mood. This course ships demonstration alerts and budgets, not a production SLO contract: a real target comes from your traffic and capacity. Do not give a quality score an availability objective’s job: alert whoever is on call because last week’s evalset scored 0.62 and they will spend an hour finding nothing wrong, because nothing is wrong now. Release evidence and runtime objectives answer different questions on different clocks.

Pair each metric with the one that punishes gaming it. Lower model calls against trajectory and groundedness. Higher pass rate against required-case status and a stable evalset digest. Lower cost against answer and schema quality. Lower latency against error ratio and cancellations. Fewer guardrail rejections against adversarial regression cases. Whoever fills the row records both halves, so a cost that fell while schema failures rose reads as one finding. If changing only the measurement can improve the score, the scorecard is not finished.

Your turn: fill one scorecard row from a real command

A scorecard is useful the moment one row is real.

• Mode: inspect — you run read-only commands and write down what they printed.
• Goal: produce three scorecard rows where each number names its command, candidate, and proof class, and mark every row you cannot measure as unknown.
• Files to touch: none. Keep your notes outside the repository.
• Preflight: mise run --cd agents/go test and mise run --cd evals eval:validate both exit zero.
• Steps: predict the agent module’s measured coverage total, then run mise run --cd agents/go coverage and record what it says; run mise run --cd evals test and record pass or fail; then try to fill the runtime latency row without starting anything, and notice what you are tempted to write.
• Gate that proves completion: your three rows each carry a command and a proof class, the coverage row shows the enforced floor and the measured percentage as two different things, and the latency row says unknown rather than a remembered number.
• Final state: no files changed in the repository; you keep the scorecard.

A finished scorecard usually carries these rows; the right-hand column is the one people skip:

Write unknown when a value was not measured: a number filled from memory reads exactly like a measured one.

What you can do now

• You can put any of this agent’s signals on the correct side of the gate-versus-evidence line, and say what goes wrong on the other one.
• You can point at the exact command line where every enforced threshold lives, and name the two numbers measured instead.
• You can write a scorecard row naming its command, candidate, and proof class, and unknown for the latency row.
• You can name a metric that improves when the agent gets worse, and the counterweight that catches it.

Continue to 4.4. Evaluations, which produces the rows this page could only mark unknown.