7.5. Online Evaluation

Offline cases cannot cover the inputs real users send

Offline evaluation asks whether a candidate regressed known behaviors against fixed, versioned cases. Online evaluation asks whether drift has set in: a change in real inputs or outcomes, measured against a stated baseline window, that the fixed set never covered. Only the first has run in this course. A committed case list was written before anyone touched the system, so it cannot notice a change nobody anticipated; scoring live traffic can, but it means reading what users wrote — a privacy, consent, and retention design rather than a feature.

The worked example: twenty-one committed cases pass, the release ships, a support ticket reports the answers “feel worse,” and every number you own was measured against inputs you wrote yourself.

This page pulls a bounded sample of your own traffic out of Tempo over a stated window, proves it holds no evaluation traffic, and turns the scorer you are not switching on into a written design. Start with the sample:

now=$(date +%s)
curl -s "http://localhost:3200/api/search?tags=service.name%3Dagentops-agent&limit=5&start=$((now-14400))&end=$now" \
  | jq -r '.traces[]? | "\(.traceID) \(.rootTraceName) \(.durationMs)ms"'

be014c35ffa2038f1b14d4fce3bd6034 invoke_agent agentops_agent 333925ms
f6310c8a5550385867bf607597572dc8 invoke_agent agentops_agent 33ms
3e8d2df81ff06371331e74d2505434d9 invoke_agent agentops_agent 30ms
525e677f26a6eb6433ac4fbd886fec5a invoke_agent agentops_agent 895571ms
7488083813448af2ab00db9c0649e1ee invoke_agent agentops_agent 181374ms

Five real turns off a four-hour window. State start and end yourself: the search API defaults to roughly the last hour, so an empty result usually means your traces are older than the question you asked. limit=5 is a demonstration size rather than a sample size — a drift comparison needs the minimum sample you fixed in advance.

Now answer that support ticket from those rows. Two came back in about thirty milliseconds and three ran for minutes, one of them for a quarter of an hour. That is a genuine latency finding, and it tells you precisely nothing about whether any of the five answers was correct.

What metadata-only traces cannot tell you about answer quality

Correctness is what a metadata-only trace cannot score, and the reason is structural rather than temporary. Content capture defaults to false, so runtime traces keep operation, model, tool, token, latency, status, and correlation metadata while excluding prompts, responses, and tool payloads. Without the answer and the rows it was built from, no later job can decide whether that answer was correct or grounded. It can only score coarse properties: tool-name order, status, latency, model selection.

Your own sample is the exception: it carries content only because 7.1. Tracing had you accept that risk in one shell, on fictional data. Design for the shipped default, because that is what a fleet runs — and enabling capture duplicates user data into a durable store, changing your privacy, access, and retention obligations.

What you can watch honestly are bounded proxies: signals split only by dimensions from a small, known set, no session ids or prompt text, so they stay comparable across windows. Request rate, latency, and error ratio by operation and model; guardrail injection-neutralization and gateway rejection counts; structured-report schema failures; model-call and token distributions; circuit openings and dependency errors. A changed proxy says something moved, never that the answers got worse.

Comparing proxies across windows takes discipline. Choose a baseline window, a candidate window, a minimum sample size, and bounded dimensions in advance; confirm the source and model identity for both windows before comparing them, using the tuple from 7.0. Reproducibility; and require a difference larger than sampling noise before acting, because lab traffic is sparse and noisy. The absence of a quality metric is not evidence of stable quality: when nothing scores content, the correct report is “no online correctness trend exists,” not “quality is fine.”

One population trap is worth closing before you sample anything. Evaluation runs use service.name=agentops-evals and stable agentops.eval.* signals, and they export only when EVAL_OTEL_EXPORTER_OTLP_ENDPOINT is set explicitly; the harness also forces child-agent telemetry off. Ask Tempo whether any of that traffic is in your window:

curl -s 'http://localhost:3200/api/search?tags=service.name%3Dagentops-evals&limit=5' | jq -c '.traces // []'

[]

Empty, as it should be on a runtime sample. Estimating user drift from a population containing your own committed eval prompts would mean measuring your test suite and reporting it as customer behavior. If rows come back, an eval run reached this endpoint: unset EVAL_OTEL_EXPORTER_OTLP_ENDPOINT, then move your window past that run or filter to service.name=agentops-agent. The separation is architectural, but checking it costs one command.

You now hold a defensible runtime sample: a stated window, your own traffic, and one command proving your test suite is not hiding inside it.

What the course ships is runtime telemetry, offline model evaluation, and a path for promoting sanitized findings into committed cases. What it does not ship is a sampling rule, a consent mechanism, an online judge scheduler, reviewer access control, assessment storage, alert thresholds, or incident routing for answer quality. Do not claim drift detection is active.

Those gaps are not laziness; each is a control a production design has to earn:

1. An inclusion rule that samples the traffic mix rather than the convenient slice, rate-limited against spikes.
2. Consent, minimization, and retention for every field read.
3. Redaction before any external model call.
4. Versioned scorer and judge identities, so a scorer upgrade is never misread as a quality change.
5. Cost, timeout, and error budgets.
6. Deduplicated verdict storage: one verdict per trace, linked to the source trace rather than written into it, so re-runs cannot double-count.
7. Sustained thresholds and accountable incident owners.
8. A path that promotes sanitized failures into offline cases.

One structural rule outranks the eight: run the job outside the request path, so scorer latency or a scorer outage cannot harm the user whose answer is being judged. An online scorer cannot retract an answer already sent — request-time policy owns prevention, asynchronous scoring can only detect and route.

Your turn: sample your own traffic, then design the scorer

Write the one-page design rather than code: the design makes the missing controls visible before somebody switches something on.

Predict first: of the seven fields below, how many can you fill from telemetry you already have, and how many require a decision nobody has made yet?

• Mode: inspect — you read Tempo and write a page of your own; nothing in the repository changes.
• Goal: produce a bounded runtime sample you can defend, and a design that states plainly what it could and could not score.
• Files to touch: none in this repository. Keep the design wherever your team keeps designs.
• Preflight: confirm Tempo is ready with curl --fail http://localhost:3200/ready, and confirm the trace switch is still accepted in this shell with echo $ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS printing true. If the search window comes back empty, send a few turns with the agent running under those exports — a shipped-default agent will never fill it.
• Steps: run both search commands above over a window you state. Then fill seven fields: included and excluded traces; metadata or content fields read; redaction and retention; scorer version and cost cap; baseline, threshold, window, and minimum sample; verdict store and trace-link key; owner, alert route, and rollback action.
• Gate that proves completion: your sample returns your own runtime traces and an empty agentops-evals result over the same window, and every one of the seven fields has a concrete value — including an owner by name and a rollback action.
• Final state: no judge is running, no scorer is scheduled, and your design says in one sentence whether it can assess answer correctness at all. If it needs content, it names the privacy cost explicitly.

Most designs get that last sentence wrong. Record which scorer you chose and why; whoever inherits the design will otherwise assume it reads content.

What you can do now

• You can pull a bounded Tempo sample over a window you state, and prove service.name=agentops-evals is not in it.
• You can say precisely what metadata-only traces can score and what they cannot, without hedging.
• You can list the controls a live scorer would need, and name the one structural rule that keeps it out of the request path.
• You can fill the seven fields of a scorer design, and say plainly whether it could score correctness at all.

Continue to 7.6. Governance, where the question stops being whether the answer was good and becomes who authorized what it did.