7.0. Reproducibility

What a commit hash cannot pin about an agent run

Source identity is a typed, machine-produced name for what a checkout is: a revision when the tree is clean, unknown+dirty.<digest> when it is not. It exists because an agent run is decided by more than a commit — also by the model artifact behind a model name, the seed data, leftover runtime state, the transport, and the sampling settings.

Change one silently and you have not run the experiment you think you ran: two candidates differing in an undeclared input were never compared, and a measurement you cannot attribute to one tree is a number rather than evidence. That is how a command goes green one day and red the next on code everyone agrees is identical. So this page pins the measurement first: run the identity command in both modes, read the refusal, and write down the tuple a comparison must match.

Read the source identity, and watch release mode refuse

The identity command has two modes: development names a working tree honestly, dirt included; release refuses to name anything it cannot honestly claim. Start with development:

go -C tools run ./cmd/source-identity --root .. --mode development | jq

Predict first. Your tree almost certainly has uncommitted edits, so should this print the commit you are sitting on?

It does not:

{
  "mode": "development",
  "display": "unknown+dirty.066737d90d99",
  "tree_digest": "sha256:066737d90d99404dabb8009400b1bcaa42484f434deecf98483135170a55cb19",
  "dirty": true,
  "shallow": false
}

There is no revision field at all. A dirty tree is not the commit it sits on, so it never claims one: it gets a visible unknown+dirty.<digest> name plus tree_digest, a deterministic hash of the tracked and untracked inputs as they sit on disk. Your digest will differ from the one above; that is the point of a digest.

Now ask the same tree for a release identity:

go -C tools run ./cmd/source-identity --root .. --mode release | jq

source-identity: release source is dirty; tracked and untracked inputs must match HEAD
exit status 1

Release mode refuses, non-zero, before producing anything a downstream artifact could quote. It reads git status with untracked files included, so a stray scratch file counts as dirt like an edited one: a build that compiled a file nobody committed is not reproducible from the commit. On a clean checkout it prints the full revision, the tree digest, and "dirty": false, and the image tasks stamp that validated identity into the binary and the OCI labels.

Git owns the instruction version for the same reason: the root instruction is committed source with contract tests pinning its content, so the binary and its OCI revision identify the instruction and the tool wiring together. No mutable prompt alias drifts out from under a review, and a rollback is a revision plus an image digest, not an edit to a live prompt.

Record every input that can change an answer

Source identity is one line of a longer receipt, and any line below can move an answer. The evalset line is the committed set of evaluation cases a run scored, pinned by digest. Record them all before comparing two candidates:

The model half of that table has its own command, printing the resolved configuration with secrets masked:

cd agents/go
mise run config:check

It prints every setting in one alphabetical list. Here is the header plus the five consecutive AGENT_MODEL* lines from the middle:

[config:check] $ go run ./cmd/agent config:check
Agent configuration is valid. Resolved settings (secrets masked):
...
- AGENT_MODEL = qwen3:4b-instruct
- AGENT_MODEL_FALLBACK = (unset)
- AGENT_MODEL_PROVIDER = openai-compatible
- AGENT_MODEL_TEMPERATURE = (unset)
- AGENT_MODEL_TIMEOUT_S = 60
...

The ... marks are mine: the real run prints 48 settings with nothing elided, and yours will differ wherever your .env does. Record the provider, the model name, the base-URL class rather than the URL, the temperature, and any fallback: an endpoint URL can carry infrastructure detail, so sanitized artifacts record provider, name, and optional digest instead. For local Ollama, also keep the model artifact digest the catalog resolved — qwen3:4b-instruct is a label, and labels get re-pointed.

State is the line people forget. Each evaluation case starts the agent on isolated temporary state and tears it down afterwards; a comparison you run by hand inherits whatever the last run left behind:

cd agents/go
mise run data:reset

Reset before each run, not once: otherwise anything the baseline wrote stays behind and the candidate reads a different generation. Never mutate the committed seed at agents/data/incidents.db.

What the evaluation artifact pins for you

The harness writes most of that receipt for you. mise run eval runs three samples per case against a 0.33 minimum pass rate with five required safety cases and per-turn entity groundedness on, and writes a content-free artifact to results.json: the run id, the source revision and dirty flag, the sanitized model identity, the evalset id and SHA-256 digest, the transport and timestamps, per-case scores, pass state and token usage, and a summary carrying pass_rate, minimum_pass_rate, and required_cases_passed.

A floor plus named cases beats one aggregate: the floor absorbs the spread across samples, and a required safety case cannot be averaged away by wins elsewhere.

The file existing is not a pass. Read those three summary fields together, then check that source.revision is the candidate you meant to measure: a dirty run reports no revision at all, which is the artifact saying it cannot be attributed.

With an evaluation exporter explicitly configured through EVAL_OTEL_EXPORTER_OTLP_ENDPOINT, the same run emits one agentops.eval.run span tree with case and score children, plus bounded metrics carrying the run, source revision, model, evalset, transport, case, and outcome. JSON stays the release handoff; Tempo and Prometheus are the comparison view. The harness forces the child agent’s exporter off, so one captured case cannot be counted twice, once as an evaluation result and once as production traffic.

A comparison is invalid the moment two candidates differ in more than the declared variable. Reject one when:

• The source identities differ without a reviewed diff.
• The evalset digest or the sample count differs.
• The seed or the runtime state differs, unless state is the declared variable.
• The provider, model artifact, endpoint class, or temperature differs.
• One side ran REST and the other A2A.
• One run exported evaluator telemetry and the other did not, and someone read dashboard presence as quality.

mise run eval:ab refuses mismatched evalset digests, sample counts, and non-distinct revisions, and recomputes pass rates from the per-case records rather than trusting either summary. The rest of the tuple is yours to check.

git worktree add ../agentops-baseline "$BASELINE_SHA"
git worktree add ../agentops-candidate "$CANDIDATE_SHA"
mise run --cd ../agentops-baseline/agents/go build
mise run --cd ../agentops-baseline/evals eval
mise run --cd ../agentops-candidate/agents/go build
mise run --cd ../agentops-candidate/evals eval
mise run --cd evals eval:ab -- \
  --baseline ../../agentops-baseline/evals/results.json \
  --candidate ../../agentops-candidate/evals/results.json

Temperature zero buys lower variance, not determinism: model selection, serving kernels, concurrency, and upstream behavior all still move. The deterministic Go tests cover the parsers, policies, scorers, and wire folds around the model, not the model itself. Report a mixed sample rather than averaging away the one failure that mattered — 0.2. Evidence owns that reasoning.

You already made this refusal happen from the build side in 6.1. Containers. What this page adds is the digest either side of it: go -C tools run ./cmd/source-identity --root .. --mode development | jq -r .tree_digest before you create an untracked file, and again after you remove it, returns the same value byte for byte. That is what makes a recorded digest an identity rather than a timestamp.

What you can do now

• You can name what your checkout is — revision or unknown+dirty.<digest> — without asking Git twice.
• You can read the same tree_digest before and after a scratch file exists, and say why the value in between was neither wrong nor usable.
• You can list the tuple behind a run, and say which line makes a given comparison invalid.
• You know why a dirty run’s results.json carries no revision at all.

Attribution is why the rest of this chapter’s numbers are worth writing down.

Continue to 7.1. Tracing, where one turn stops being a paragraph and becomes a tree you can time.