Skip to content

7.4. Feedback

Why must feedback reference a trace?

"Bad answer" is not actionable without the model, prompt, tool trajectory, evidence, latency, policy decisions, and final output that produced it. A trace id links the human judgment to those facts without placing full prompt text in a Prometheus label or issue title.

What feedback should you collect?

Use a small rubric tied to decisions:

  • Correct and grounded in tool/runbook evidence?
  • Chose the necessary tools without waste?
  • Recommendation safe and actionable?
  • Approval/policy behavior correct?
  • Free-text rationale with sensitive details removed?

A thumbs-up rate alone cannot distinguish wrong facts, poor style, or unsafe action selection.

How can you log feedback to self-hosted MLflow?

With a recent trace in the local ops-copilot experiment (id 0):

cd agents/python
export MLFLOW_TRACKING_URI=http://localhost:5000
TRACE_ID="$(uv run mlflow traces search \
  --experiment-id 0 \
  --max-results 1 \
  --order-by 'timestamp_ms DESC' \
  --output json \
  --extract-fields info.trace_id \
  | jq -r '.traces[0].info.trace_id')"
export TRACE_ID
uv run python - <<'PY'
import os

import mlflow
from mlflow.entities import AssessmentSource, AssessmentSourceType

mlflow.set_tracking_uri(os.environ["MLFLOW_TRACKING_URI"])
mlflow.log_feedback(
    trace_id=os.environ["TRACE_ID"],
    name="safe_and_grounded",
    value=True,
    source=AssessmentSource(
        source_type=AssessmentSourceType.HUMAN,
        source_id="course-reviewer",
    ),
    rationale="Matched the incident and cited the expected runbook.",
)
print(os.environ["TRACE_ID"])
PY

This attaches an assessment to the latest trace. Confirm the selected trace before judging it; production systems should take the trace id from the user-visible response context rather than assume "latest".

Does the A2A application collect feedback automatically?

No. The shipped A2A server has no authenticated feedback endpoint or UI. MLflow provides the storage/assessment API for the lab. A production feedback API must authenticate the reviewer, authorize access to the trace, validate the rubric, rate-limit writes, protect PII, and prevent one user from overwriting another assessment.

How does feedback improve the agent?

Cluster feedback by failure mode, inspect representative traces, reproduce the issue on sanitized seed data, add an offline test or eval case, change one component, and compare on a holdout set. Never directly paste private production conversations into the committed eval dataset.

What is the feedback checkpoint?

Attach one human assessment to a known trace, find it in MLflow, and write the smallest sanitized regression case that would have caught the issue. Record reviewer source and rationale; do not use feedback text as a metrics label.