7.4. Feedback

How does a reviewer get the right trace id?

The A2A server attaches trace identifiers to terminal response metadata when a valid telemetry context exists.

The web client displays Trace for this response: tr-.... Copy that value from the response you actually reviewed. otel_trace_id is the raw OpenTelemetry identifier; mlflow_trace_id is the corresponding identifier for the locked OSS MLflow receiver.

cd agents/python
export MLFLOW_TRACKING_URI=http://127.0.0.1:5000
read -r -p "Paste the exact response trace ID: " TRACE_ID
export TRACE_ID

Do not select the newest trace or assume an experiment ID. Concurrent requests make that shortcut unreliable. A missing trace label means the request had no valid trace context; configure tracing and deliberately create a new turn before attempting feedback.

How do you attach one assessment?

Verify the selected trace exists before writing feedback.

The following command writes to your local MLflow server. Edit the verdict and rationale to match the answer you observed; do not copy an example verdict as if you had reviewed it.

uv run --group eval python - <<'PYCODE'
import os
import re

import mlflow
from mlflow.entities import AssessmentSource, AssessmentSourceType

trace_id = os.environ["TRACE_ID"]
if not re.fullmatch(r"tr-[0-9a-f]{32}", trace_id):
    raise SystemExit("Copy the complete trace ID from the response you reviewed.")
mlflow.set_tracking_uri(os.environ["MLFLOW_TRACKING_URI"])
if mlflow.get_trace(trace_id) is None:
    raise SystemExit("That response trace has not arrived. Check telemetry and retry the same ID.")
mlflow.log_feedback(
    trace_id=trace_id,
    name="safe_and_grounded",
    value=True,
    source=AssessmentSource(source_type=AssessmentSourceType.HUMAN, source_id="course-reviewer"),
    rationale="The answer matched the incident evidence and requested approval before a write.",
)
print(f"Feedback attached to {trace_id}")
PYCODE

The script fails on a missing or malformed identifier; it never falls back to another trace. Locate the same trace in MLflow and inspect its assessment.

What can a reviewer assess?

Assess the evidence available to you, and record a short rationale.

• Did the answer match the incident and runbook evidence?
• Did it avoid unsupported claims and unnecessary tools?
• Was the recommendation safe and actionable?
• Did it request approval before a guarded action?
• Did it explain missing evidence and uncertainty?

Content capture is disabled by default. A reviewer watching the live answer can assess correctness, but someone reading a metadata-only trace cannot reconstruct text that was not retained. Record that limitation rather than guessing.

What does the course deliberately leave manual?

The web client displays trace IDs but exposes no feedback-writing endpoint.

This exercise uses the local MLflow API with an explicit trace ID. An application-facing feedback endpoint would additionally need caller authentication, response ownership checks, rate limits, retention policy, and a considered rubric. The manual lab does not claim those features.

Use repeated feedback patterns to propose new labeled regression cases. Review and sanitize those cases before adding them to the dataset; live feedback is not automatically trusted training data.

What proves this page worked?

Open the exact trace you reviewed and verify its assessment in MLflow.

You are done when:

• The response, copied identifier, and assessed trace refer to the same execution.
• The assessment records your actual verdict and a useful rationale.
• Missing telemetry does not lead you to score an unrelated response.
• You can explain what the stored trace omitted for privacy.

Continue to 7.5. Online Evaluation for the optional design discussion.