Skip to content

description: Gain insight into the agent in production: reproducibility, tracing, monitoring, cost, feedback, online evaluation, and governance.

7. Observability

How will you operate the agent after deployment?

Your Ops Copilot now runs as a private Kubernetes workload (Chapter 6). This chapter closes the AgentOps loop with evidence: release lineage, OpenTelemetry traces, span-derived/gateway metrics, explicit cost assumptions, trace-linked human feedback, a safe online-evaluation design, and auditable approved actions.

The shipped OSS stack is MLflow, OpenTelemetry Collector, Prometheus, Alertmanager, Loki, and Grafana. Pages distinguish implemented signals from desired production extensions: there is no fake dollar-cost panel, automatic live judge, external paging integration (alerts stop at the local Alertmanager), cryptographically immutable audit store, or HA claim.

This chapter covers:

  • 7.0. Reproducibility: Version code, images, model path, prompt, data, tools, and evaluation evidence together.
  • 7.1. Tracing: Export privacy-preserving ADK/gateway spans through OTel into MLflow.
  • 7.2. Monitoring: Query the shipped RED/gateway metrics, Loki agent logs, and provisioned Grafana dashboard, and respond to the shipped alerts.
  • 7.3. Costs: Bound model work and state conditional local/GKE cost assumptions honestly.
  • 7.4. Feedback: Attach human MLflow assessments to concrete traces and promote regressions safely.
  • 7.5. Online Evaluation: Inspect bounded trace samples and design, but do not pretend to run, live scoring.
  • 7.6. Governance: Connect identity, confirmation, transactions, append-only evidence, persistence, and residual risk.

The chapter checkpoint uses local or already-running lab telemetry. It does not deploy GCP or call a model unless the learner explicitly chooses that step.