7. Observability

Part II — Platform engineering. This part is more demanding and assumes container and Kubernetes knowledge. Complete 4.8. Developer Handoff or begin from its tested reference checkpoint. Prepare with Kubernetes Basics if needed.

How will you operate the agent after deployment?

Your AgentOps Agent runs behind agentgateway (Chapter 5), and optionally as a Kubernetes workload (Chapter 6). This chapter closes the AgentOps loop with evidence: seeing what the agent does, proving what it did, and reacting when it breaks.

The telemetry stack is a Docker Compose stack, not a cluster feature. Seven of the eight pages below run entirely on it; 7.6. Governance is the single page that reads its evidence off the deployed workload and therefore needs mise run install:platform and a running k3d cluster. Read the rest whenever Chapter 5 is behind you.

mise run doctor:gateway    # checks Docker, Compose, and the other container prerequisites
mise run observability:up  # MLflow, the collector, Prometheus, Alertmanager, Loki, Grafana

Every later page assumes one telemetry topology and one set of ports. This landing page is the map: which signal answers which question, what runs where, and which port each piece listens on.

Which signal answers which question?

Traces, metrics, logs, assessments, and audit rows each answer a different operational question; open the page that owns the signal you actually need:

Each of those pages is also explicit about where the shipped stack stops.

What does the shipped telemetry stack look like end to end?

One collector receives what the agent and gateway emit, then fans that single stream out to three stores. Prometheus scrapes the collector and feeds the dashboard and the alerts:

flowchart LR
    Agent[agentops-agent] -->|OTLP :4318| Collector[OTel Collector]
    Gateway[agentgateway] -.->|OTLP traces :4317, k8s only| Collector
    Collector -->|traces, experiment 0| MLflow[MLflow :5000]
    Collector -->|logs /otlp| Loki[Loki :3100]
    Collector -->|span_metrics :8889| Prometheus[Prometheus :9090]
    Prometheus --> Grafana[Grafana :3002]
    Loki --> Grafana
    Prometheus --> Alertmanager[Alertmanager :9093]

Three names in that diagram are worth pinning down now:

• OTLP — the OpenTelemetry wire protocol that carries traces, metrics, and logs off the agent (0.7. Glossary).
• span_metrics — the collector connector that turns spans into request-count and duration metrics.
• experiment 0 — MLflow's built-in default experiment, which the course image renames to agentops-agent.

The shipped OSS stack is MLflow, the OpenTelemetry Collector, Prometheus, Alertmanager, Loki, and Grafana. Each piece listens on one port, and every later page assumes you already know which:

Which scraper — the process that pulls metrics from an endpoint on a schedule — and which UI you actually get depends on the deployment profile: host Compose, the local k3d overlay, or GKE.

What proves this chapter worked?

The chapter checkpoint uses local or already-running lab telemetry. It does not deploy GCP or call a model unless the learner explicitly chooses that step.

You are done when:

• mise run observability:up has finished and http://localhost:5000 serves the MLflow UI.
• http://localhost:3002 opens Grafana without asking you to log in.
• For a trace, a metric, a log line, an assessment, and an audit row, you can name the page that owns it.
• You can say which port the agent exports to, and which store keeps each of the three signals.
• You finished the required drill in 7.2. Monitoring: your own alert rule passed promtool check rules, reached firing at http://localhost:9090/alerts, and resolved when you cleared the condition.
• Without reopening Chapter 4, you can name the offline gate that proves a guardrail before release and the runtime signal that proves it in production — and say why neither replaces the other.

Continue to 7.0. Reproducibility when the stack is up and you know which page to open for the signal you need.