7.2. Monitoring

How are agent metrics produced?

The OTel span_metrics connector turns spans into request count and duration histograms under namespace agentops. It retains only bounded dimensions:

connectors:
  span_metrics:
    namespace: agentops
    dimensions:
      - name: gen_ai.operation.name
      - name: gen_ai.request.model
      - name: error.type

Both collector profiles expose OTLP/span-derived metrics at :8889. The Kubernetes collector also scrapes agentgateway :15020, so its endpoint includes gateway metrics. In the host Compose profile, Prometheus scrapes collector :8889, MLflow /metrics, and gateway :15020 directly. The local Kubernetes overlay ships its own single-replica Prometheus that scrapes only otel-collector:8889 and evaluates the same alert rules; the GKE overlay still leaves :8889 on a ClusterIP for a separately operated Prometheus-compatible scraper.

Which dashboard is shipped?

The host Compose profile provisions AgentOps overview at http://localhost:3002/d/agentops-overview with six metric panels and one logs panel:

sum(rate(agentops_calls_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(agentops_duration_seconds_bucket[5m])))
sum(rate(agentops_calls_total{status_code="STATUS_CODE_ERROR"}[5m]))
  / clamp_min(sum(rate(agentops_calls_total[5m])), 1)
sum(rate(agentgateway_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(agentgateway_request_duration_seconds_bucket[5m])))
sum(rate(agentgateway_guardrail_checks_total{action="Reject"}[5m]))

These are request rate, p95 latency, error ratio, gateway rate/latency, and guardrail rejection rate. The Agent logs panel below them queries Loki for ops-copilot log lines, optionally filtered by the trace id textbox variable. A graph with no data is a telemetry problem to investigate, not zero traffic by definition.

Where do my agent logs go?

After ADK configures its providers, the application installs exactly one OTel LoggingHandler on the agent logger when OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is configured. A trace-only endpoint, no endpoint, or OTEL_SDK_DISABLED=true installs none. Both collector profiles forward the resulting logs pipeline to Grafana Loki, which ingests native OTLP at /otlp/v1/logs:

logs:
  receivers: [otlp]
  processors: [memory_limiter, batch]
  exporters: [otlp_http/loki]

Loki runs pinned in single-binary mode with filesystem storage and 7-day retention, matching Prometheus: as a Compose service on the host (infra/observability/loki.yaml) and as a PVC-backed Deployment in Kubernetes (infra/k8s/base/loki.yaml). Grafana provisions the Loki datasource next to Prometheus; the Kubernetes overlay exposes only the loki:3100 ClusterIP, mirroring the external-scraper stance for metrics.

Content capture stays off by default, so ADK does not intentionally duplicate prompts or model responses into log records. Before OTLP export, the handler filters a copy of each record: it locally redacts concrete PII and obvious credential/token patterns, caps every exported string and body at 2048 characters, removes exception messages, tracebacks, and stack bodies, and retains only exception.type. Console handlers still receive the untouched local record. This is defense in depth, not a reason to log secrets; apply the same access and retention policy to Loki as to the trace store.

How do I jump from a trace to its logs?

Log records emitted inside a span keep their trace context, and Loki stores trace_id as structured metadata on every OTLP-ingested line. To correlate one agent turn:

1. Send one request, open its trace in MLflow at http://localhost:5000, and copy the trace id.
2. Open http://localhost:3002/d/agentops-overview, paste the id into the trace id variable, and read the Agent logs panel.
3. Alternatively, query Loki from Grafana Explore or its HTTP API:

{service_name="ops-copilot"} | trace_id="<trace id from MLflow>"

curl -fsS -G 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="ops-copilot"}' | jq '.data.result | length'

In Kubernetes, forward the ClusterIP first with kubectl -n agentops port-forward svc/loki 3100:3100. The overlay ships no Grafana; query the forwarded API directly or point an externally operated Grafana at it.

Why avoid session or prompt labels?

Prometheus labels form an in-memory/index cardinality dimension. User ids, session ids, incident ids, prompts, or trace ids can make the store expensive and leak sensitive information. Keep correlation ids in traces/logs and metrics dimensions bounded to known model/operation/error values.

When should the platform page a human?

Page only on sustained, user-visible symptoms; everything else is a ticket-severity signal reviewed during working hours. The shipped rules encode that split: page for error-budget burn against a 99% span-success SLO and for a dark telemetry pipeline, ticket for latency, missing token counters, guardrail spikes, and schema failures. The SLO alert uses a multiwindow burn rate — both the 5m and 1h error ratios must exceed 14.4 times the 1% budget — so a single flaky request does not page, but stopping the model provider does within minutes:

- alert: AgentErrorBudgetBurn
  expr: |-
    agentops:calls:error_ratio_rate5m > (14.4 * 0.010)
      and agentops:calls:error_ratio_rate1h > (14.4 * 0.010)
  for: 2m

On sparse lab traffic a handful of consecutive failures crosses that threshold immediately, which is intended for the course; production traffic makes the same math proportional.

What alerts ship with the course?

infra/observability/prometheus-rules.yml defines two recording rules (agentops:calls:error_ratio_rate5m/rate1h) and six alerts, each grounded in a metric the stack verifiably exports:

1. AgentErrorBudgetBurn (page): multiwindow error-budget burn on agentops_calls_total.
2. ObservabilityCollectorDown (page): up{job="otel-collector"} == 0 — traces, metrics, and logs are all dark.
3. AgentTurnLatencyP95High (ticket): p95 of agentops_duration_seconds_bucket above 15s; tune to your hardware.
4. AgentTokenTelemetryMissing (ticket): spans flowing but no agentops_tokens_token_total increase (the collector's Prometheus exporter appends the token unit suffix).
5. AgentInjectionNeutralizedSpike (ticket): more than 3 agentops_guardrails_injections_neutralized_total increments in 15m.
6. AgentTriageSchemaFailures (ticket): any agentops_triage_report_schema_failures_total increment in 15m.

The host Compose stack loads the rules into Prometheus (http://localhost:9090/alerts) and routes fired alerts to a pinned Alertmanager at http://localhost:9093 whose webhook points at a documented placeholder (infra/observability/alertmanager.yml); replace the URL to integrate a real channel. The local Kubernetes overlay runs the identical rules in an in-cluster Prometheus/Alertmanager pair (infra/k8s/overlays/local) whose Alertmanager receiver has deliberately no integration: default-deny egress keeps notifications cluster-internal, and network policies allow only Prometheus to reach it. There is no external paging service anywhere.

Note one Prometheus counter subtlety when triggering alerts deliberately: increase() needs at least two samples of a series inside its window, so the very first guardrail or schema event after startup may not register — send a few.

How do I respond when AgentErrorBudgetBurn fires?

1. Symptom: agent turns fail or return errors; the dashboard error-ratio panel rises with the alert.
2. Diagnose: curl -fsS http://127.0.0.1:4000/v1/models through the gateway and ollama ps on the host; in Kubernetes, kubectl -n agentops get pods and the agentgateway logs.
3. Likely cause: the model provider is down (stopped Ollama, missing qwen3:4b pull, wrong OLLAMA_HOST binding) or a gateway route/policy rejects the upstream.
4. Fix: restart ollama serve with the documented OLLAMA_HOST, re-pull the model, or revert the gateway config change; confirm the error ratio decays and the alert resolves in Alertmanager.

How do I respond when AgentTurnLatencyP95High fires?

1. Symptom: turns complete but slowly; p95 stays above 15s.
2. Diagnose: open the slowest recent trace in MLflow at http://localhost:5000 and read which span dominates; check ollama ps for model load state and host CPU pressure.
3. Likely cause: model cold starts/reloads, CPU contention with other workloads, or oversized contexts from long sessions.
4. Fix: keep the model warm, free host resources or pick a smaller pinned model, and trim session growth; if your hardware is simply slower, raise the threshold in the rules file instead of deleting the alert.

How do I respond when ObservabilityCollectorDown or AgentTokenTelemetryMissing fires?

1. Symptom: dashboards flatten while the agent still answers — the pipeline, not the agent, is broken.
2. Diagnose: docker compose -f infra/observability/compose.yaml ps otel-collector (host) or kubectl -n agentops get pods -l app.kubernetes.io/name=otel-collector; then curl -fsS http://localhost:8889/metrics | grep agentops_tokens after a port-forward to see whether the counter is exported at all.
3. Likely cause: collector crash or memory-limit kill, a port conflict with the other profile (do not run host Compose and the forwarded in-cluster stack together), or an agent started without OTEL_EXPORTER_OTLP_ENDPOINT so spans arrive from one process and metrics from none.
4. Fix: restart the collector, resolve the port clash, and relaunch the agent with the documented OTLP environment; telemetry gaps for the outage window are permanent, which is exactly why this pages.

How do I respond when a guardrail or triage-schema alert fires?

1. Symptom: AgentInjectionNeutralizedSpike or AgentTriageSchemaFailures shows up as a ticket; user traffic may look normal.
2. Diagnose: filter Loki for the recent turns ({service_name="ops-copilot"}), open the matching MLflow traces, and inspect which tool output carried injection markers or which report failed validation.
3. Likely cause: for injections, adversarial content in the data the tools read (or someone running the red-team suite); for schema failures, model or prompt drift after a model swap.
4. Fix: for injections, confirm the neutralization worked and clean or quarantine the offending source records; for schema failures, re-run the offline tests and evaluation gates and restore the pinned model/prompt combination before trusting new reports.

How do you query the stores directly?

curl -fsS 'http://localhost:9090/api/v1/query?query=sum(rate(agentops_calls_total%5B5m%5D))' \
  | jq '.data.result'
curl -fsS http://localhost:15020/metrics | head

Those Prometheus queries apply to the host Compose profile. In the local Kubernetes overlay, forward the in-cluster Prometheus and Alertmanager, or inspect the raw collector endpoint they scrape:

kubectl -n agentops port-forward svc/prometheus 9090:9090
kubectl -n agentops port-forward svc/alertmanager 9093:9093
kubectl -n agentops port-forward svc/otel-collector 8889:8889
curl -fsS http://localhost:8889/metrics | head

The GKE overlay ships no scraper: an operator points an existing Prometheus-compatible one at the otel-collector:8889 ClusterIP.

How much load can the platform take?

The repository ships Grafana k6 scenarios under load/ — k6 is AGPL-3.0 open source, consistent with the rest of the stack. Each script isolates one layer of the host quickstart:

1. load/health.js: raw /healthz on MCP :8000 and A2A :8080, plus a low-rate hop through agentgateway :3001 — the latency floor and the pure proxy overhead.
2. load/mcp-read.js: the MCP streamable HTTP handshake and a tools/call list_incidents loop through the gateway :3000 — gateway plus FastMCP plus SQLite, with no model call.
3. load/a2a-send.js: a bounded A2A message/send conversation through :3001 — a full agent turn, model included, deliberately capped at 1 VU and 3 iterations.
4. load/fake_model.py: a deterministic OpenAI-compatible response on Ollama's host port, so the same A2A turn measures the platform without inference.

First run the isolated mise run smoke:host composition check. Then, with the manual host stack running (mise run mcp:http, mise run a2a, mise run gateway:host, and Ollama serving qwen3:4b), run a pinned k6 from the mise registry — no permanent install:

mise x k6@2.1.0 -- k6 run load/health.js
mise x k6@2.1.0 -- k6 run load/mcp-read.js
mise x k6@2.1.0 -- k6 run load/a2a-send.js # spends real model time — keep 1 VU

Now hold that last request constant and replace only inference. Stop Ollama so :11434 is free, run mise run model:fake, keep AGENT_A2A_STREAMING=false, and restart A2A with AGENT_MODEL_PROVIDER=openai-compatible plus OPENAI_BASE_URL=http://127.0.0.1:4000/v1. Then run load/a2a-send.js again. The fake returns a valid, fixed chat completion with token usage and no tool call. Both host and k3d gateway profiles already target the host's :11434, so no route or test script changes between samples. This mode is for platform latency and concurrency, not answer-quality evaluation; restore Ollama before any evaluation exercise.

The first honest answer is that the shipped gateway policies cap throughput on purpose: 120 MCP, 60 A2A, and 30 model requests per minute per gateway instance. The default rates stay under those budgets, and mcp-read.js counts every HTTP 429 in an mcp_rate_limited metric whose threshold is zero — a breach means you measured your own rate limiter, not the platform. Raise maxTokens in the gateway config for a real capacity probe, or point MCP_URL at the raw :8000/mcp server to take the gateway out of the path. In the local Kubernetes overlay, run the identical scripts against the port-forwarded agentgateway and raw services and watch pod CPU limits in parallel. Only ever target your own local stack: a load test aimed at a shared or third-party endpoint is a denial-of-service attempt, not a lab.

What is a latency budget?

A latency budget is a pass/fail number agreed on before the test — "this path answers within X ms at percentile Y" — instead of a vibe read of a dashboard afterwards. k6 encodes budgets as thresholds, so a breach fails the run with a non-zero exit code, exactly like a failing unit test. From load/mcp-read.js:

thresholds: {
  // Latency budget — a starting point for localhost, tune to your hardware.
  http_req_failed: ['rate<0.01'],
  'http_req_duration{op:tools_call}': ['p(95)<250'],
  mcp_rate_limited: ['count==0'], // any 429 means the gateway budget, not the platform, was measured
},

The shipped starting points are a p95 under 50 ms for raw health, 100 ms for the gateway hop, 250 ms for the MCP read, and 15 s for a full A2A turn — the last one matches AgentTurnLatencyP95High, so the load test and the alert rules cannot disagree. Read the percentiles from the end-of-run summary (http_req_duration ... p(95)=...): averages hide tail latency, and the tail is what a user waiting on an agent turn actually feels. On slower hardware, tune a breached budget deliberately instead of deleting it — the same instruction the alert rule carries.

Where does the time actually go?

Subtract the layered measurements to place the bottleneck with evidence instead of guesses:

1. Raw /healthz is the floor: process, loopback, and HTTP parsing — typically around a millisecond locally.
2. The gateway hop minus the raw A2A number is pure agentgateway proxy overhead — also millisecond-class.
3. The MCP tools/call p95 minus the floor is FastMCP protocol handling plus the SQLite read — tens of milliseconds.
4. The A2A message/send p95 is all of the above plus the model — multiple seconds per turn on a local Qwen3-4B.

The fake-model A2A run is the control: subtract its p95 from the Ollama A2A p95 to measure inference rather than assuming it. On a typical local Qwen3-4B path, that gap is orders of magnitude larger than the gateway/MCP layers, so pushing many VUs through the real model mostly saturates inference (or your token bill). Record your two measured p95 values; the course provides the method and budgets, not a hardware-independent result. Load-test the fake-backed platform path with rate; sample the real model path at 1 VU.

To correlate a budget breach, start mise run observability:up before the run. The dashboard's gateway panels (agentgateway_request_duration_seconds) show the hop the gateway sees while the agent panels (agentops_duration_seconds) show time inside the process: a flat-fast gateway with a slow agent places the problem behind the proxy. Then open the slowest turn in MLflow at http://localhost:5000, read which span dominates, and filter Loki with {service_name="ops-copilot"} plus that trace id for errors or rate-limit rejections — the same three-pillar walk as any alert response above.

What is the monitoring checkpoint?

With the host Compose stack, generate allowed, rejected, and failed requests. Verify all six metric panels receive data, compare the Prometheus result with the trace count, and confirm labels contain no raw prompts, users, sessions, or trace ids. Then correlate one turn across all three pillars: its MLflow trace, the request-rate increase in Prometheus, and its Loki log lines filtered by that trace id. Finally, fire one alert deliberately: stop Ollama, send a few requests, and watch AgentErrorBudgetBurn move from pending to firing at http://localhost:9090/alerts and appear in Alertmanager at http://localhost:9093; restart Ollama and watch it resolve. In the local Kubernetes overlay, repeat against the forwarded svc/prometheus and svc/alertmanager; on GKE, stop at verifying the bounded metrics on :8889 and the forwarded Loki API unless you already operate a scraper. Tear down with docker compose -f observability/compose.yaml down, which preserves the named volumes; adding -v deletes the stored metrics and logs.

How would you add an alert rule and its runbook?

Exercise: close the loop from a firing alert to a documented response.

• Goal: add one new Prometheus alert rule tied to an observable outcome (e.g. sustained tool-error rate or p95 latency breach), and write the runbook an on-call engineer would open when it fires.
• Files to touch: infra/observability/prometheus-rules.yml for the rule (with a meaningful for: window and a runbook annotation), and a new entry under agents/data/runbooks/ describing symptoms, checks, and remediation.
• Gate that proves completion: promtool check rules infra/observability/prometheus-rules.yml passes, and deliberately driving the condition on the host stack moves your alert from pending to firing in Prometheus (:9090/alerts) and into Alertmanager (:9093), then resolves when the condition clears.