5.6. Gateway Observability

Which gateway signals are available?

The gateway sees every request, including the ones it rejects before any upstream runs. That makes gateway telemetry different in kind from application telemetry: the agent process cannot report a call it never received, and a 400 produced by a policy exists only in the gateway's own signals.

Three signals ship here:

1. Structured JSON logs on standard output in every profile. config.logging.format: json is configured in each gateway profile (host, host auth, k3d, gke), so log shape does not change as you move profiles.
2. Prometheus metrics on internal port 15020 in every profile. On the host, infra/scripts/gateway-host.sh injects .config.statsAddr = "0.0.0.0:15020" into the rendered config and publishes it on 127.0.0.1:15020. In Kubernetes, the same port is a named metrics container port on the Deployment and a port on the ClusterIP Service.
3. OTLP gateway traces in the k3d and GKE profiles only, whose in-cluster collector is part of the deployment. OTLP is the OpenTelemetry wire protocol a process uses to push telemetry to a collector.

The host gateway deliberately leaves OTLP disabled so the optional Compose stack can be down without exporter retries. Host application traces are still available when the agent's OTEL_* variables are set. Kubernetes gateways export to otel-collector.agentops.svc.cluster.local:4317.

This page covers what the gateway itself emits and where it lands. The dashboard panels, alert rules, Loki queries (Loki is the log store in the same Compose stack), and measured latency numbers built on top of these signals belong to 7.2. Monitoring.

What does a rejected request look like across all three signals?

Do this before reading further: send one request the gateway refuses, then go find it. This is the exercise that teaches gateway observability, because it is the one case where the signals must disagree.

With the host gateway from 5.1. Gateway Setup running, send the prompt-guard curl from 5.5. Gateway Security. The email address in the prompt is what trips the guard:

curl -i http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gemini-3.5-flash",
    "messages": [{"role": "user", "content": "Email jane.doe@example.com"}]
  }'

Expected status: 400, before the provider is called. Now read the counter that rejection just moved:

curl -fsS http://localhost:15020/metrics | grep agentgateway_guardrail_checks_total

After your first rejection since the gateway started, the line reads:

agentgateway_guardrail_checks_total{phase="Request",action="Reject"} 1

Send the curl again and re-read the counter. If the number does not move, your request never reached the policy you think it did — a wrong port, a wrong listener, or an unrendered config change.

Here is the whole path, including the branch you did not take:

flowchart TD
    Curl[curl :4000 with an email in the prompt] --> Guard{promptGuard request regex}
    Guard -->|match| Reject[400 Request rejected by the course prompt guard.]
    Reject --> Log["gateway JSON log: status 400, reason DirectResponse"]
    Reject --> Metric["agentgateway_guardrail_checks_total phase Request action Reject +1"]
    Reject --> Counter["agentgateway_requests_total status 400 +1"]
    Reject -.->|never called| Model[Configured model provider]
    Guard -->|no match| Allow[proxied to provider]
    Allow --> Model
    Allow --> Log2["gateway JSON log: status 200"]
    Allow --> Counter2["agentgateway_requests_total status 200 +1"]

Diagram in words: The gateway checks the request before contacting the provider. A match returns 400 and increments rejection/request counters; an allowed request reaches the provider and records its outcome.

The rejection log and counter show the gateway decision. This direct curl never enters the agent, so an absent MLflow agent trace adds no independent proof. When the agent makes a rejected call, its trace may contain a failed model span.

reason="DirectResponse" in both the log line and the metric labels is the gateway saying so explicitly. On the optional Ollama profile, stop your own Ollama process and repeat the rejected request. A continuing 400 demonstrates that the guard can answer without that upstream.

The rest of the page takes those signals one at a time: the metric names, the log line, and the plumbing that carries both to Prometheus.

Which metric names should you actually look for?

:15020/metrics returns hundreds of series, and three metric families carry the request-level story:

• agentgateway_requests_total — how many requests the gateway handled, labelled by status, listener, and protocol.
• agentgateway_request_duration_seconds_bucket — how long they took, as histogram buckets (a count of requests under each latency threshold).
• agentgateway_guardrail_checks_total — how many requests a policy rejected: the counter you just watched move.

Most of the rest are runtime gauges (agentgateway_tokio_*, agentgateway_process_*, agentgateway_config_synchronized). Useful for a crash investigation, useless for answering "is the data plane healthy".

sum(rate(agentgateway_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(agentgateway_request_duration_seconds_bucket[5m])))
sum(rate(agentgateway_guardrail_checks_total{action="Reject"}[5m]))

The raw series behind the first and third of those families, captured on the host profile after one prompt-guard rejection:

agentgateway_requests_total{backend="/default/default/bind/4000/llm/default/route0/backend0",protocol="llm",method="POST",status="400",reason="DirectResponse",bind="bind/4000",gateway="default/default",listener="llm",route="default/route0",route_rule="unknown"} 1
agentgateway_guardrail_checks_total{phase="Request",action="Reject"} 1

Read the labels: every dimension is bound by your configuration (bind, listener, route, backend, protocol, method, status, reason), never by the caller or the prompt. That is why these series are safe to keep at high resolution, and why 7.2. Monitoring can refuse user, session, and trace-id labels without losing the operational picture. protocol="llm" distinguishes model traffic from mcp and a2a on the other binds, so one expression can be split per listener.

agentgateway_guardrail_checks_total is the only place where a policy decision becomes a number. Its phase label separates the request guard (phase="Request", the email/override regex, status 400) from the response guard (phase="Response", status 502) configured in 5.5. Gateway Security. Without it, a rejection is indistinguishable from a client-side bug: both look like a 400 to the caller.

A pitfall worth internalizing: agentgateway_requests_total counts what the gateway handled, not what the model did. A rejected request increments it with status="400" and reason="DirectResponse" — the gateway answered directly. Do not build a "model call rate" panel on this counter; use the agent's own agentops_calls_total for that.

What does one gateway log line contain?

Your rejected request left one line behind. Here is one real request line from the host profile, produced by the same rejected model request from 5.5. Gateway Security.

The gateway writes it as a single line of JSON; it is wrapped here so the fields are readable:

{
  "level": "info",
  "time": "2026-07-17T14:45:37.407581Z",
  "scope": "request",
  "gateway": "default/default",
  "listener": "llm",
  "route": "default/route0",
  "endpoint": "host.docker.internal:11434",
  "src.addr": "172.17.0.1:45434",
  "http.method": "POST",
  "http.host": "localhost",
  "http.path": "/v1/chat/completions",
  "http.version": "HTTP/1.1",
  "http.status": 400,
  "protocol": "llm",
  "reason": "DirectResponse",
  "duration": "0ms"
}

Four things to notice:

1. scope selects the line's kind. scope: request is the access log; startup lines carry agentgateway::proxy::gateway or agent_core::readiness instead. Filter on it before anything else: mise run gateway:host:logs | grep '"scope":"request"'.
2. The routing decision is in the line. listener, route, endpoint, and protocol tell you which config block matched, which is the fastest way to prove a request went where you think it did. If endpoint is not the upstream you expect, the bug is in the config, not the agent.
3. reason explains the status. DirectResponse means the gateway itself produced the body without a backend hop — with http.status:400 and protocol:"llm", that is a prompt-guard rejection. A 400 forwarded from the provider would show a real upstream reason instead.
4. No prompt, no response, no caller identity. The line carries metadata only, and src.addr is Docker's bridge address (172.17.0.1), not the original client — the gateway runs in a container and sees the connection from the host bridge. Correlation therefore relies on time plus these request attributes, not on an identifier the line does not have.

Log format is a contract, not a convenience. JSON on stdout means the container runtime captures it, a collector can parse it without regexes, and field names stay stable across the four profiles.

Fields were read from agentgateway v1.4.1 on the host profile; they are the current shape, not a stability guarantee from upstream. Verify before writing a parser against them.

How do you inspect raw metrics?

With the host gateway:

curl -fsS http://localhost:15020/metrics | head -n 20

That gives you the endpoint. Getting Prometheus to read it is a separate question, with one wrinkle worth knowing before you debug it.

Prometheus and the host gateway share the wrapper-owned Docker bridge. Prometheus scrapes the stable agentops-gateway:15020 network alias directly, while the wrapper also publishes 15020 on loopback for your host-side curl. The native-Linux relay is only for loopback-bound MCP, A2A, and model upstreams (5.1. Gateway Setup).

If Compose Prometheus shows the agentgateway target as down, run mise run smoke:host — its container-side check uses that exact alias and network.

scrape_configs:
  - job_name: otel-collector
    static_configs:
      - targets: [otel-collector:8889]
  - job_name: mlflow
    metrics_path: /metrics
    static_configs:
      - targets: [mlflow:5000]
  - job_name: agentgateway
    static_configs:
      - targets: [agentops-gateway:15020]

kubectl -n agentops port-forward svc/agentgateway 15020:15020

Why does the host gateway expose metrics but not traces?

Metrics are pulled, traces are pushed. A scrape target that nobody scrapes costs the gateway nothing. An OTLP exporter is a client with a connection, a queue, and a retry loop, and it fails noisily whenever its collector is absent.

The host observability stack is optional (mise run observability:up). Configuring tracing.otlpEndpoint there would make the default quickstart log exporter errors for a service the learner was never asked to start. The k3d and GKE configs do set it, because their collector is deployed in the same manifest set.

flowchart TD
    Gateway[agentgateway container] -->|stdout JSON| Logs[docker logs / gateway:host:logs]
    Gateway -->|:15020 published for host curl| Published[127.0.0.1:15020]
    Gateway -->|agentops-gateway:15020<br/>shared scoped bridge| Prometheus[Compose Prometheus]
    Prometheus --> Grafana
    Gateway -.->|no OTLP exporter configured| Collector[OTel Collector]
    Agent[ADK agent process] -->|OTLP HTTP when OTEL_* is set| Collector
    Collector --> MLflow[MLflow trace store]

Diagram in words: The gateway writes metadata-only request logs to container stdout. Its metrics reach a loopback host port and Compose Prometheus directly over their shared scoped bridge, then Prometheus feeds Grafana. The host gateway sends no OTLP traces; only an explicitly configured agent sends OTLP HTTP to the collector, which stores traces in MLflow.

The dashed edge is the deliberate gap: on the host, gateway spans do not exist, and only the agent's own spans reach MLflow. Do not spend an afternoon looking for a gateway hop in a host trace.

The same wrapper that turns the metrics surface on turns the debug surface off. render_base_config sets three fields together:

.config.statsAddr = "0.0.0.0:15020" |
.config.readinessAddr = "0.0.0.0:15021" |
.config.adminAddr = "off"

See gateway-host.sh, and scripts/check-infra.sh asserts all three on the rendered output so the trio cannot drift.

The distinction is the point of the exercise. Metrics and readiness are read-only, bounded, and consumed by a machine. An admin interface is an unauthenticated control surface on a process that holds your policies. Observability does not require it, so it is not exposed — even on loopback, even in a lab. If you need it while developing agentgateway itself, turn it on knowingly and locally, not by default.

How do traces reach MLflow?

In the cluster the gateway also exports traces; on the host it does not, so on this page a gateway hop never appears in MLflow.

flowchart TD
    Gateway[Kubernetes agentgateway] -->|OTLP gRPC| Collector[OTel Collector]
    Agent[ADK agent] -->|OTLP HTTP| Collector
    Collector -->|OTLP HTTP| MLflow[MLflow trace store/UI]
    Collector -->|spanmetrics| Prometheus
    Prometheus --> Grafana

The collector batches traces, applies a memory limiter, exports them to the self-hosted MLflow service, and exposes span-derived RED metrics (rate, errors, duration) for Prometheus to scrape. Chapter 7 inspects the implementation.

tracing:
  otlpEndpoint: http://otel-collector.agentops.svc.cluster.local:4317
  randomSampling: true

Are prompts stored in traces?

No. The agent opts out explicitly rather than relying on library defaults, in setup_telemetry():

# Content capture is opt-in: traces retain timing, model, tool, token, and
# status metadata without duplicating user prompts or model responses.
os.environ.setdefault("ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS", "false")
os.environ.setdefault("OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT", "false")

That default matters more than it looks. Trace content capture is the single highest-risk default in any GenAI stack: it turns an operational store into a copy of every prompt and completion, replicated wherever traces go.

See telemetry.py. setdefault is the deliberate choice: an operator who exports either variable keeps control, but nobody gets content capture by accident. On the gateway side, none of the four configs enable prompt logging, and the access log shown above carries no message body.

Do not read that as "traces are anonymous". Three gaps remain:

• Metadata is still sensitive: model names, tool names, timings, token counts, and error types describe your operation.
• Span attributes you add yourself are not covered by these two variables.
• Application log records take a separate path with its own local redaction and truncation, and any logging a learner adds to a tool bypasses both.

Review every exporter and its retention before pointing this stack at real data. Where application log records go is owned by 7.2. Monitoring.

How do you correlate a failure?

Read the signals from the outside in: caller first, gateway second, application last. Correlation is a discipline of ordering signals by what each one can and cannot know.

1. Start from the caller. The A2A client's status and wall-clock time bound the search window. A client-side timeout with no gateway log line means you never reached the gateway.
2. Ask the gateway what it did. Filter "scope":"request" around that timestamp and read listener, route, endpoint, http.status, and reason. This tells you whether the request was rejected (DirectResponse), proxied and failed upstream, or proxied and succeeded slowly (duration).
3. Confirm with a counter. A single log line can be a coincidence; agentgateway_requests_total by status and agentgateway_guardrail_checks_total by action tell you whether it is one event or a pattern.
4. Then go inside the application. The agent's own trace in MLflow, its Loki log lines, and the persisted audit state (7.6. Governance) cover everything behind the proxy — and only exist for requests the gateway actually forwarded.

On the host, the gateway hop and the application trace are two separate stores joined by time and request attributes, because the host gateway emits no spans. In Kubernetes the gateway also exports OTLP, so shared trace context — the ids a request carries from hop to hop — connects those hops directly, and step 4 becomes a click instead of a timestamp comparison.

Either way, keep correlation identifiers in traces and logs. Never promote them to Prometheus labels, where they become an unbounded cardinality (one new series per identifier) and a privacy problem at once.

What proves this page worked?

Issue one allowed MCP call and one rejected model request, then compare the gateway log and counters.

1. Gateway JSON logs record both outcomes, each with "scope":"request" and the expected listener, http.status, and reason.
2. :15020/metrics changes: agentgateway_requests_total gains a status="200" and a status="400" series, and agentgateway_guardrail_checks_total{phase="Request",action="Reject"} increments by exactly one per rejection.

These checks need neither a model call nor Compose. If you already have host observability running, the gateway dashboard should reflect those events through the shared bridge. An agent trace requires a separate traced agent turn; direct MCP and rejected-model curls do not create one. 7.1. Tracing owns that exercise.

After recording the evidence, stop every Chapter 5 process before Kubernetes reuses the same local ports:

mise run gateway:host:stop
# Only if you started the optional host observability stack:
mise run observability:down

Press Ctrl-C in any foreground MCP, A2A, agent, or gateway terminal still running. Do not add -v: preserving the observability volumes keeps this evidence available for Chapter 7.

You are done when:

• docker container logs --since 10m agentops-host-gateway | grep '"scope":"request"' shows the rejected request with "http.status":400 and "reason":"DirectResponse" — that form terminates, while mise run gateway:host:logs follows the stream until you interrupt it.
• agentgateway_guardrail_checks_total{phase="Request",action="Reject"} on :15020/metrics went up by exactly one for each rejection you sent.
• You can explain why the host gateway produces no OTLP span and why direct curls do not prove application tracing.
• Any optional dashboard evidence is identified separately from the required raw log/metric checks.
• Your host gateway and optional Compose stack are stopped, and your Chapter 5 processes no longer own a course port.

Continue to 6. Platform when the evidence is recorded and the host processes are stopped, so the cluster can reuse their ports.