0.3. Ecosystem

Which components form the course stack?

Seven layers, one owner each. ADK runs the agent, agentgateway carries the traffic, kagent runs it on Kubernetes, Gemini provides the default hosted model (Ollama is optional), and MLflow, OpenTelemetry, Prometheus, and Grafana record what happened.

An AgentOps stack is not one framework; it is a set of boundaries — one job, one owner, and a stable contract across the edge. The chapters that follow build each boundary in depth. Treat this page as the index you return to when you lose the shape of the whole.

Each component owns one clear boundary:

The table reads top to bottom; the ownership actually nests as layered planes. In the diagram below the boxes show who owns what, and the arrows only show who calls whom.

The repository README.md carries a different diagram, one that traces a single request end to end. This one groups components by the boundary they own:

flowchart TD
    subgraph App["Application - Google ADK"]
        Agent["AgentOps Agent<br/>loop, sessions, tools, A2A serving"]
    end
    subgraph Data["Data plane - agentgateway"]
        GW["listeners :3000 MCP, :3001 A2A, :4000 model"]
    end
    subgraph Model["Model runtime"]
        Gemini["Gemini API<br/>default hosted model"]
        Ollama["Ollama + Qwen3<br/>optional local inference"]
        Vertex["Vertex AI Gemini<br/>optional, WIF"]
    end
    subgraph Platform["Platform - kagent + Kubernetes"]
        KA["BYO Agent, ModelConfig, RemoteMCPServer"]
    end
    subgraph Obs["Observability"]
        OTel["OTel Collector"]
        MLflow["MLflow traces"]
        PromGraf["Prometheus + Grafana"]
        Loki["Loki logs"]
    end
    Agent -->|MCP + model| GW
    Agent -->|Part I: native Gemini| Gemini
    GW -->|host / k3d: HTTPS| Gemini
    GW -->|optional local profile| Ollama
    GW -->|GKE| Vertex
    KA -. reconciles .-> Agent
    Agent -->|OTLP| OTel
    OTel --> MLflow
    OTel --> PromGraf
    OTel --> Loki

Diagram in words: ADK calls native Gemini in Part I. In Part II, agentgateway routes MCP and model calls; it reaches Gemini API, optional Ollama, or GKE Vertex. kagent manages the application on Kubernetes. ADK sends OTLP to the collector, which routes traces, metrics, and logs to MLflow, Prometheus/Grafana, and Loki.

The model plane is owned by 0.4. Providers; the rest of this page walks each remaining plane in turn.

Why keep each responsibility in a separate tool?

An all-in-one agent framework bundles the loop, the model client, the router, the deployment target, and the telemetry backend into one dependency. That is fast to start and expensive to change. You inherit its opinion at every layer, and swapping one piece — a different model, a different runtime, a different trace store — drags the others with it.

This course takes the opposite bet. Each boundary is a contract, and the payoff is concrete rather than aesthetic:

1. The application does not change when the platform does. The same OCI image (agentops-agent:dev) runs as a plain host process and as a kagent workload. infra/kagent/agent.yaml declares type: BYO, so kagent wraps a Kubernetes lifecycle around the course's own container instead of generating a new one. Nothing in the Python code knows whether kagent is present.
2. The application does not change when the model backend does. The agent speaks one OpenAI-compatible contract to gateway :4000. Behind the gateway, Gemini API (host and local-gemini), Ollama (optional local), and Vertex AI (GKE) are separate validated profiles. Part I instead uses ADK's native Gemini client; the GKE overlay obtains identity through Workload Identity Federation and mounts no secret.
3. The application does not know its observability backend. It emits OTLP and stops there. MLflow, Prometheus, and Loki are wiring in the collector, not imports in the agent.

The honest cost of this design is coordination: more moving parts means more versions that must agree, which is exactly what What should you verify before upgrading the stack? exists to manage. The trade is deliberate — a little version bookkeeping in exchange for boundaries you can test and replace one at a time.

What does Google ADK own?

ADK is the Python framework running inside the application process. It defines agents and tools, drives the model-and-tool loop, persists sessions, runs callbacks, and serves the AgentOps Agent over A2A.

The repository owns ADK's compatible range in agents/python/pyproject.toml and locks the exact resolution in uv.lock. It never runs against whatever version happens to be newest. The [a2a] extra pulls in the serving layer, and the comment in the manifest deliberately avoids the unused Spanner db extra to keep the dependency tree tight.

ADK is not the gateway, the model provider, the Kubernetes operator, or the observability backend. Keeping those out of the framework is what makes the separation above possible.

The concrete code is owned by 2.1. First Agent and 2.2. Models: how the root agent is constructed, how the model is built, how tools and callbacks attach. This page only fixes ADK's boundary.

What does agentgateway own?

agentgateway is the course's data plane — the proxy every request passes through on its way somewhere else. It is open source, speaks HTTP and gRPC, and carries ordinary service traffic plus AI-native MCP, A2A, and LLM traffic.

It is the single place the course applies routing, rate limits, and content policy, so the application stays a plain client. It exposes three explicit listeners:

1. :3000 for MCP tool traffic.
2. :3001 for A2A traffic.
3. :4000 for an OpenAI-compatible model endpoint.

Those three are the front door. The rest of the stable network contract is two more ports the gateway itself exposes, plus the raw upstream services it fronts:

1. :15020 — the gateway's own metrics.
2. :15021 — the host wrapper's readiness probe.
3. :8000 — behind the gateway: the raw MCP server it fronts, which stays off the workstation's LAN interfaces.
4. :8080 — behind the gateway: the raw A2A server it fronts, kept off the LAN interfaces the same way.

The same contract ships in three profiles under infra/agentgateway/{host,k3d,gke}: a loopback host data plane, an in-cluster k3d variant, and the GKE variant that swaps the model backend for Vertex. The host profile is grounded in infra/agentgateway/host/config.yaml. Its three binds carry per-minute rate limits (120 MCP, 60 A2A, 30 model), an mcpAuthorization allowlist of the six read-only tools, and a prompt/response guard on the model route.

Part I uses native Gemini by default. Chapter 5 selects the OpenAI-compatible client and the gateway endpoint; the gateway translates to Gemini and holds its credential. Optional Ollama stays an explicit alternative. Here the point is only that one component owns connectivity for every protocol.

The policy details behind each listener are owned by 5.0. Gateway and 5.5. Gateway Security.

What is the stable port contract?

Three ports are the ones to remember: MCP :3000, A2A :3001, and the model route :4000 — the gateway's three listeners.

Every other port in the course is fixed too, so every task, probe, and troubleshooting step names the same numbers. You do not need to memorize them; come back when a port collides.

When two of these collide with something already listening, 0.6. Troubleshooting walks the ss -ltnp and gateway-status checks that free them.

What does kagent own?

kagent is a CNCF Sandbox project that manages agents as Kubernetes custom resources. A custom resource is an object type a cluster gains from an add-on, then treats like any built-in one.

The course installs the stable Helm chart pinned in infra/helmfile.yaml and drives it entirely through the kagent.dev/v1alpha2 API. Three manifests under infra/kagent/ declare the whole platform contract:

1. agent.yaml — a kind: Agent with type: BYO, which tells kagent to run the course's own A2A image (agentops-agent:dev) under a hardened pod spec (non-root, read-only root filesystem, dropped capabilities) rather than synthesizing one.
2. modelconfig.yaml — a kind: ModelConfig that points the platform's model access at the in-cluster agentgateway :4000 endpoint, so even kagent's view of the model goes through the data plane.
3. toolserver.yaml — a kind: RemoteMCPServer describing the governed read-only tools reached over STREAMABLE_HTTP through the gateway :3000/mcp route.

The application stays responsible for its logic and sessions; kagent is responsible for translating those custom resources into a running Kubernetes workload. That clean split is exactly why the same container runs without kagent during host development. The full walkthrough is owned by 6.0. Platform.

Why use both MLflow and OpenTelemetry?

OpenTelemetry is the transport and semantic layer; MLflow is the agent-specific workspace on top of it. They are not redundant — one moves signals, the other curates them.

The collector is a single vendor-neutral OTLP entry point on :4317/:4318. infra/observability/otel-collector.yaml fans the incoming signals out three ways:

1. Traces to MLflow (otlp_http/mlflow at mlflow:5000).
2. Span-derived RED metrics — rate, errors, duration — to Prometheus (the span_metrics connector feeding a prometheus exporter on :8889).
3. Logs to Loki (otlp_http/loki at loki:3100).

MLflow then adds prompt versions, evaluations, runs, and trace exploration around those traces. Neither backend hides behind a hosted SaaS account; the MLflow server itself is a locked, non-root image (infra/mlflow/Dockerfile).

The reproducibility side (prompts, runs, logged models) is owned by 7.0. Reproducibility and tracing by 7.1. Tracing; this page only names the boundary and the fan-out.

How do the components fit together at runtime?

An agent turn can make several model and tool calls across these boundaries. The application handles the request, the data plane brokers each external call, the model plane produces tokens, and the observability plane records what happened — no component reaches past its own edge:

sequenceDiagram
    participant Client
    participant GW as agentgateway
    participant ADK as ADK loop
    participant Model as Gemini API (or optional provider)
    participant OTel as OTel Collector
    Client->>GW: A2A message/send :3001
    GW->>ADK: forward task
    loop until the model answers
        ADK->>GW: MCP tool call :3000
        GW-->>ADK: typed tool result
        ADK->>GW: model completion :4000
        GW->>Model: provider request
        Model-->>GW: tokens
        GW-->>ADK: completion
    end
    ADK-->>GW: final A2A result
    GW-->>Client: response
    ADK--)OTel: OTLP spans :4317 / :4318

Diagram in words: The client sends A2A through agentgateway to ADK. ADK repeatedly requests read tools and model completions through gateway listeners. The gateway forwards model requests to the configured provider and returns responses. ADK sends the final response back through the gateway and emits telemetry to the collector.

1. A client (an engineer, the offline web client, or another agent) sends an A2A message/send to gateway :3001; ADK runs the loop.
2. When the model needs a tool, ADK calls MCP through gateway :3000; when it needs a completion, it calls the OpenAI-compatible endpoint on :4000, which the gateway routes to Gemini, optional Ollama, or Vertex.
3. Throughout, ADK and the gateway emit OTLP to the collector, which splits the signals apart.

One OTLP export becomes three things: a trace you can open, metrics you can alert on, and logs you can correlate by trace id. 7.2. Monitoring walks that correlation end to end.

flowchart TD
    In["Agent + gateway<br/>OTLP :4317 / :4318"] --> Recv["otlp receiver"]
    Recv --> Traces["traces pipeline"]
    Recv --> Logs["logs pipeline"]
    Recv -->|direct OTLP counters| Metrics["metrics pipeline"]
    Traces --> ML["otlp_http/mlflow to mlflow:5000"]
    Traces --> SM["span_metrics connector"]
    SM --> Metrics
    Metrics --> PromExp["prometheus exporter :8889"]
    Logs --> LokiExp["otlp_http/loki to loki:3100"]

Diagram in words: The client sends A2A through agentgateway to ADK. ADK repeatedly requests read tools and model completions through gateway listeners. The gateway forwards model requests to the configured provider and returns responses. ADK sends the final response back through the gateway and emits telemetry to the collector.

Because traces feed both MLflow and the span-metrics connector, one turn becomes a trace you can open, a set of RED metrics you can alert on, and a stream of logs you can correlate by trace id — from a single OTLP export. [7.2. Monitoring](../7.%20Observability/7.2.%20Monitoring.md) walks that three-pillar correlation end to end.

What do Prometheus and Grafana own?

Prometheus owns the metric store and the alert engine; Grafana owns the read-only visualization.

Prometheus scrapes two targets: the collector's :8889 endpoint (the span-derived series such as agentops_calls_total and agentops_duration_seconds_bucket) plus the gateway's own :15020 metrics. It then evaluates the rules in infra/observability/prometheus-rules.yml: two recording rules and six alerts — AgentErrorBudgetBurn, ObservabilityCollectorDown, AgentTurnLatencyP95High, AgentTokenTelemetryMissing, AgentInjectionNeutralizedSpike, and AgentTriageSchemaFailures.

Grafana provisions the AgentOps overview dashboard from infra/observability/grafana/dashboards/agentops.json: six metric panels (agent and gateway request rate, p95 latency, error ratio, guardrail rejects) and one Loki logs panel filterable by trace id.

On the host Compose profile these surface on well-known loopback ports — Prometheus on :9090, Grafana on :3002, MLflow on :5000 — while the Kubernetes overlays keep them as ClusterIPs behind port-forwards. Every alert is grounded in a metric the stack verifiably exports, and none routes to an external pager.

Read the names as a preview, not as homework. 7.2. Monitoring owns the alert semantics, the dashboard, and the alert-response runbooks.

Which open standards connect the components?

Boundaries only hold if the contract across them is a standard, not a private API:

1. MCP describes how an agent discovers and invokes tools and resources.
2. A2A describes how independently deployed agents advertise capabilities and exchange tasks.
3. OpenTelemetry describes and transports traces, metrics, and logs.
4. OCI images package the same application for local and cloud runtimes.
5. AGENTS.md gives coding agents repository-local instructions, a convention this repository dogfoods.

Standards reduce coupling; they do not guarantee interoperability by themselves. Two implementations can each claim MCP or A2A and still disagree on a version, an auth mode, or a streaming detail. The course pins concrete implementations and verifies the versions together rather than trusting the labels.

How do these ideas transfer to other agent frameworks?

This conceptual crosswalk was reviewed on 31 July 2026 against the official Google ADK and LangGraph documentation, plus the official smolagents and Microsoft Agent Framework documentation.

It maps responsibilities, not APIs. ADK remains the only implemented and release-gated runtime in this repository; the other columns tell you which concept to look for when transferring the engineering method.

The portable lesson is ownership: identify the trusted state boundary, the callable capability boundary, the resumable workflow boundary, and the evidence boundary before translating class names. Do not assume that two similarly named features preserve the same approval, persistence, or privacy guarantees.

Where do the AAIF and CNCF fit?

The Agentic AI Foundation is a Linux Foundation home for open agent infrastructure, including agentgateway, MCP, and A2A. The Cloud Native Computing Foundation stewards Kubernetes, Prometheus, and kagent among many other projects.

Foundation status is useful governance context — a signal about open licensing and shared maintenance — not evidence that a pre-1.0 API is stable. kagent and agentgateway are young; treat their v1alpha2 and pre-2.0 surfaces as movable and let the pins, not the foundation logo, decide what you run.

What should you verify before upgrading the stack?

Six pinned versions have to agree: ADK, agentgateway, kagent, MLflow, the OpenTelemetry Collector, and Python.

Separate boundaries mean separate release cadences, so an upgrade is a coordination exercise, not a single bump. The repository holds the authoritative numbers, so read the pin files rather than a floating latest tag.

Upgrade one component at a time, read its release notes and schema changes, regenerate the lock, and run the local validation gate:

mise run install:maintainer
mise run format
mise run check
mise run test

These commands call no model, cluster, or cloud resource. Dependency and image audits may still refresh advisory data over the network.

For an infrastructure upgrade, also render and validate the manifests before mise run platform:dev; the task derives the required source revision before Skaffold builds. Never infer compatibility from a project name, a shared foundation, or a latest tag; the whole point of the boundaries is that each contract is verified where it is pinned.

What proves this page worked?

Nothing here needs to run. The check is whether the map still holds once you close the page.

You are done when:

• You can name the owner of each layer in the table: the agent loop, the traffic, the Kubernetes lifecycle, the model, and the signals.
• You can say in one sentence why the agent code does not change when the model backend or the deployment target changes.
• You know which section to reopen when a port collides or a pinned version needs bumping.

Continue to 0.4. Providers when you can name the tool that owns each boundary without scrolling back up.