0.2. AgentOps

What is AgentOps?

AgentOps is the practice of building, evaluating, securing, deploying, and operating AI agents as reliable production software. Getting an agent to answer correctly once is easy; keeping it correct, safe, affordable, and observable as it runs against real traffic is the hard part. AgentOps is the set of habits and tools that make that repeatable.

It is the agent-shaped sibling of MLOps and LLMOps. Where those disciplines operate models, AgentOps operates something that acts: an agent calls tools (0.1. Agents), takes multi-step decisions, and can change the world. That autonomy is exactly what makes evaluation, guardrails, and observability first-class concerns rather than afterthoughts.

This course does not teach AgentOps in the abstract. Every claim below is grounded in one completed reference: the AgentOps Agent under agents/python/. A phase is never a bullet point you have to imagine — it is a file you can open and a command or observation you can inspect.

Where does AgentOps begin?

AgentOps begins after you already have a working agent. That is the framing this course is built on: the main branch is a completed, executable reference that you inspect and extend, not a collection of illustrative snippets. A first correct answer is a demo, and demos are cheap. The discipline starts the moment you ask the production questions a demo never has to answer:

• Will the same prompt still call the right tool next week, on a new model version?
• What happens when a tool call would change real state, or racks up an open-ended bill, or carries an instruction planted by an attacker?
• When it misbehaves at 3 a.m., can you see why from a trace instead of guessing?

Those questions are what separates a prototype from an operated system, and they are exactly the concerns the phases below make routine. This page assumes you already decided an agent is the right tool and now have to run the thing.

If you are still choosing, that decision belongs one page earlier: 0.1. Agents — when should you not use an agent?.

How do MLOps, LLMOps, and AgentOps relate?

They are three concentric layers of the same lineage. Each outer layer keeps everything the inner one does and adds the new concern its unit of work introduces:

flowchart TB
    subgraph AgentOps["AgentOps — adds tool use, actions, guardrails, HITL"]
        subgraph LLMOps["LLMOps — adds prompts, context, retrieval, token cost"]
            subgraph MLOps["MLOps — trains, versions, serves, monitors models"]
                Core["Model lifecycle:<br/>data · training · deployment · drift"]
            end
        end
    end

• MLOps operates trained models: data pipelines, training, versioning, deployment, and drift monitoring. The unit of work is a model you build.
• LLMOps operates large language models you mostly consume rather than train: prompts, context windows, retrieval, token cost, latency, and output evaluation.
• AgentOps operates agents built on those LLMs: tool use, multi-step control flow, autonomy, guardrails, human-in-the-loop, and the safety of actions, not just text.

Everything you already know about MLOps still applies: reproducibility, testing, CI/CD, monitoring. AgentOps inherits those and adds the concerns that come with an autonomous, tool-using system. The named tools this course uses to cover each layer — ADK, agentgateway, kagent, MLflow, OpenTelemetry, Ollama — are surveyed in 0.3. Ecosystem.

What is the AgentOps lifecycle?

The lifecycle is a loop of six phases. You build an agent, give it capabilities, prove it works, and ship it through a gateway. Then you run it as a platform workload, watch it, and feed what you learn back into the next iteration.

What makes it a discipline rather than a slogan is that every node has an artifact that implements it and a check or observation that shows how it behaves:

flowchart TD
    Build["Build<br/>composition.py · model.py"]
    Capabilities["Capabilities<br/>tools · skills · MCP · memory"]
    Quality["Quality<br/>test gate · model-backed eval evidence"]
    Gateway["Gateway<br/>infra/agentgateway"]
    Platform["Platform<br/>infra/kagent · infra/k8s"]
    Observe["Observe<br/>telemetry.py → OTLP → MLflow / Prometheus"]
    Build --> Capabilities --> Quality --> Gateway --> Platform --> Observe
    Observe -->|distil a trace into a regression case, re-run| Build

Each pass around the loop makes the agent a little more production-ready. The next section names the artifact and proof behind each node, so the loop is not an abstraction. The section after it explains what the feedback edge actually consists of in this repository.

Nothing here expects you to recognize the names in that diagram yet. Each has a one-line definition in 0.7. Glossary, and the chapter that owns it explains it in full.

How does each phase show up in the reference agent?

This is where AgentOps stops being a diagram. Each phase maps to a concrete file or directory in this repository:

All source paths are under agents/python/src/agent/. You are not meant to memorize the file names. The point is that "harden it" and "watch it" are not aspirations here — they are guardrails.py and telemetry.py. Each chapter walks its row in full.

Each phase also has a command that checks or observes its property. Most belong to later chapters, so they are collapsed here rather than dropped:

How is operating an agent different?

Compared to a stateless model endpoint, an agent introduces four new failure modes. Each one already ships a concrete defense you can read, not just a warning to heed:

Each defense also has an edge the chapter is explicit about. That is why each defense is layered rather than trusted alone.

What closes the AgentOps loop?

The loop's hardest edge is the one from Observe back to Build. "Feed what you learn back" is easy to assert and easy to leave hollow.

Three concrete habits close it in this repository. None of them is an automated online scorer or a feedback API — do not read more into the loop than ships:

1. A bad trace becomes a regression case. When a trace shows a wrong trajectory or an unsafe proposal, you distil it into one eval case that names that behavior. mise run eval:validate gates its structure; model-backed evals report whether the agent passes it. One new failing case does not necessarily push the aggregate 25% floor below its threshold, so inspect the named case instead of treating the aggregate as its gate.
2. Every change re-runs the same gates. The continuity is enforced by re-running mise run format, mise run check, and mise run test after a change, so the reference cannot silently rot into a broken snapshot. That test gate enforces ≥95% combined line-and-branch coverage: the share of if/else paths the tests actually exercise.
3. Time-sensitive claims get re-verified on a schedule. The .github/workflows/freshness.yml workflow opens one quarterly tracking issue so pinned versions, prices, and model names are re-checked each release cycle instead of drifting.

Observability itself is the input to all three. 7. Observability is where the trace, metric, and audit evidence that feeds the next iteration is produced.

How does the lifecycle map to the course?

Each phase of the lifecycle is a chapter, so the table of contents is the lifecycle:

Before that, 1. Setup prepares your environment, and 8. Community covers sharing and sustaining what you build.

That last chapter ends with 8.7. Capstone, which turns this completed reference into an evidence-backed agent platform for your own domain.

What proves this page worked?

This page has nothing to run. What it has to leave behind is a map you can redraw from memory.

You are done when:

• You can name the six phases in order: Build, Capabilities, Quality, Gateway, Platform, Observe.
• You can name the course chapter that owns each phase, without scrolling back to the table above.
• You can name one reference artifact for any two phases, such as guardrails.py for Quality.
• You can say in your own words why an agent needs guardrails and evaluation that a stateless model endpoint does not.
• You can describe the Observe-to-Build edge concretely: a bad trace becomes one validated eval case, and model-backed runs report that case separately from the aggregate floor.

Continue to 0.3. Ecosystem when you can name the six phases in order and the chapter that owns each.