Skip to content

4. Quality

In one glance

  • You will: See how the chapter's seven quality layers fit together and which page owns each check.
  • You need: Workshop step 6 for grader calibration; the contributor installation for reference tests and security checks.
  • Time: about 5 minutes, orientation.

Part I — Agent development. Work through 2.6. Workshop and consult this chapter when the exercise introduces its subject. Python fundamentals are assumed; no Kubernetes knowledge is needed here.

How will you make the agent trustworthy?

Trust comes in layers. Each page below adds one, and each layer catches a class of failure the cheaper layers below it cannot.

Your agent now holds a conversation (Chapter 2) and has bounded capabilities (Chapter 3). This chapter makes it defensible.

The early checkpoints need no model, account, network, or bill. The full maintainer security gate may refresh advisory data. The marker on each line says what that page's own checkpoint needs; run mise run config:check before a model-backed one. doctor:model is specific to optional Ollama inference.

  • 4.0. Typing (concept · offline): Python typing with ty, parsing tool I/O at the boundary.
  • 4.1. Linting (hands-on · offline): Lint and format with ruff and dprint.
  • 4.2. Testing (hands-on · offline): Fast, offline unit tests with pytest, against an isolated dataset copy.
  • 4.3. Metrics (reference · needs a model): A scorecard of deterministic gates, model-backed evidence, and observed operational indicators.
  • 4.4. Evaluations (hands-on · offline first): Recorded-answer grader calibration, optional MLflow recording, then explicitly selected live evaluations.
  • 4.5. Guardrails (hands-on · offline, except the last checkpoint step): Boundary redaction, stable errors, confirmation, transactions, and audit evidence.
  • 4.6. Security (hands-on · model-free; scans may use network): Threat modeling, offline adversarial regressions, identity, and supply-chain scanning.

Expect to write code, not just read. 4.5. Guardrails carries the chapter's required ## Your turn drill — turn a guardrail into a test that fails if it ever weakens — and 4.4. Evaluations distinguishes grader calibration from live model evidence.

Where is gate versus evidence explained?

4.7. Evaluation Reference owns the definition, workflow map, and task-by-task decision. This index only marks each page's prerequisites so you can enter the chapter without learning the same policy twice.

What proves this chapter worked?

Two offline commands cover the chapter: the full test suite, then the adversarial regression suite.

cd agents/python
mise run test
mise run redteam

Neither needs a model, a provider key, or a network.

You are done when:

  • mise run test passes, including the enforced 95% combined line-and-branch coverage floor.
  • mise run redteam passes every adversarial case in tests/test_security.py.
  • The chapter's required drill is done: the ## Your turn in 4.5. Guardrails added a regression you watched fail against a deliberately weakened guard, then restored.
  • You can use the page markers above to say which checkpoints need a configured model and which run offline.
  • You can point to 4.7. Evaluation Reference for the chapter's gate-versus-evidence policy.
  • Without reopening Chapter 3: you can name which of the six memory stores a value belongs in when it must survive the next turn but not the next session.

Continue to 4.0. Typing when you know the first three pages need no model or provider account.