3.4. Memory

What retrieval adds when the request carries no runbook slug

Memory in an agent is plural by construction: six stores hold what it remembers. Those are the conversation, an A2A task, operational state, long-term notes, shared knowledge, and per-session state — each with a different owner and lifetime. One word hides all six. Retrieval finds the right document inside the largest of those stores, ranking candidates against a query with no model involved.

Both fail silently. A fact written into the wrong store inherits the wrong access rules and the wrong lifetime, and nothing complains at the time. A knowledge base reachable only by an exact slug returns nothing for a request that arrives as a symptom sentence: “checkout is slow”, “we are seeing 503s from stock lookups”.

This page gives you the store map, ranking you can watch, compaction as the bound on history, and the conditions under which embeddings earn their cost. An incident record carries a runbook field, so a slug lookup covers any request that names its procedure; the case worth testing names none.

Ask the knowledge base directly, using the MCP server you started in 3.3. MCP so no model is involved and the ranking is entirely the retrieval code’s doing. Predict first: of the seven runbooks in the corpus, which one should come back for a sentence about crash-looping pods and 503s, and does the word “service” — which appears in nearly all of them — help or hurt?

curl -sN -X POST http://127.0.0.1:8000/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_runbooks","arguments":{"query":"inventory pods are crash-looping and stock lookups return 503"}}}' \
  | sed -n 's/^data: //p' | jq '.result.structuredContent | {count, retrieval, slugs: [.runbooks[].slug]}'

{
  "count": 3,
  "retrieval": "keyword",
  "slugs": ["service-down", "disk-full", "cascade-failure"]
}

service-down first, which is the procedure INC-002 actually links to — reached from the symptom sentence rather than from the record that names it. Note the retrieval field: the answer says how it was ranked, in the payload the model reads rather than only in a log line. Both remaining sections turn on it.

You just watched retrieval pick the right procedure from a slug-free sentence, with no model call and no service to sign up for.

What each memory store owns, and what shares the context window

The six, by scope and implementation:

A note one caller wrote is not shared knowledge, and an A2A task is not a conversation. Choose the store from the lifetime and the authority the fact needs before you write any code, because moving a fact between these later means moving its access rules with it.

What deserves to survive a session is narrow: facts the next investigation needs and cannot safely recompute — an attempted remediation, an observed outcome, an explicit decision. The save_incident_note and recall_incident_context tools make those writes and reads visible in traces, and notes are keyed by the caller’s stable identity — the durable subject the runtime resolves rather than a per-request value — checked against known incidents, length-bounded, and redacted before they persist.

Making them explicit tools is a deliberate trade, and the alternative it beats is ADK’s own. The notes store is a memory.Service implementation — agents/go/memory/service.go declares var _ memory.Service = (*Notes)(nil), a third alongside ADK’s in-memory and Vertex AI ones, because a course that promises durable state across a restart needs one that survives a restart. Implementing that interface buys ADK’s stock consumption path: load_memory, which searches memory on demand, and preload_memory, which injects matching memories into the prompt before the model sees it. This course wires neither, and takes the explicit tools instead: silent context injection hides what was read, automatic history ingestion hides what was persisted, and an explicit call is observable in a trace, assertable in a test, and reachable by policy. The cost is behavioral: the model has to decide to recall or save, so the instruction requires a recall at the start of an investigation and trajectory evaluation checks that ordering.

The unauthenticated A2A path derives a synthetic user from its context id, so a new context is a new logical user with no memory of the last one. Human-level memory needs an authenticated, durable subject propagated by a trusted gateway — which is 5.5. Gateway Security’s job, not this page’s.

Those six stores do not compete for disk. Their selected contents compete for the model’s context window, which on any given request can hold the root instruction, every tool declaration, conversation history, tool calls and their results, retrieved runbooks, long-term notes, and the current prompt.

The policy plane — the callbacks that run around every model call — answers with compaction: replacing older messages with one deterministic marker while keeping the newest messages and tool result:

func (p *Policy) CompactHistory(_ agent.Context, request *model.LLMRequest) (*model.LLMResponse, error) {
	// Compaction depends only on the outgoing request, never on the context.
	if p.maxHistoryMessages == nil || request == nil {
		return nil, nil
	}
	keep := *p.maxHistoryMessages
	contents := request.Contents
	if len(contents) <= keep {
		return nil, nil
	}

	cut := len(contents) - keep
	// Prefer a standalone message boundary, but a request can end mid-tool-loop
	// with only function responses in the retained tail. Stopping one short of
	// the end keeps at least the newest result: replacing the entire request
	// with a marker would discard the evidence the model just asked for.
	for cut < len(contents)-1 && hasFunctionResponse(contents[cut]) {
		cut++
	}
	request.Contents = slices.Concat([]*genai.Content{compactionMarker(contents[:cut])}, contents[cut:])
	return nil, nil
}

The loop that advances cut past function responses exists because a request can end mid-tool-loop: stopping one short keeps the newest tool result instead of replacing what the model just asked for. The rewrite touches only the outgoing request — ADK rebuilds contents from stored session events every turn, so nothing is deleted from the session and the marker never accumulates. Set AGENT_MAX_HISTORY_MESSAGES with headroom: one user turn can create several messages, and a tight value removes context the model still needs.

When the window is tight, drop cheapest-to-lose first and hard-won evidence last: duplicated instructions, then irrelevant retrieved text, then whole documents in favor of the exact runbook sections you need, then catalog descriptions — and only ever last, recent tool results. Compaction is a cost and availability control, not a substitute for choosing better sources.

Runbooks and logs are data even when committed locally: tool outputs are wrapped as untrusted content and screened for embedded instructions before they return to the model. The single exception is the locally constructed skill loader from 3.2. Skills, whose reviewed body bypasses injection neutralization but still gets recursive secret and PII redaction — keyed on the concrete tool value, not a name an MCP server could choose for itself.

How keyword and semantic retrieval differ, and when embeddings pay

When an incident already supplies a slug, use it. The exact read is the whole tool:

func (m *Memory) runGetRunbook(ctx agent.Context, args GetRunbookArgs) (GetRunbookResult, error) {
	var result GetRunbookResult
	err := m.guard(callContext(ctx), GetRunbookToolName, func(ctx context.Context) error {
		var err error
		result, err = m.readRunbook(ctx, args.Slug)
		return err
	})
	return result, err
}

runGetRunbook runs readRunbook inside the same guard the other reads use, and the inner reader normalizes the model-controlled slug into a validated domain value before any filesystem access. A traversal string cannot be expressed as that type, so it never becomes a path — the same parse-at-the-boundary rule from 3.1. Tools, protecting a filesystem instead of a database.

Free-text search is the fallback when no trusted slug exists, and the default scorer is a TF-IDF-style ranker: it tokenizes every runbook, weights a term higher when it is rare across the corpus so ubiquitous words cannot dominate, adds a large flat boost when a term appears in a runbook’s slug, and breaks score ties by slug so equal scores always order the same way. It is fast, inspectable, account-free, and reproducible.

It also misses paraphrases. “Checkout is slow” does not contain the word “latency”, and no boost tuning changes that. That is the argument for embeddings; here is what it costs.

Turn semantic retrieval on with AGENT_SEMANTIC_RETRIEVAL=true and the same query is answered by cosine distance over locally computed embeddings instead. A brute-force scan, deliberately: the runbook corpus is well under a hundred vectors at a few hundred dimensions, scanning every one takes microseconds, and the whole index is a few hundred kilobytes. At that size a vector extension buys no useful latency and costs deployment complexity and cgo — and cgo is exactly what the static distroless binary from 3.0. Packaging rules out. Vectors are SQLite blobs; cosine distance is computed in process. Add an approximate index when a measurement says the simple scan has become material, not before.

An index is a cache, and a cache that cannot name its inputs cannot be invalidated, so it carries a typed Provenance record binding it to everything that could: CorpusSHA256 for the content, EmbeddingModel and ModelDigest for the artifact that produced the vectors, ChunkerVersion for how the text was split, BuiltAt for when, FormatVersion for the table shape, and Dimensions and ChunkCount for the vectors themselves. The first caller takes a SQLite write lock and builds a complete replacement generation in one transaction, so a failed rebuild rolls back and concurrent readers never see a half-built index. If provenance changes during a query or a build, the semantic result is refused.

Refused, not fatal — and the fallback is visible. Here is what a first semantic query looked like on a CPU-only laptop where building the index exceeded AGENT_EMBEDDING_TIMEOUT_S:

level=WARN msg="semantic retrieval unavailable, falling back to keywords" error="Embeddings unavailable at http://<IP_ADDRESS>:11434 with model \"nomic-embed-text\"; start Ollama and `ollama pull nomic-embed-text`, or unset AGENT_SEMANTIC_RETRIEVAL."

The tool still answered, with "retrieval": "keyword" in the result. The transcript itself records that the answer came from the deterministic scorer, so nobody later reads a keyword result as evidence that the embedding path works.

So when do embeddings earn their cost? Only when a reviewed retrieval set shows a better hit rate on real paraphrases without unacceptable latency or outage behavior. Retrieval quality is a separate pipeline rather than an evalset, so it stays a direct command on the evaluation CLI, running both retrievers against seed-derived incident queries through separate read-only MCP runtimes:

mise run --cd agents/go build
cd evals
set -a && . ../.env && set +a
go run ./cmd/agentops-eval retrieval

The third line loads the repository-root .env by hand, because mise’s dotenv loader stays inside mise tasks and this go run is outside one. The run writes retrieval-results.json with keyword and semantic hit-rate at 1 and 3, source and model identity, and a corpus digest — and it stores no queries and no retrieved text. Its semantic runtime needs the configured local embedding endpoint; the keyword runtime does not.

Before you quote a hit rate out of that file, check its four identity fields against the ones you have now. A run whose corpus hash, model digest, chunker version, or evalset has changed since is a Historical checkpoint (superseded): a measurement of a system that no longer exists. Rerun it rather than carrying the old figure forward; the exercise below changes the corpus in exactly that way. A hit rate is an observation about one run rather than a gate, which is the distinction 0.2. Evidence makes once for the whole course.

Whichever retriever answers, never present a nearest neighbor as truth: cosine distance always ranks something, even when every candidate is wrong.

Your turn: add a runbook and watch retrieval rank it

Add one procedure from your own domain, then watch the ranker put it first for a symptom that never names it.

• Mode: temporary experiment — the committed corpus is what other pages measure against, so this one goes back.
• Goal: add a runbook and observe it move from absent to first place for a paraphrased query.
• Files to touch: one new file under agents/data/runbooks/ only.
• Preflight: pick a slug that does not exist yet and require test ! -e agents/data/runbooks/<your-slug>.md; have the MCP server from 3.3 running.
• Steps: run the search_runbooks call from the top of this page with a query describing your symptom in a sentence, and record the three slugs it returns before you add anything. Write the runbook as Markdown with the same H2 shape as the shipped ones — Symptoms, Diagnosis, Remediation, Related — using your symptom’s own vocabulary. Run the identical query again.
• Gate that proves completion: the same query returns your slug first, and you can point at which words in it earned the ranking. Adding a queue-backlog runbook and asking “consumer lag keeps growing and the queue depth will not drain” gave these two answers, through the same projection as the command above:

{ "count": 3, "retrieval": "keyword", "slugs": ["cascade-failure", "memory-leak", "disk-full"] }

{ "count": 3, "retrieval": "keyword", "slugs": ["queue-backlog", "cascade-failure", "memory-leak"] }

Before the file existed the ranker had to answer with something, and it did — three procedures about failures that are not this one. That is the nearest-neighbor problem, from the keyword scorer rather than embeddings.

• Final state: rm -- agents/data/runbooks/<your-slug>.md, then confirm the query returns the original three slugs again. If you had semantic retrieval enabled, rebuild and remeasure rather than trusting vectors built against the previous corpus hash.

Keeping a runbook for real is a bigger commitment than deleting one: add or update the incidents that refer to its normalized slug, extend the retrieval cases with realistic queries, and run cd agents/go && go test ./memory ./policy -count=1 plus cd evals && mise run eval:validate before you rely on it.

What you can do now

• You can say why service-down wins a slug-free symptom sentence, and which words in a runbook you add earn it first place.
• You can map any fact to one of six stores by lifetime and authority, and say why an A2A context is not a durable user.
• You can read a retrieval field and know whether an answer came from embeddings or the keyword fallback, and name the provenance that would invalidate an index.
• cd agents/go && go test ./memory ./policy -count=1 passes with no model and no network, and git status --porcelain ../data/runbooks is how you prove a temporary runbook left the committed corpus untouched.

Continue to 3.5. Workflows, where the order of the investigation stops being the model’s decision.