7.3b. Cost Governance

Why a per-session budget cannot bound a shared endpoint

A shared model endpoint is one route that many agents point at, because that is where the credentials, guardrails, and audit log already live. Bounding what it spends takes a control that sees every caller at once: a token bucket at the gateway, an allowance denominated in tokens that refills on a fixed interval and answers HTTP 429 once it is drained. Same algorithm as the rate limits in 5.5. Gateway Security, different unit: those buckets hold requests, this one holds model tokens, and the word “token” means something different on each side.

Nothing you built in 7.3. Costs can play that role. AGENT_MAX_TOKENS_PER_SESSION lives inside one application, counts one conversation, and resets when a client opens a new session — which a batch job does on every single request. It is the right control for a runaway conversation and the wrong one for a runaway team.

The failure it cannot catch is quiet: several teams share the endpoint, one ships a retrieval change that pastes whole runbooks into every prompt, their agent still answers, their dashboards stay green, and the first signal is an invoice. You will shrink the gateway’s token bucket until it refuses a call, read the tokens and dollars it meters per route, route budget alerts to a daily digest instead of a pager, and name which of three enforcement layers can actually stop money.

Cap model spend with a token bucket, not a request cap

The model route you built in 5.4. Model Gateway has always carried a cap, and 7.3. Costs named its size: thirty requests per sixty seconds. That cap does nothing to a greedy caller, because it prices a six-word question and a 200,000-token prompt identically. A caller that sends the same number of calls as everyone else, each one enormous, never trips it. So the route carries a second bucket that counts the thing the bill is actually made of:

# Two buckets on one route, both enforced on every call.
localRateLimit:
  # Requests. This one prices every call alike, so it cannot tell a
  # six-word question from a caller that pastes 200k tokens of runbook
  # into each prompt.
  - maxTokens: 30
    tokensPerFill: 30
    fillInterval: 60s
  # Tokens. agentgateway charges this bucket after the call completes,
  # so an oversized call is admitted and drains it; the *next* call is
  # the one refused with 429. 100000/min restates the request cap in
  # tokens — 30 calls at the ~2,700 tokens a course turn's first model
  # call actually spends is ~81,000 — so it binds on context growth
  # rather than on ordinary use.
  - maxTokens: 100000
    tokensPerFill: 100000
    fillInterval: 60s
    type: tokens

Both buckets apply to every call, and either one can refuse it. To watch the token bucket bite, shrink it far below any real call: open infra/agentgateway/host/config.yaml and change the second bucket to maxTokens: 10, tokensPerFill: 10, fillInterval: 300s. The five-minute interval is deliberate, because a local model can take minutes per call on a busy machine and a sixty-second bucket would refill between your two commands and hand you a false pass. Then start the gateway and ask it for one word, twice:

mise run gateway:host:start
curl -sS -i -X POST http://localhost:4000/v1/responses \
  -H 'content-type: application/json' \
  -d '{"model":"qwen3:4b-instruct","input":"Reply with the single word OK."}'

The first call is served, and its own usage block says it spent 72 tokens against a bucket that holds 10:

HTTP/1.1 200 OK
...
"usage":{"input_tokens":14,"output_tokens":58,"total_tokens":72}

Run the identical command again and the gateway never reaches the model:

HTTP/1.1 429 Too Many Requests
x-ratelimit-limit: 10
x-ratelimit-remaining: 0
x-ratelimit-reset: 52

rate limit exceeded

The order of events is the whole shape of token-based limiting. A call’s token cost is not known until the call finishes, so the gateway admits the request, waits for the response, and charges the bucket afterwards. The call that busts the budget is never the call that gets refused. The next one is. The application’s own budget overshoots the same way for the same unavoidable reason, which makes both caps ceilings rather than tripwires.

The headers repeat that lesson. x-ratelimit-limit: 10 names the bucket that refused you, so on a two-bucket route it tells you which cap you hit. x-ratelimit-reset counts down a refill window, but not necessarily the window of the bucket you emptied: this refusal came from a 300-second bucket and reported 52 seconds, which belongs to the sixty-second request bucket beside it. Trust the limit; verify the reset.

Now restore the config with git restore -- infra/agentgateway/host/config.yaml and restart the gateway. The shipped value is 100,000 tokens per minute, which is the request cap restated in the unit that matters: thirty calls at the roughly 2,700 tokens a course turn’s first model call actually spends is about 81,000. On your laptop it will never bind, which is what a ceiling is for: one caller cannot quietly multiply its context by fifty.

You just watched a gateway refuse to spend money it had not been given, on a rule an operator wrote rather than a limit a provider imposed.

Three enforcement layers, and only one can stop the money

The gateway also meters what it lets through. Its metrics endpoint carries a token counter and a dollar counter per route, the fleet-scale counterpart of the per-session numbers you read off a span in 7.3. Costs. Prometheus scrapes this endpoint as the agentgateway job, so these counters are queryable; read them raw first:

curl -fsS http://localhost:15020/metrics \
  | grep -E 'gen_ai_client_(token_usage_sum|cost_usd_total)|cost_catalog_lookups'

agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input",listener="llm"} 14.0
agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="output",listener="llm"} 58.0
agentgateway_gen_ai_client_token_usage_sum{gen_ai_token_type="input_cache_read",listener="llm"} 0.0
agentgateway_gen_ai_client_cost_usd_total{listener="llm"} 0.0
agentgateway_cost_catalog_lookups_total{status="Exact",listener="llm"} 1

The listener label names the gateway that served the call: here the llm entry under gateways: on port 4000, which keeps model spend separate from MCP and A2A traffic on the same proxy.

That zero is not a missing feature. The gateway prices each call from config.modelCatalog, whose rates are quoted per million tokens rather than per thousand, and the course catalog sets self-hosted Qwen to exactly "0" — so the dollar counter is a measured zero rather than an estimate. The application prices per thousand (AGENT_INPUT_PRICE_PER_1K), so the same published rate enters the two configs a thousandfold apart, and a rate pasted into the wrong one still looks plausible.

status="Exact" is the line that makes the dollar figure trustworthy: the catalog was consulted and it knew this model. The same label reads NoCatalog when no rates were configured at all, which is what the GKE profile does deliberately rather than invent a Vertex price. Missing means a catalog existed and did not contain the model that was served, so its tokens are counted and its cost is silently reported as nothing. A cost dashboard reading zero because nobody priced the model looks exactly like a cost dashboard reading zero because nothing was spent.

Three layers can refuse a model call, and they are not interchangeable:

Only the third layer can truly stop the spending, and it is the only one this repository does not ship: it lives in a provider’s console behind an account. The first two are how you avoid needing it.

Know the edges of the gateway layer before you rely on it. Its buckets are local: agentgateway’s own schema calls them “per-proxy, without coordination between instances of the proxy”, so running a second replica doubles the fleet’s allowance. A budget that survives scaling needs remoteRateLimit and an external rate-limit service: agentgateway speaks the Envoy ratelimit protocol, and its descriptors accept type: tokens with a cost expression in CEL, the small expression language a gateway evaluates against a request to select policy. This course runs no such service, so that path is described here and proved nowhere. A token bucket also has no calendar: it refills on an interval and cannot express “four hundred dollars this month”. That number has to be watched rather than enforced, which the next section does.

The alert that fires, and the one that cannot fire here

Three shipped rules read those gateway counters. AgentModelBudgetRefusals fires when the model route returns 429 from either bucket — either a caller is spending far beyond its share and the cap worked, or the cap is smaller than one honest turn and every turn is now failing. AgentModelUnpriced fires on status="Missing", because a budget alert built on an under-reported counter can never fire at all. AgentModelSpendBudgetBurn watches the burn rate against a stated monthly budget, in exactly the shape the SLO rule uses for errors:

sum(rate(agentgateway_gen_ai_client_cost_usd_total[1h])) * 3600 > (4 * (50 / 730))

The 50 is a monthly budget in dollars, a placeholder for your real number, and 730 is the hours in an average month, so 50 / 730 is the even burn that spends that budget exactly on the last day; four times that empties the month in about a week. With the shipped zero rates it evaluates to 0 > 0.274 forever, so it cannot fire on the local path and never will. It starts working the hour someone points the route at a priced provider and puts that provider’s rates in the catalog. Until then its behavior is proved by promtool test rules against a synthetic series rather than by anything you can watch, which is the distinction 0.2. Evidence exists to keep visible.

Where those alerts go matters as much as when they fire. All three carry category: cost, and Alertmanager routes that label off the default path:

routes:
  - matchers:
      - category = "cost"
    receiver: cost-budget
    group_by: [alertname]
    group_wait: 5m
    group_interval: 30m
    repeat_interval: 24h

Money is not an emergency. A budget breach is a decision somebody makes during working hours with the month’s numbers in front of them, and each key sets part of that pace. group_by: [alertname] puts every firing instance of one rule in a single notification. group_wait: 5m holds that first notification while related alerts arrive, against thirty seconds on the default path. group_interval: 30m is the shortest gap before a group that has already notified sends an update. repeat_interval: 24h leaves an unresolved alert quiet for a day instead of repeating six times a night. Routing is where an alert’s urgency is really expressed; the severity label only describes it.

Compose binds alertmanager.yml into the container by inode, so a rewritten file needs docker restart agentops-observability-alertmanager-1 before Alertmanager can see it, and until that restart the check below still answers local-webhook. Check the tree rather than trusting it:

docker exec agentops-observability-alertmanager-1 amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  category=cost alertname=AgentModelSpendBudgetBurn severity=ticket

cost-budget

Ask the same question with alertname=AgentErrorBudgetBurn severity=page and it answers local-webhook: two alerts from the same Prometheus, sorted by a label, arriving in two different places.

The same counters build the spend panel. Add one to your Grafana dashboard — the spend per model per hour, sum by (gen_ai_request_model) (rate(agentgateway_gen_ai_client_cost_usd_total[1h])) * 3600, next to the tokens behind it, sum by (gen_ai_request_model, gen_ai_token_type) (increase(agentgateway_gen_ai_client_token_usage_sum[1h])). Both return series on a lab that has served a single call. One caveat: a counter that appears at a non-zero value shows no increase until its next scrape, so the very first refusal or the very first call of a fresh gateway is invisible to increase().

Your turn: give one caller its own token budget

Splitting the route’s single bucket per caller turns a fleet-wide ceiling into attribution. The metrics say who is spending, and the refusal lands only on them.

One constraint decides the shape of the config: the schema accepts a flat list of buckets or a conditional set, never both, so per-caller budgets on this route replace the flat thirty-requests-a-minute cap rather than joining it.

Predict before you start: agentgateway can select a rate-limit policy with a CEL condition over the request. If you write one bucket for a named client and one fallback for everyone else, how many buckets does a single call get charged to?

• Mode: temporary experiment.
• Goal: prove that a per-caller token budget refuses one client while its neighbour keeps working, and that the gateway can attribute tokens and dollars to each of them.
• Files to touch: infra/agentgateway/host/config.yaml only.
• Preflight: git diff --quiet -- infra/agentgateway/host/config.yaml must exit zero, and docker ps --all --quiet --filter name=agentops-host-gateway must print nothing, so you start from the shipped config and no stale container.
• Steps: replace the llm route’s localRateLimit list with the conditional form — one entry whose condition is request.headers["x-agentops-client"] == "batch-summarizer" carrying maxTokens: 40, tokensPerFill: 40, fillInterval: 300s, type: tokens, and one final entry with no condition carrying the same fields at maxTokens: 200. Then add metrics: {fields: {add: {agentops_client: 'request.headers["x-agentops-client"]'}}} under the top-level config: key. Validate with agentgateway --validate-only -f infra/agentgateway/host/config.yaml before you start anything, bring the gateway up with mise run gateway:host:start, and send the same short prompt repeatedly under each header value.
• Gate that proves completion: batch-summarizer receives its first HTTP 429 strictly before incident-triage receives any, and curl -fsS http://localhost:15020/metrics | grep gen_ai_client_cost_usd_total shows one series per agentops_client value rather than one series for the route.
• Final state: mise run gateway:host:stop, then git restore -- infra/agentgateway/host/config.yaml, and git status --short reports nothing for that path.

The answer is one, because a conditional policy picks the first matching entry and stops. Per-caller accounting belongs in a rate-limit service that can hold more than one dimension; a local bucket was only ever meant to be a ceiling.

What you can do now

• You can cap model spend on a shared endpoint in tokens rather than requests, and you have watched the gateway refuse a call against that cap.
• You can read tokens and dollars per route out of the gateway, and say why a Missing catalog lookup makes a cost dashboard lie.
• You can name which of the three enforcement layers stops a runaway conversation, a runaway team, and the money itself.
• You routed a budget alert to a daily digest instead of a pager, and verified the routing with amtool rather than by waiting for one to fire.

No single layer owns agent cost: the application knows what a conversation is and cannot see the fleet, the gateway sees the fleet and does not know what a conversation is, and only the invoice is authoritative about either. You now have a place to stand in each layer, and can say which of your numbers is measured, which is priced from a catalog somebody maintains, and which is still just the bill.

Continue to 7.4. Feedback, which turns a vague complaint about answer quality into a reproducible evaluation record — the judgement no cost meter can make.