5.2. MCP Gateway
In one glance
- You will: Ask the gateway which tools exist, call one, watch a write tool be refused, and take a read tool away from every caller by editing one rule.
- You need: The MCP server on
:8000and the host gateway from 5.1. Gateway Setup running; no model is needed on this page. - Time: about 30 minutes, hands-on.
List the tools the gateway exposes, then call one
An MCP server tells its clients what it can do, and the client believes it. That is the protocol working as designed, and the reason a compromised or merely upgraded server can hand your agent a new capability — with new description text the model reads as instructions — without anyone approving anything. An MCP tool allowlist closes that opening: a fixed list of tool names the route will serve, applied in front of a server that stays free to advertise more.
This page drives such a route by hand on :3000: a catalog smaller than the server behind it, a write tool refused as a name that does not exist, a handshake that fails closed and times out deterministically, and one read removed from every place that has to agree before it is gone.
Ask the gateway rather than the server. The handshake is three requests: initialize returns a session id in a header, an initialized notification completes it, and then you can ask for the catalog. Read it as the older shape on purpose: the MCP 2026-07-28 revision, published 28 July 2026, removed protocol-level sessions and this handshake outright, and the capture below is the pinned agentgateway v1.4.1 still answering it.
session=$(curl -sS -D - -o /dev/null http://localhost:3000/mcp \
-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}' \
| tr -d '\r' | sed -n 's/^mcp-session-id: //p')
curl -sS http://localhost:3000/mcp -H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' -H "mcp-session-id: $session" \
-d '{"jsonrpc":"2.0","method":"notifications/initialized"}' >/dev/null
curl -sS http://localhost:3000/mcp -H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' -H "mcp-session-id: $session" \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
| sed -n 's/^data: //p' | jq -r '.result.tools[].name'get_incident
get_runbook
get_service_status
list_incidents
search_runbooks
search_service_logsSix reads, alphabetical, and nothing else. Now use one of them on the same session:
curl -sS http://localhost:3000/mcp -H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' -H "mcp-session-id: $session" \
-d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_incident","arguments":{"incident_id":"INC-002"}}}' \
| sed -n 's/^data: //p' | jq -c '.result.structuredContent.incident | {id, service, severity, status, runbook}'{"id":"INC-002","service":"inventory","severity":"SEV1","status":"open","runbook":"service-down"}The INC-002 seed row, arriving through a proxy with its service-down runbook slug. That jq filter picks five fields; the real payload also carries the incident’s title, summary, and timestamps. Now reach for something the agent genuinely can do in-process — restart the inventory service — and change only the tool name:
curl -sS http://localhost:3000/mcp -H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' -H "mcp-session-id: $session" \
-d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"restart_service","arguments":{"service":"inventory"}}}'{"jsonrpc":"2.0","id":4,"error":{"code":-32602,"message":"Unknown tool: restart_service"}}The wording matters. The gateway is not saying “you may not”; it is saying the tool does not exist on this route. A caller cannot request an escalation it cannot name, and cannot learn the name by asking.
You just talked to a tool catalog smaller than the server behind it, without having to trust the server to make it so.
Why the tool allowlist is enforced in three places
The list you just read passed three independent filters, each owned by different code that fails in a different way.
The Go server registers only six reads and applies Host and Origin allowlists, so a tool that does not exist cannot be called even by something on the same machine. The gateway applies its own list as CEL rules — Common Expression Language, one boolean expression per allowed name — plus a rate limit, an authentication profile, and a failure mode. And the ADK client inside the agent filters a third time, before a tool ever joins the slice the model sees:
func NewMCPToolset(cfg MCPConfig) (tool.Toolset, error) {
transport, credentials, err := cfg.transport()
if err != nil {
return nil, err
}
built, err := mcptoolset.New(mcptoolset.Config{
Transport: transport,
Auth: credentials,
// The in-Config filter rather than tool.FilterToolset. Both work here —
// unlike the skill toolset, this one has no catalog injection for an
// external wrapper to drop — and the in-Config one runs before the tool
// is wrapped, which is the earlier of the two boundaries.
ToolFilter: tool.AllowedToolsPredicate(MCPReadToolNames()),
})
if err != nil {
return nil, fmt.Errorf("%w: %w", ErrMCP, err)
}
return built, nil
}tool.AllowedToolsPredicate(MCPReadToolNames()) runs inside the toolset, so a rejected tool’s declaration never reaches a request. The third copy earns its cost because the gateway and the client may be pointed at a different, upgraded, or compromised server tomorrow. Tool names, descriptions, and schemas reach the model as instruction-like input, so a closed list at every trust boundary means an upstream change can widen nothing and inject no new prose. The infrastructure conventions gate asserts that the configured gateway list and the executable Go list agree, so three copies never become three opinions — a gate rather than an observation, in the sense 0.2. Evidence defines.
The write tools are absent from all three lists for a different reason than the reads are present. restart_service and resolve_incident stay inside the Go application because in-process the handler can check that ADK paused for a human, read a verified identity, and commit the state change and its audit row in one transaction. A gateway route can do none of those three things, so an allowlist entry for a write would trade a guarantee for a config line. 5.5. Gateway Security shows how a verified caller reaches that audit row without ever crossing this route.
The deployed agent picks this path with one variable. AGENT_MCP_URL selects the endpoint and builds the filtered HTTP toolset; an optional AGENT_MCP_TOKEN becomes a bearer credential through ADK’s credential provider; and AGENT_TOOL_TIMEOUT_S bounds each exchange, because a read served with no deadline can hang an entire turn. Only the conversational agent takes that switch; the workflow stages and coordinator specialists keep binding local tools, because a specialist’s narrow tool set is its least-privilege boundary and no remote server should decide what it holds.
The checked-in config names localhost upstreams and the wrapper rewrites them at start time, for the container-versus-host reason 5.1. Gateway Setup takes apart. Kubernetes replaces that rewrite with service DNS such as agentops-mcp.agentops.svc.cluster.local:8000/mcp.
How the MCP route fails when its backend cannot answer
The MCP backend is configured failureMode: failClosed, which is narrower than it sounds. It decides what happens when a policy or target check cannot produce a trustworthy answer: deny, rather than forward and hope. It does not make the backend correct or available — a closed policy trades availability for confidentiality or authorization, deliberately.
An unreachable backend is this section’s failure. A 4xx is the other one, and there the status names the cause: 0.7. Troubleshooting maps 401, 403, 404, and 421 to their fixes.
See it yourself. Stop the gateway — Ctrl-C in its terminal, or the command below if you started it detached — and bring it back with its MCP upstream pointed at a port where nothing listens. Start it detached this time, because the request that follows needs your prompt back:
mise run gateway:host:stop
AGENTOPS_MCP_UPSTREAM_PORT=8009 mise run gateway:host:startThen send the initialize request again, asking curl to print the status too:
curl -sS -w '\nHTTP %{http_code}\n' http://localhost:3000/mcp \
-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"failed to send message: http upstream error: http request failed: upstream call failed: SendRequest: connection error: Connection reset by peer (os error 104)"}}
HTTP 500Notice what did not happen: the client did not receive an empty tool list. A silently smaller catalog is the dangerous failure, because an agent with no tools produces a confident paragraph from memory and nobody sees an error. Failing the handshake makes the outage loud.
Deadlines fail the same way, and the repository ships a switch that proves it without a real outage. AGENTOPS_GATEWAY_RESILIENCE_LAB=timeout renders the same read-only MCP route with a six-second delay, against the route’s existing five-second total deadline. Restart the gateway in that mode:
mise run gateway:host:stop
AGENTOPS_GATEWAY_RESILIENCE_LAB=timeout mise run gateway:host:startThen send the same initialize request, this time asking curl for the elapsed time as well:
curl -sS -w '\nHTTP %{http_code} in %{time_total}s\n' http://localhost:3000/mcp \
-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'request timeout
HTTP 504 in 5.002463sFive seconds and a 504, every time, on a route that also retries twice on 502, 503, and 504 with a 100 ms backoff. That retry policy sits behind the allowlist on purpose: only the six reads can reach it, so it can never replay a confirmed write, and the application does not retry remote MCP calls at all, because two retry loops stacked on one call is how a slow backend becomes a self-inflicted outage. Put the ordinary gateway back before continuing, with mise run gateway:host:stop followed by mise run gateway:host:start.
The route’s last bound is a token bucket: 120 requests per 60 seconds, per gateway instance, so restarting or scaling an instance changes the tokens available. That is why 5.5. Gateway Security refuses to call it a quota.
Your turn: take one read tool off the allowlist
Decide that log search is too broad for the agent’s remote route. Take it away — and find out how many places have to agree before it is really gone.
Predict before you edit: if you remove search_service_logs from the gateway rules only, what does a tools/call for it return — a permission error, an empty result, or something else?
- Mode:
temporary experiment. - Goal: prove that one removed read disappears from governed discovery, and that a gateway-only edit is an incomplete change.
- Files to touch:
infra/agentgateway/host/config.yaml, and afterwardsagents/go/compose/mcp.gofor the second half. - Preflight:
git diff --quiet -- infra/agentgateway/host/config.yaml agents/go/compose/mcp.gomust succeed, and the tool list above must show six names. - Steps: delete the
'mcp.tool.name == "search_service_logs"'line from the MCP route’smcpAuthorization.rules, restart the gateway withmise run gateway:host:stopfollowed bymise run gateway:host:start, and re-run the discovery and call requests from the top of this page. Runmise run check:infrabefore you touch any Go, and read what it says. Only then openagents/go/compose/mcp.go, removetools.SearchServiceLogsToolNamefromMCPReadToolNames()— the entry is a typed constant, not the literal string you deleted from the YAML — and runcd agents/go && go test ./compose ./mcpserver. - Gate that proves completion: the discovery request now returns five names,
get_incident, get_runbook, get_service_status, list_incidents, search_runbooks, and a call for the removed name is refused with{"jsonrpc":"2.0","id":3,"error":{"code":-32602,"message":"Unknown tool: search_service_logs"}}— the same shaperestart_serviceproduced, from a tool the server is still perfectly willing to serve.mise run check:infrathen exits non-zero on that gateway-only edit, because the shipped allowlist and the agent’s executable list no longer agree. Expect the Go suite to go red as well once you make the second edit:TestMCPAllowlistMatchesTheLocalReadSurfaceincompose/mcp_test.gocomparesMCPReadToolNames()againstLocalReadToolNames(), which still registers six local reads. - Final state: run
git restore -- infra/agentgateway/host/config.yaml agents/go/compose/mcp.go, restart the gateway, and confirmcd agents/go && go test ./compose ./mcpserver,mise run check:infra, andmise run smoke:hostare green again on the six-read contract.
Both reds are correct. A real removal edits all three lists, and the checks exist because “we removed it at the edge” is the kind of half-change that survives review.
Your turn: put a second tool server behind the same governed route
targets is plural, and every profile this course ships lists exactly one. That is a course simplification, not a gateway limit: multiplexing many tool servers behind one governed endpoint is the capability that makes an MCP gateway worth running, and it is what the 2026-07-28 MCP revision was reshaped around. Find out what merging does to discovery, and what it does not do to policy.
Predict before you start: with two targets behind one route, does tools/list return six names, twelve, or something that is neither?
- Mode:
temporary experiment— the shipped configuration is never edited; the whole experiment lives in a git-ignored lab file. - Goal: watch two catalogs merge into one governed listing, and confirm the CEL allowlist still binds across both servers rather than per server.
- Files to touch:
infra/agentgateway/lab/config.yaml, which does not exist yet. The shippedinfra/agentgateway/host/config.yamlis not touched at all. - Preflight:
test ! -e infra/agentgateway/lab/config.yaml,mise run check:infragreen, and the six-name tool list from the top of this page. - Steps: start a second copy of the course’s own MCP server on a port of your choosing —
cd agents/go && MCP_HOST=127.0.0.1 MCP_PORT=8010 MCP_TRANSPORT=streamable-http ./bin/agent mcp— noting that:8010is yours for this exercise and is not part of the stable port inventory in 0.4. Ecosystem. Copy the shipped config to the lab path, and add one entry to the MCP backend’stargetslist besideagentops-agent, nameddocs, pointing athttp://localhost:8010/mcp. Validate it, start the gateway on it, and re-run thetools/listhandshake from the top of this page. Then call a tool by its merged name, and callrestart_serviceagain. - Gate that proves completion:
tools/listreturns each target’s tools under a target-prefixed name, so the six reads appear twice under two prefixes and never collide; atools/callfor one prefixed read returns the same seed rows as before; andrestart_serviceis stillUnknown toolthrough both prefixes, becausemcpAuthorizationbinds the route rather than a target. - Final state:
mise run gateway:host:stop, stop the second server,rm -- infra/agentgateway/lab/config.yaml, restart the ordinary gateway, andmise run check:infrais green with the shipped single-target config untouched.
AGENTOPS_GATEWAY_CONFIG=infra/agentgateway/lab/config.yaml infra/scripts/gateway-host.sh validate
AGENTOPS_GATEWAY_CONFIG=infra/agentgateway/lab/config.yaml infra/scripts/gateway-host.sh startThe interesting result is the one you cannot see: nothing about the allowlist changed. A rule written once at the route governs every catalog the route merges, which is why federating a second server is a configuration edit rather than a policy review — and why a rule written per target would have been the wrong design to teach.
What you can do now
- You can drive the
initializehandshake on:3000and read a six-read catalog smaller than its server. - You can say what
restart_servicereturns through the gateway:Unknown tool, not a refusal a caller can probe. - You can predict a
500on a dead backend, a504at five seconds on a slow one, and say why an empty catalog is worse. cd agents/go && go test ./compose ./mcpserverpasses on the restored six-read contract, with no model and no gateway running.
Three places have to agree before a tool exists on this route, and you have watched a gate refuse the change when only one of them did.
Continue to 5.3. A2A Gateway when the governed tool path cannot acquire write authority.