Skip to content

5.4. Model Gateway

In one glance

  • You will: Route the agent through a governed endpoint and explain where the upstream credential lives.
  • You need: Part I completed, container knowledge, and the Chapter 5 host gateway prerequisites.
  • Time: about 25 minutes, hands-on.

How do you start the Gemini gateway?

The main host gateway profile forwards model requests to Gemini using the provider key in your root .env.

In the repository root:

mise run gateway:host

This starts the loopback-only gateway container. It does not itself send a model prompt. Keep the terminal open; Chapter 5's MCP and A2A processes provide the other upstreams.

The task selects infra/agentgateway/host/config-gemini.yaml. The wrapper mounts only the Gemini key into the container, using its private runtime directory; it does not mount the whole .env. Provider authentication and caller authentication are separate.

How do you point the agent at the gateway?

Change the provider block in .env to use the gateway's OpenAI-compatible listener.

AGENT_MODEL_PROVIDER=openai-compatible
AGENT_MODEL=gemini-3.5-flash
OPENAI_BASE_URL=http://127.0.0.1:4000/v1
OPENAI_API_KEY=local-agentgateway

Keep GOOGLE_API_KEY for the host gateway task. The non-secret OpenAI marker satisfies the local SDK; it is not a Gemini key. The host's shared dotenv is a teaching convenience. The Kubernetes profile gives the upstream credential only to the gateway pod.

mise run config:check
cd agents/python
mise run web

Gateway calls still consume hosted model quota

Sending a message now reaches Gemini through agentgateway. The gateway does not make inference free. Repeated evaluations, retries, and load tests can be billed.

What must you verify after changing the transport?

Verify the same behavior you measured before the transition.

Ask List the open incidents and inspect tool events. The answer must contain INC-002, INC-005, and INC-010, with no invented incident. Then use the reference's critical evaluation cases to inspect tool results and approval behavior.

The gateway owns routing, rate policy, and its prompt guard. The application keeps domain authorization and approval checks. A prompt guard is an extra boundary, not a replacement for those controls.

The default model remains pinned to the platform's compatibility pair. Do not replace it without testing the synthetic tool-result turn and the reference evaluation dataset.

How do you use the optional Ollama profile?

The original account-free profile remains available explicitly.

Stop the current host gateway, then start the optional profile:

mise run gateway:host:stop
mise run gateway:host:ollama

Set AGENT_MODEL=qwen3:4b-instruct and keep the OpenAI-compatible gateway URL. Ollama must already be serving the downloaded model. The canonical infra/agentgateway/host/config.yaml remains the Ollama configuration used by deterministic platform smoke tests.

mise run smoke:host uses the optional local/fake path. It is not proof of a Gemini-backed turn. The secured host profile remains a separate local authentication/TLS exercise; it does not turn a loopback lab into a public production service.

What proves this page worked?

Record the model, transport, and result of your chosen bounded turn.

You are done when:

  • The application sends model requests to :4000 and the gateway owns the upstream route.
  • You distinguish the caller marker from the real provider key.
  • Your observed result is grounded in the same seed as the direct-model run.
  • You stopped the agent UI and gateway container when finished.

Continue to 5.5. Gateway Security to inspect caller authentication and TLS.