2.2. Models

Call the model directly and read its function-call proposal

The model in this agent is not a brand name or a config string. It is a value your code constructs, and it speaks HTTP: one request carrying the question, the history, and every tool the agent declares; one reply carrying text, or a proposal to call one of those tools.

Every “the AI decided to” sentence reduces to that exchange. So this page sends one request by hand, then names the two methods ADK requires of a model and the four settings that pick which one answers.

In 2.1 the proposal arrived behind the instruction, the tool layer, and ADK. Predict what happens without them: one question, one tool declaration, no agent and no session, straight to the process listening on port 11434. Prose, or a request for the tool?

curl --fail-with-body http://127.0.0.1:11434/v1/responses \
  -H 'Authorization: Bearer local-ollama' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3:4b-instruct",
    "input": "What is the status of checkout?",
    "tools": [{
      "type": "function",
      "name": "get_service_status",
      "description": "Return one service status",
      "parameters": {
        "type": "object",
        "properties": {"name": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": false
      }
    }]
  }'

Here is the reply. It arrived with thirty-one top-level fields and three of them are shown, each one whole, nested ids and sub-fields included. The twenty-eight left out are the echoed tool list, every sampling setting, the response id, the timestamps, and a long tail of nulls.

{
  "status": "completed",
  "output": [
    {
      "id": "fc_resp_329409_0",
      "type": "function_call",
      "status": "completed",
      "call_id": "be1ATCAFE9C8w5JhEnIdTSUbM1Tlg3vz",
      "name": "get_service_status",
      "arguments": "{\"name\":\"checkout\"}"
    }
  ],
  "usage": {
    "input_tokens": 152,
    "output_tokens": 21,
    "total_tokens": 173,
    "input_tokens_details": { "cached_tokens": 0 },
    "output_tokens_details": { "reasoning_tokens": 0 }
  }
}

No prose: nothing in that exchange but a question and a tool declaration, and the model asked for the tool.

Three details matter later. The route is /v1/responses, the OpenAI Responses API rather than chat completions, because ADK’s adapter speaks only that surface and the Chapter 5 gateway route matches it; an offline test in agents/go/model/openai_test.go pins that path, so a silent route change fails in CI rather than during an incident.

The arguments field is a string containing JSON, written by a model guessing at your schema, so nothing downstream trusts it: ADK decodes the shape, then the agent parses identifiers and validates domain rules before any read or write runs.

A token is the unit a model reads and bills in, a few characters of text. The usage block is where every token count in this course comes from: the policy plugin’s budget guard and the cost chapters both meter these numbers. One sentence plus one tool declaration cost 152 input tokens, which calibrates what a full instruction and the agent’s entire tool list cost on every single turn.

Every agent, gateway, and evaluation later in this course is machinery around the exchange you just sent by hand.

Why tokens, context length, and parameters bound every request

A language model predicts the next token from the context supplied for one request, and two properties follow. Generation is probabilistic: the same request can produce different valid-looking output. It is also stateless: the weights do not remember the previous turn, so the session service rebuilds the relevant history into every request.

The context window is the token budget shared by the instruction, the history, the tool schemas, the tool results, and the answer being generated. More context means more latency, and on priced providers more money, which is why the policy plugin carries a budget guard and a compaction callback that bound the envelope before a request is sent.

Parameters are the learned numbers in the model artifact. They are not current facts about your platform, and they are not memory. That is why get_service_status exists: what is true right now comes from tool reads and explicit stores, never from weights. Notice how thin the model’s contribution was above: it picked a tool and filled in one string.

An instruction-tuned model has been trained to follow conversational instructions and emit tool-call conventions; a base model only continues token sequences and is not a supported agent boundary here without further tuning and measurement. And open weights is not open source: the default Qwen weights are Apache-2.0 licensed and runnable locally, but the training data and training code needed to reproduce them are not published.

How ADK’s two-method model interface keeps providers swappable

That model runs on your laptop, where inference is slow and free. Moving the same agent onto a GPU host behind a governed gateway is a refactor in most agent codebases, because the provider SDK is stitched through the tool layer. Here it is a URL, because ADK’s portability seam is two methods wide:

// simplified: the installed ADK interface also carries package documentation.
type LLM interface {
	Name() string
	GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]
}

Name identifies the implementation; GenerateContent takes a typed request and yields responses or errors, the last of which carries the usage metadata you just read off the wire. The snippet omits only doc comments: those two methods are the whole interface. The agent composes against it and never against a vendor client, which is why a test can swap in a scripted model while ADK, the plugins, the tools, the sessions, and the state stay real.

ADK takes a model instance, not a model name: nothing in the framework turns the string qwen3:4b-instruct into a client. The factory does that, once, at startup:

func Build(ctx context.Context, cfg config.Config) (adkmodel.LLM, error) {
	build, err := provider(ctx, cfg)
	if err != nil {
		return nil, err
	}
	primary, err := build(cfg.Model)
	if err != nil {
		return nil, err
	}
	if cfg.ModelFallback == nil {
		// No fallback configured: hand back the provider model untouched, so the
		// single-model path carries none of the wrapper's behavior.
		return primary, nil
	}
	secondary, err := build(*cfg.ModelFallback)
	if err != nil {
		return nil, fmt.Errorf("%s: %w", config.EnvModelFallback, err)
	}
	return NewFallback(primary, secondary), nil
}

Set AGENT_MODEL_FALLBACK and Build wraps the primary so a dead primary fails over to a second model. Both come out of the same factory and one validated set of credentials, so they share a provider, an endpoint, a deadline, and a connection pool. A failover therefore reaches a second model behind an endpoint you already approved: it is not multi-provider failover, and not a way to smuggle a hosted model into an account-free run.

The four settings that choose the model and its endpoint

Every field a run needs is parsed and cross-checked once, at startup, in this block. Its first line, AGENT_ENTRYPOINT, chooses which composition gets built and belongs to 2.5. Dev Loop; the rest is the model.

Entrypoint    Entrypoint    `env:"AGENT_ENTRYPOINT"     envDefault:"agent"`
ModelProvider ModelProvider `env:"AGENT_MODEL_PROVIDER" envDefault:"openai-compatible"`
Model         string        `env:"AGENT_MODEL"          envDefault:"qwen3:4b-instruct"`

// "openai-compatible" describes the ADK client contract, not the deployment
// topology. Point this URL directly at Ollama for the account-free first
// run, or at agentgateway when the governed data plane is introduced.
OpenAIBaseURL string `env:"OPENAI_BASE_URL" envDefault:"http://127.0.0.1:11434/v1"`
OpenAIAPIKey  Secret `env:"OPENAI_API_KEY"  envDefault:"local-ollama"`

// Optional Gemini paths: an AI Studio API key, or Enterprise/Vertex via ADC
// with an explicit project and location.
GoogleAPIKey        Secret `env:"GOOGLE_API_KEY"`
GoogleCloudProject  string `env:"GOOGLE_CLOUD_PROJECT"`
GoogleCloudLocation string `env:"GOOGLE_CLOUD_LOCATION"`

The stable surface is four of them:

The move to a governed endpoint is the third row and nothing else: http://127.0.0.1:11434/v1 becomes http://127.0.0.1:4000/v1, where agentgateway — the proxy Chapter 5 puts in front of the model to route and record every request — is listening. The composition, the tools, the instruction, and the tests are untouched. That is what explicit construction buys: endpoint, credential, deadline, and sampling arrive from validated configuration, so no run depends on what your shell exported.

Native Gemini is optional: an AI Studio API key, or enterprise mode through Application Default Credentials with an explicit project and location. Both need an account and can cost money, so they are comparisons rather than prerequisites.

The wire contract and the factory both check out offline, in under a second, with no endpoint running:

cd agents/go
go test ./model -run 'TestOpenAICompatible|TestBuild' -count=1

ok  	github.com/MLOps-Courses/agentops-open-course/agents/go/model	0.081s

Your turn: hold everything still and change only the temperature

Comparing two models honestly means holding everything else still: same code, same dataset, same prompt, same decoding settings, same transport. Then change one thing and run more than once, because a single sample of a probabilistic system is an anecdote.

Predict first: AGENT_MODEL_TEMPERATURE=0 asks for greedy decoding. Will two runs of the same question then be word-for-word identical?

• Mode: inspect — the variable goes on the command line, the agent only reads, and no file on disk changes.
• Goal: isolate one variable across three runs, and see how much of a turn’s wording is the model’s freedom and how much is the rows underneath.
• Files to touch: none. In particular, do not put the variable into .env, and do not run data:reset — it is rm -rf .state and would throw away the conversation you built in 2.1 for no benefit here.
• Preflight: mise run doctor:model reports the model ready.
• Steps: from agents/go, run AGENT_MODEL_TEMPERATURE=0 mise run run and ask List the three open incidents.. Exit with Ctrl-D, launch the identical command again, and ask the identical question. Then do it a third time with the variable left off entirely. Each console launch opens a fresh session, so the three runs do not contaminate each other.
• Gate that proves completion: you can say which parts of the two temperature-zero answers matched exactly and which did not, and whether the incident identifiers changed across any of the three runs.
• Final state: no tracked file changed. Your .state directory has gained three short sessions and nothing else.

Whatever your runs show, hold on to the shape of the result: temperature zero narrows the distribution, it does not make a run reproducible, and the identifiers should be stable for a reason unrelated to sampling — they came from a tool. Scaled up, that is how you compare candidate models: run the same evalset repeatedly and report pass rate, variance, tokens, model calls, latency, and failures rather than one transcript. 0.2. Evidence is blunt about how little a single green run tells you.

Your turn: move the pinned tag to a newer generation for one turn

Qwen3.5 shipped after the tag this course pins, and qwen3.5:4b is its closest replacement by size. Every capture in this course was produced against qwen3:4b-instruct, which is why the tag is a default in agents/go/config/config.go rather than a suggestion — and why moving it is an experiment rather than an upgrade.

Predict first: you are about to change that default and then print the resolved configuration. If your .env also names AGENT_MODEL, which of the two values does mise run config:check report?

• Mode: temporary experiment — one tracked file changes and goes back.
• Goal: find out what holds the model pin in place, and see one answer cross a model generation.
• Files to touch: agents/go/config/config.go only.
• Preflight: git diff --quiet -- agents/go/config/config.go succeeds, cd agents/go && go test ./config -count=1 is green, and ollama pull qwen3.5:4b has finished — a 3.4 GB download, so start it before you read on.
• Steps: in config.go, change the AGENT_MODEL field’s envDefault:"qwen3:4b-instruct" to envDefault:"qwen3.5:4b" — the AGENT_PII_MODEL field carries the same string and is not the one you want. Run cd agents/go && go test ./config -count=1 and read what it says, then mise run config:check and read which model it reports. Finally run AGENT_MODEL=qwen3.5:4b mise run run, which outranks both the default and any .env, ask List the three open incidents., and exit with Ctrl-D.
• Gate that proves completion: you can quote the assertion the config package objected to, say which value config:check reported and what outranked what to produce it, and say whether the newer tag answered with the same three incident identifiers through the same tool call. An answer in plain prose with no tool call is a result about the tag, not a broken setup.
• Final state: git restore -- agents/go/config/config.go, then cd agents/go && go test ./config -count=1 green again and mise run config:check reporting AGENT_MODEL = qwen3:4b-instruct. ollama rm qwen3.5:4b returns the 3.4 GB.

A newer generation is a change to the thing being measured, so it enters through a run you can compare, never through a default nobody re-scored. That is the same argument as the temperature exercise above, at the scale where it costs money.

What you can do now

• You can predict a bare /v1/responses reply: a function_call, not prose, and a usage bill.
• You can name the two methods ADK requires of a model, and why portability lives there.
• You can name the one setting that moves the agent from direct Ollama to the gateway, with no code change.
• You can say what temperature zero narrows, why the identifiers held steady, and what outranks the envDefault pin.

What the model does with a question is largely decided by the prose the runtime packs into each request.

Continue to 2.3. Instructions, which reads that prose and shows which of its sentences a test can pin.