Models Are Components

Model quality is not a single ladder. A task may depend on instruction following, long-context comprehension, tool selection, structured output, visual understanding, mathematical accuracy, coding, tone, latency, privacy, or cost. A model strong at open-ended reasoning may be wasteful for extraction and inconsistent for a tightly formatted operation.

Define the job profile before comparing providers. Include the hardest judgment, context size, tool behavior, output contract, risk, speed target, and volume. Choose from evidence on that profile rather than general reputation.

The same model and instruction can produce different paths across runs. Temperature and reasoning settings influence behavior, but variance also comes from retrieved context, tool responses, conversation state, provider changes, and ambiguity in the task. A single successful demonstration is therefore weak evidence.

Test repeated runs, boundary cases, and changes over time. Use deterministic validation around identifiers, calculations, permissions, schemas, and state transitions. Reserve model judgment for the part that benefits from interpretation.

Models can invent facts, lose instructions in long contexts, mishandle exact arithmetic, call the wrong tool, overfit examples, and produce convincing explanations for incorrect actions. Larger models reduce some errors but do not remove the need for architecture.

Use retrieval for current facts, calculators and code for exact computation, schemas for output, policy checks outside the model, approval for consequential action, and verification against the environment. When the system needs a guarantee, encode it in deterministic software rather than a sentence asking the model to be certain.

A multi-model system can route simple classification to a fast inexpensive model, reserve frontier reasoning for ambiguous planning, use a specialist for coding or vision, and invoke a second model as a critic. This helps only when the route has measurable value and a stable interface.

Avoid provider churn disguised as architecture. Every model change can alter tool use, formatting, latency, refusal behavior, and prompt interpretation. Keep model-specific adapters behind a common task and receipt contract, then compare candidates on the same evaluation set.

The economics of an agent include input and output tokens, cached context, tool calls, retries, parallel workers, verification, and human review. The user experiences latency across the whole loop, not the model response alone. A cheaper call that doubles retries or review time is not cheaper.

Measure cost and duration by job class and outcome. Stream when early visibility helps. Parallelize only independent work. Cache stable context when policy allows. Use budgets and stopping conditions. Optimize after correctness is measured, because a fast wrong system creates downstream work that rarely appears on the inference bill.

A model evaluation must name the prompt version, context policy, tools, harness, decoding settings, and grader. Otherwise the result cannot explain which change improved or harmed performance. Compare outcome quality, trajectory, safety, latency, cost, and recovery—not one aggregate score.

Run shadow tests before replacing a production route. Examine disagreements by failure category. A smaller model may graduate for a narrow task even when it performs worse overall; a frontier model may remain necessary where rare reasoning failures carry high consequence.