Agentic Development

Model routing for AI coding agents

Model routing is the practice of selecting, coordinating, and switching between language models as an AI system works. Rather than sending every prompt to one expensive frontier model or one low-cost model, a router assigns work according to task difficulty, model strengths...

MMichael SwindellAugust 11, 202615 min read

AI coding agent model routing diagram showing implementation models escalating to a frontier supervisor model | Levr

Model routing is the practice of selecting, coordinating, and switching between language models as an AI system works. Rather than sending every prompt to one expensive frontier model or one low-cost model, a router assigns work according to task difficulty, model strengths, privacy needs, available context, and cost. This matters most for coding agents because software tasks change shape mid-session, moving from repository exploration to implementation, testing, and debugging. A shared agent-first control plane helps make those changing tasks visible by connecting issues, acceptance criteria, tests, and review gates to the agents doing the work. Effective routing is therefore an orchestration problem, not simply a model price comparison.

At a Glance

Model routing works best when it adapts to the work instead of making one permanent model choice.

Core idea: Use different models and agent roles for different parts of a task.
Main risk: A cheap model outside its domain can consume more tokens, tools, and time than a stronger model.
Context rule: Pass compact, useful state between agents rather than duplicating every file and trace.
Routing signal: Escalate when an agent is uncertain, loops on tools, or enters a more complex phase of work.
Engineering goal: Optimize for successful completed tasks, not the lowest token price.

What is model routing in an agentic development workflow?

Model routing is a decision layer that chooses which model should plan, implement, inspect, test, or continue a task at a given moment. It can also decide whether work should stay local, move to the cloud, run in parallel, or return to a stronger coordinating model.

A simple router sends a request to a model based on an initial classification. A more capable router treats work as a sequence of changing subtasks. This distinction matters for coding agents. A request that begins with “explain this repository” can become “implement a feature,” then “run the test suite,” and finally “debug an unexpected production-like behavior.”

Each phase can require a different combination of reasoning, tool use, context, and cost tolerance. Model routing should therefore be evaluated at the level of a completed engineering outcome, not an isolated prompt.

The need for this approach is consistent with the broader move toward AI-assisted development. The Stack Overflow Developer Survey 2024 documents widespread use of AI tools among developers alongside ongoing concerns about trust and accuracy. Routing does not solve those concerns on its own, but it gives teams a structured way to match capability and oversight to the work.

Why is choosing one model at the start of a task fragile?

Choosing a model only once is fragile because agentic tasks evolve. A model suitable for an inexpensive, familiar subtask may become ineffective when the agent must reason across unfamiliar code, use tools reliably, or investigate a failure that was not visible at the start.

Naive routing often assumes that a task category stays fixed. For example, a router may send “coding” to a lower-cost model because it performs well on a coding benchmark. But coding is not one uniform capability. Repository navigation, dependency analysis, API design, test repair, UI changes, and live debugging each impose different demands.

Model capability is also uneven. A model can outperform another model on a narrow task while being weaker on a neighboring task. Training data, post-training, tool-use behavior, and familiarity with a technical domain all affect performance.

This is why benchmark rankings are useful but incomplete. They can identify promising candidates, but they do not replace evaluation on the actual sequences of work your agents perform. The right question is not “Which model is best at coding?” It is “Which model, with which context and tools, completes this step safely and efficiently?”

How should frontier and smaller models work together?

A practical multi-model pattern keeps a stronger frontier model responsible for planning, supervision, and escalation while delegating bounded implementation or exploration tasks to smaller models. The smaller models can use more tokens or parallel effort where their lower unit cost creates a real advantage.

The frontier model does not need to perform every action itself. It can define the plan, select a delegation target, review progress, and intervene when the task changes. A lower-cost implementation model can inspect a focused module, search the repository, produce an initial patch, or run a constrained investigation.

This approach avoids two opposite mistakes:

Overusing the frontier model: Sending routine, predictable work through the most expensive path.
Overtrusting the small model: Leaving a weaker model responsible for a task after it has moved beyond its useful domain.

Parallelism can improve coverage when the work is separable. For instance, several lower-cost agents may investigate different parts of a codebase while a stronger model compares their findings and decides what to implement. Parallel work is not automatically cheaper, however. It must be bounded by a clear budget, shared task state, and a way to reconcile results.

How do you know when a smaller model is out of its depth?

A smaller model is likely out of its depth when it repeatedly calls tools without narrowing the problem, produces inconsistent plans, fails to make progress after new evidence, or encounters a task outside the domain it handles reliably. These are escalation signals, not merely errors.

Token count alone is not enough. Some valid tasks require extensive exploration. But long, unproductive traces can indicate that an agent is thrashing: revisiting the same files, issuing similar commands, or generating explanations without converging on a testable next step.

Useful escalation signals include:

Repeated tool loops: Similar searches, edits, or commands recur without reducing uncertainty.
Failed verification: Tests remain red after several materially similar attempts.
Scope expansion: The task moves from a bounded change to architectural or cross-service reasoning.
Low confidence: The model identifies missing information or conflicting evidence.
Policy boundaries: The agent needs access to sensitive data, write permissions, or a human approval gate.

Teams can use a stronger model periodically as a supervisor, especially when a cache refresh or other natural checkpoint occurs. The supervisor does not need every detail. It needs enough state to determine whether the current approach is sound, whether a different model should take over, and whether the task should be decomposed again.

Give agents shared task context, not isolated prompts.

Levr keeps issues, acceptance criteria, tests, runs, and review gates connected so coding agents can operate against the same live project state.

Explore agent-first project workflows

Why does context management determine routing cost and quality?

Context management determines routing quality because each handoff can duplicate tokens, omit critical facts, or overload the next model with irrelevant history. Good routing shares the smallest reliable representation of task state while keeping durable evidence available through files, tools, and project records.

Passing an entire agent trace to every model is expensive and often counterproductive. The receiving model may need only the plan, files examined, constraints discovered, commands run, current hypothesis, and verification status. The complete artifacts can remain accessible in the repository, terminal output, issue record, or test results.

This is context compaction. It is inherently lossy, so it should preserve the information that lets the next agent recover details when necessary. A compact handoff should answer:

What is the requested outcome?
What has been established as true?
What files, tests, or external systems matter?
What actions have already been attempted?
What remains uncertain?
What must happen before the task can be marked complete?

Cache behavior adds another constraint. Reusing an active context can be cheaper than rebuilding it for a new model. But retaining a cache should not prevent an escalation when the current agent is failing. Routing logic needs to weigh the remaining cache value against the expected gain from changing models or strategies.

Watching Routing Decisions Play Out: The Levr Dashboard

Model routing only works if you can see whether it's actually helping. The Levr Dashboard aggregates active runs, escalations, and completed tasks across models into a single pane of glass, so routing decisions are visible, not just theoretical.

Agentic Software Development | Control Plane | Manual & Automatic Verification Issue Tracking |Levr

The Levr Dashboard shows how routing decisions play out across active agent runs, in real time.

When should coding work stay local instead of going to the cloud?

Local and cloud model routing should consider privacy, latency, hardware utilization, task sensitivity, model capability, and the cost of long contexts. Local models can handle suitable work near private data, while cloud models can be reserved for tasks that require stronger general reasoning or specialized capabilities.

A hybrid model strategy can keep sensitive prompts or source material on-device when that is appropriate. It can also transform or minimize sensitive context before sending a subsequent request to a cloud-hosted model. The boundary should be defined by explicit policies, not inferred casually at runtime.

Self-hosting changes the economics. With an API provider, pricing is commonly tied to token usage and cache policies set by the provider. With self-hosted AI infrastructure decissions, teams have greater control over cache duration, model selection, and workload scheduling. They also take responsibility for capacity planning and performance management.

The NVIDIA NIM documentation describes deployment patterns for optimized inference services. Infrastructure choices such as available GPU memory, context length, and concurrent request volume can materially affect the routing decision. A locally hosted model is not automatically the least expensive option if lengthy contexts reduce throughput or cause queueing.

How can an engineering team operationalize multi-model routing?

Engineering teams can operationalize multi-model routing by defining task classes, quality gates, escalation rules, context handoff formats, and outcome-based measurements. The implementation should begin with a small number of observable routing decisions, then improve from production traces and verified results.

A useful rollout sequence looks like this:

Define work objects: Create issues with explicit acceptance criteria, dependencies, and verification requirements.
Start with two routing tiers: Use a lower-cost model for bounded work and a stronger model for planning, review, and escalation.
Set transition rules: Escalate on repeated failed tests, tool loops, changing scope, or security-sensitive actions.
Standardize handoffs: Require agents to record the current plan, evidence, changed files, commands, and open questions.
Measure completed outcomes: Track successful merges, test results, rework, latency, spend, and human intervention.
Use review gates: Keep high-impact changes behind automated verification and human approval.

One way to implement this in Levr's agentic workflow is to define the intent and acceptance criteria first, let an appropriate coding agent pick up the task, record automated verification against the same work item, then route the result through human review and approval. The project state becomes the shared reference point across models, agents, and people.

What mistakes make model routing less effective?

The most common routing mistakes are optimizing only for per-token price, treating benchmarks as production guarantees, duplicating context indiscriminately, and allowing agents to switch models without preserving task state. These failures create hidden costs in retries, lost context, and unverified changes.

Avoid these patterns:

Permanent assignment by task label: “Coding” is too broad to choose one model forever.
Blind parallelism: Multiple agents can multiply duplicate work and context cost.
Unbounded retries: A looping agent needs escalation or a new plan, not another identical attempt.
Missing verification: Cheap implementation has little value if humans must rediscover every defect later.
Prompt portability assumptions: Different models respond differently to instructions, tool schemas, and recovery prompts.
Ignoring the control plane: If task status, tests, and decisions live in separate systems, routing cannot reliably reason about project state.

Routing improves through empirical feedback. Production traces can reveal when a model was upgraded, downgraded, replaced, or corrected after the fact. Those signals are valuable training data for better routing policies, provided teams preserve privacy and treat the data as evidence rather than unquestioned truth.

Technical Deep Dive FAQ

What is a model router?

A model router is software that selects one or more language models for a request or stage of an agent workflow. It may make a one-time selection based on the prompt, or it may continuously route among models as the task changes. More advanced routers consider model cost, latency, privacy constraints, tool-use behavior, available context, and likelihood of successful completion. In multi-agent systems, the router can also choose which agent plans, executes, reviews, or escalates a task.

What is multi-model orchestration?

Multi-model orchestration is the broader system design for making several models work together toward one outcome. Routing is one part of orchestration. Other parts include task decomposition, context handoffs, tool permissions, caching, verification, retries, and human approval. A useful orchestration design assigns clear roles. For example, a frontier model can supervise and plan, while lower-cost models investigate bounded subtasks and a separate agent runs tests or reviews a proposed change.

Is the cheapest model always the best choice for simple coding tasks?

No. A lower token price does not guarantee a lower completed-task cost. A smaller model may be effective for a familiar, well-bounded task, but it can become expensive when it repeatedly calls tools, misunderstands the repository, or produces patches that require extensive rework. Measure the full cost of success, including tokens, runtime, retries, test failures, review time, and developer intervention. Start with constrained tasks and establish escalation rules before routing more consequential work to lower-cost models.

How should a router detect an out-of-distribution task?

A router can combine several practical signals: repeated tool calls, slow progress, inconsistent reasoning, low confidence, failed verification, and a shift in task scope. It can also use periodic review by a stronger model to inspect whether the current agent is following a productive plan. No single signal is definitive. A long trace may be necessary for complex work, while a short trace can still be wrong. The goal is to detect loss of productive progress early enough to change strategy.

What context should pass from an implementation agent to a supervising model?

Pass a compact operational summary rather than the full raw trace. Include the requested outcome, the current plan, files and systems examined, important findings, changes made, commands or tests run, results, unresolved questions, and the recommended next action. Keep full artifacts retrievable through durable sources such as the repository, issue history, CI/CD output, and test records. This gives the supervising model enough state to decide while preserving a path back to details when it needs them.

Does context compaction reduce costs?

Context compaction can reduce the amount of information sent to a new model, but it is not a universal cost-saving technique. Compacting may require rebuilding context and losing cache reuse, which can increase input cost. It is often most valuable for maintaining model quality when a context becomes too long or noisy. The better framing is to use compaction when the next agent needs a smaller, clearer working state, then compare that benefit against the cost of a cache miss and the risk of lost information.

Should the large model or the small model act as the orchestrator?

There is no universal answer. A stronger model may make better delegation and recovery decisions, particularly for ambiguous research or complex engineering work. A smaller model may reduce coordination overhead for predictable workflows. Cache economics can also alter the result because an already-warm model has useful task context. Test both patterns on representative workloads. Evaluate successful completion, cost, latency, and the rate of unnecessary escalations rather than assuming that either orchestration role is always optimal.

How do local models fit into model routing?

Local models can serve privacy-sensitive, low-latency, or hardware-utilization goals. A router can keep eligible requests local, use a cloud model for more difficult reasoning, or minimize and sanitize context before a cloud handoff. The right design depends on the model available locally, GPU capacity, context length, concurrency, and policy requirements. Teams should also account for long-context throughput. A self-hosted model may become slower or more costly in practice when the workload keeps large contexts active.

How do you evaluate a model routing system for coding agents?

Evaluate routing with realistic, multi-step engineering tasks rather than single-prompt benchmarks alone. Include repository exploration, feature implementation, testing, debugging, and review. Track task completion rate, test pass rate, retry count, tool-call behavior, wall-clock time, total spend, human corrections, and regressions after merge. Keep a record of routing decisions and outcomes so you can compare alternatives. A router is successful when it improves verified delivery outcomes at an acceptable cost, not when it merely lowers average token price.

Can coding agents use different models on the same project safely?

Yes, if they operate against shared task definitions, controlled tool permissions, durable context, and consistent verification gates. The main risk is not model diversity itself. It is fragmented state. One agent may change code while another works from an outdated plan or test result. Use a shared project record for issues, acceptance criteria, branches, test runs, and approvals. Attribute actions to the agent or person that performed them, and require verification before work moves into a completed state.

Key Takeaways

Model routing is becoming a core capability for teams that use coding agents across real software delivery workflows.

The strongest systems combine model specialization with shared context, verification, and escalation.

Route by phase: Reassess model choice as a task moves from planning to implementation and debugging.
Keep supervision: A capable coordinating model can catch when delegated work is no longer on track.
Preserve evidence: Compact handoffs work best when files, tests, and task history remain accessible.
Measure outcomes: Optimize for verified task completion, not raw token cost.
Build control: Shared project state gives humans and coding agents one source of truth.

The future of model routing is likely to be distributed across models, agent harnesses, infrastructure, and the project systems that coordinate all of them.