AI agents are changing how companies choose which AI model does the work

With AI agents handling more complex workflows, companies are looking beyond a single model and building systems that route each task

For businesses, picking an AI model to work with used to involve weighing benchmark scores, pricing, context window, maybe latency, then committing and building around it. That calculus is breaking down as AI applications stop behaving like chatbots and start behaving like teams.

 

Take an AI-powered coding system. It has to read a repository, dig through documentation, write code, run tests, and then respond when something breaks, often looping back to revise its own work. In such a multi-agent system, these tasks can be split across several agents, each with different demands at different points in the workflow.

 

A hard reasoning problem might call for a frontier model. A routine tool call may not need one. A smaller model could handle it just fine. This is where model routing comes in.

 

 

NVIDIA recently released NeMo Switchyard, an open-source, Rust-based proxy and library built to route LLM traffic across models and providers, translating between OpenAI and Anthropic API formats along the way. That means an application can swap backends without a rewrite every time.

 

NVIDIA frames this around what it calls a "system of models", an architecture where different models take on different jobs inside one agentic workflow. A frontier reasoning model could handle the planning, while a smaller model takes care of code review, tool use, or other narrow tasks.

 

This changes the question from which model is the best for an application, to which is the most suitable for each task.

 

The model can change during the task

 

What makes NeMo Switchyard worth paying attention to is that routing doesn't have to happen just once, at the start.

 

A conventional application fires every request at the same model. A router, instead, picks between several, and the pick can depend on the request itself, or on what already happened earlier in the workflow.

 

Switchyard supports routing that reacts to signals from the agent's own execution, tool results, errors, as well as routing tied to specific stages of a workflow.

 

Picture a coding agent working through a bug. Understanding the repository first might need a stronger model. Once there's a plan, a smaller model can take over the actual code changes. Then the agent runs tests.

 

If they pass, there's no reason to call the stronger model back in. If they fail with a messy error, the system can escalate.

 

The signal that matters is what happened during the task, not just what the task looked like going in.

 

That's especially useful for agents, because their work throws off a constant stream of information: tool outputs, execution errors, search results, context built up from earlier model calls. A routing system can read some of that and decide what comes next.

 

In effect, the router becomes part of the runtime.

 

Why this matters more with multiple agents

 

Cost becomes a bigger problem as agents rack up more model calls. A chatbot might need one or two calls per request. An agent can burn through dozens finishing a single task. A system with several agents working together can burn through even more.

 

Routing every one of those calls to the most expensive model gets costly fast, and it adds latency too. But defaulting to the cheapest model everywhere brings its own headache: smaller models fail more often, throw invalid tool calls, or need several rounds of correction on code. Sometimes the "cheap" model ends up costing more once you count the rework.

 

So routing has to look at the whole workflow, not just the price tag on a single call.

 

NVIDIA's own numbers back the point: the company says NeMo Switchyard held frontier-level accuracy while cutting task-completion costs to roughly a third of what running Anthropic’s Claude Opus 4.8 alone would cost. Worth noting these are NVIDIA's internal benchmarks, not an independent audit.

 

Switchyard is a routing layer, not the entire orchestration system

 

It's worth separating model routing from model orchestration, related, but not the same thing.

 

An orchestrator decides how a workflow moves: which agent acts next, when a tool gets called, how work passes from one agent to another, when the task is actually done.

 

A router works at a narrower level; it just decides which model handles a given call.

 

NeMo Switchyard sits in that second category. It's meant to plug into an agent application and pick a model at each step, and NVIDIA lets developers tune the routing logic around priorities like quality, latency, and cost.

 

So Switchyard can slot into a bigger orchestration system; it isn't trying to be the whole workflow.

 

That distinction matters once you look at what the other major AI companies are building.

 

OpenAI is building around the agent loop

 

OpenAI's focus has been on the execution layer that wraps the model. Its Responses API lets a model call tools, get results back, and keep looping until the task's done. Its Agents SDK goes further, letting agents hand work off to each other, one agent delegating a task and passing along the relevant conversation state to the next.

 

That's a wider lens on orchestration as the system is deciding which agent takes the next part of the job, and not just which model to call.

 

OpenAI's more recent agent work leans further into this. The GPT-5.6 family includes an "ultra" setting that can coordinate multiple agents across parallel workstreams for heavier tasks, and the company has also put out Symphony, an open-source system for orchestrating Codex agents on software projects.

 

Inside these systems, model selection ends up as just one decision nested inside a much larger workflow.

 

Anthropic is exploring different model roles

 

Anthropic has been circling the same question from a different angle. One idea it has floated is an "advisor" pattern: a stronger model sets the strategy, and smaller models carry out the actual work. This split between labour repels stronger models from burning cycles on every task, while smaller models handle the routine load.

 

Anthropic has also pushed the point that these setups need to be evaluated against a company's own tasks, since the right split of work looks different depending on what you're building. A coding system and a customer-support system won't route the same way. 

 

Google is moving routing closer to the gateway

 

Google Cloud has added model routing directly into API Gateway, which accepts compatible requests and routes them across different models and providers, Gemini included, but not exclusively. The application itself doesn't need to know in advance which model will actually handle a given request.

 

Separately, Google is building out orchestration tools too. Its Agent Development Kit supports multiple workflow patterns, from predictable pipelines to setups where agents coordinate and delegate among themselves.

 

The infrastructure between agents and models is becoming more important

 

This trend isn't confined to the big AI companies. LangChain, LiteLLM, and various API gateway providers are all building in the space between AI applications and the models behind them.

 

NVIDIA's Switchyard announcement lists a handful of these integrations: LiteLLM is adding it as a plug-in to its proxy layer, Kong is wiring it into its AI Gateway, and LangChain has tested it with its Deep Agents system.

 

That's because developers don't want their agents locked to a single model. A research agent might use one model, a coding agent another, a verification agent a third, and the models themselves keep shifting underneath. One gets cheaper. Another gets better at coding. A local model becomes good enough for a specific task. Or a company decides certain data needs to stay on its own infrastructure.

 

A routing layer makes all of that easier to absorb.  But routing can create problems   There is a catch. 

A router can make a bad decision just as a model can produce a bad answer. A request may look simple but contain an unusual edge case. A smaller model may fail several times before the system escalates it. The additional calls can then erase the cost advantage. The router itself can also add latency.

 

There is another challenge.

 

Models keep changing. A routing policy that works well today may not work as well after a model update. Prices can change. Capabilities can change. Tool-use performance can change.

 

This means companies will need to evaluate routing policies, not just individual models.