On August 11, 2026, NVIDIA announced "NeMo Switchyard," an open-source router that distributes AI agent requests across multiple large language models (LLMs). Rather than calling an expensive model for every single task, it reserves that model for the hard cases only. In LangChain's evaluation, this cut costs by 74% compared to using Claude Opus 4.8 alone, but accuracy dropped from 86.0% to 80.0%. What enterprises get isn't a blanket discount—it's a mechanism for designing a cost-versus-quality trade-off on a per-workflow basis.
A 74% cost cut, but accuracy fell 6 points
LangChain evaluated Switchyard on 145 multi-turn tasks spanning customer support, incident investigation, and automation across messaging, ticketing, and email. Each task involved an average of 6.3 model calls. The setup started processing with the cheaper NVIDIA Nemotron 3.5 Lightning, then used an "escalation" approach—running five passes in which a judge model detects when the cheap model is stuck and switches to Claude Opus 4.8.
| Configuration | Accuracy | Evaluation Cost | Cost per Completed Task |
|---|---|---|---|
| Claude Opus 4.8 only | 86.0% | $11.45 | $0.092 |
| Switchyard routing | 80.0% | $3.00 | $0.026 |
| Nemotron 3.5 Lightning only | 77.7% | $0.72 | $0.006 |
Switchyard's evaluation cost dropped from $11.45 to $3.00—a 74% reduction. However, accuracy fell by 6.0 points. While it was 2.3 points higher than Lightning alone, that gap is smaller than the 2.7-point variance observed across evaluation runs. LangChain itself does not claim that routing outperformed the cheap model alone. If minimizing cost is the priority and Lightning's 77.7% meets requirements, skipping the router entirely would be cheaper under the same evaluation.
Evaluation conditions also vary. In a separate test NVIDIA cited, Cognition routed between Opus 5 and Kimi K2.7, scoring 50.6% on the FrontierCode Main coding benchmark—2.8 points lower than Opus 5 alone, with an average cost of $3.11, about 28% cheaper. The gap between the 74% and 28% figures shows that the savings rate shifts depending on the price difference between models and the task mix.
Choosing the cost of finishing the job, not the per-token price
Switchyard is a Rust-based proxy-and-library that sits between the application and model providers. It handles OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages requests through a common internal format, then converts them to the format of the selected model. On the application side, you connect to a single model ID while switching between OpenAI-compatible servers, the Anthropic API, or locally run models underneath.
There are four basic routing methods. The random method is for A/B testing; the LLM classifier judges the required capability from the request content. The stage router reads tool execution results, errors, and task progress, sending exploration and recovery work to stronger models and mechanical edits to cheaper ones. In the escalation method, a judge model tracks the cheap model's execution results, and by default, if problems are detected on two consecutive turns, that session is pinned to the stronger model going forward.
Judging solely by per-token price misreads the actual bill. Even a cheap model can rack up costs if it produces long outputs or calls tools repeatedly, inflating the cost of finishing the job. Switchyard logs the selected model, the reasoning behind the decision, token counts, latency, and call outcomes. Where a request is routed depends on the method used—based on request content, tool history, or session progress. NVIDIA's Joey Conway told The Register that agent architectures are starting to emerge in which a strong model orchestrates the overall flow while delegating execution to smaller, specialized models.
7% went to Opus, but the judge consumed 21.2%
In LangChain's evaluation, only 7% of all calls were routed to Opus. Yet Opus accounted for 68.4% of the routed configuration's total cost. Lightning, meanwhile, handled 93% of the calls while consuming just 10.4% of the cost. The average cost per call was $0.0324 for Opus versus $0.00037 for Lightning—a roughly 87-fold difference.
The remaining 21.2% was the cost of the judge model, which determines whether the cheap model is stuck. The judge runs on every turn until a switch to the stronger model occurs, and it doesn't benefit from input caching. In LangChain's setup, the judge consumed $0.64 per evaluation, meaning the minimum share of calls that had to go to the cheap model to break even was 5.9%. The observed 93% cleared that bar, but if the price gap between the two models were narrower, the required share could exceed 100%—meaning routing would save nothing at all.
Billing isn't stable either. Across the five evaluation runs, the share of calls sent to Opus ranged from 4.1% to 9.1%, and total cost ranged from $2.16 to $3.61. Even if average cost can be lowered, runs with a cluster of difficult requests will see heavier use of the expensive model. Enterprises need to budget for the most expensive run—not just the 74% headline figure—and factor the 6-point accuracy drop into their quality bar.
Where should companies start testing a pre-alpha router?
The public repository explicitly labels Switchyard as pre-alpha, noting that the API and algorithms may change significantly before v1.0. It further warns that the software is "experimental and not intended for production use." While the Apache 2.0 license lets you inspect the code, this is not, at present, a finished product you can drop directly into an enterprise's core routing path.
Operational infrastructure also needs to be built separately. Switchyard sends requests to configured endpoints and logs the selected model, reasoning, token counts, latency, and call outcomes—but it does not launch or manage the model servers at those endpoints itself. If you're mixing multiple external APIs with self-hosted models, credentials, failover on outages, and data governance around which endpoint a request can be routed to remain the adopter's responsibility.
Each method also has a narrow scope of applicability. Escalation assumes multi-turn history and requires a session ID to use two or more consecutive judgments—it's not suited to single-shot requests. Because the judge model is called after the cheap model responds, LangChain measured roughly 700 milliseconds of added latency. If minimizing latency matters, the stage router—which doesn't require an extra model call—is a candidate, though it lacks sufficient signal in chats with sparse tool-execution history.
The first move in adoption shouldn't be flipping production traffic over immediately. Fix a set of representative tasks and measure completion rate and total cost across three configurations: the strong model alone, the cheap model alone, and the routed setup. From there, check the judge model's cost, the latency ceiling, and the share of traffic sent to the strong model. Whether Switchyard can actually curb enterprise AI spending won't be decided by the announced 74% figure—it will be decided by the cost-per-completed-task number your own organization measures, failure costs included.