In collaboration with Wenxiao Wang
Open-weight models are becoming increasingly attractive for building AI systems. The economics are better. They offer greater control over deployment and data, reduce dependence on a single model provider, and give enterprises a path toward greater sovereignty over their AI stack: owning more of the infrastructure that produces their intelligence.
But there is still a problem. Open-weight models are improving quickly, yet they continue to trail frontier models on many agentic tasks. And when the underlying model is somewhat less capable, the system around it—the agent harness—matters even more.
An agent is not just a model. It is a model plus a harness: the prompts, scaffolding, tools, and code that determine what context reaches the model and how the model acts on it.
So we asked:
How much of the remaining capability gap can be closed by optimizing the harness rather than changing the model?
Using RELAI's agent optimizer, we optimized harnesses around three open-weight models—DeepSeek-V4-Flash, MiMo-V2.5,and Nemotron-3-Super—using just 20 finance tasks from AutomationBench. We then evaluated them across the full 100-task domain, including 80 tasks never used during optimization.
Harness optimization moved the open-weight agents from 13.3% to 38.7% average pass rate, while cutting their average inference cost by 51%.
That brought them to the performance level of the three frontier agents—37.7% average pass rate—at roughly 3% of the inference cost.
Individually, optimized DeepSeek matched Claude Opus 5, MiMo came close to Gemini 3.5 Flash, and Nemotron exceeded GPT-5.5 in pass rate—all at 1.8–4.3% of the corresponding frontier agent's inference cost.
No model weights were changed.
But the more interesting result iswhythe gaps closed. The three open-weight agents were failing for completely different reasons:
- DeepSeek searched too much.
- Nemotron acted one tool call at a time.
- MiMo took too many shallow steps.
RELAI learned a different harness for each: bounding DeepSeek's search, restructuring Nemotron's planning and tool use, and giving MiMo tighter limits that produced fewer, deeper steps.
The result was not one universally better prompt. It was a different harness for each model.
This also raises a broader benchmarking question. When multiple models are evaluated under the same fixed harness, the benchmark measures the model–harness pair, not the model in isolation. A generic harness may fit some models much better than others—particularly when models have very different inference and tool-use behaviors.
Setup
We used AutomationBench finance v1.0.5: 100 workflow-automation tasks across simulated email, spreadsheets, chat, project trackers, and accounting systems. Pass rate is the fraction of tasks where every required assertion passes. Average score gives partial credit based on the fraction of assertions satisfied. Each agent can make roughly 50 model calls before the run terminates.
We evaluated:
For each open-weight model, RELAI optimized the harness using 20 tasks. The remaining 80 were held out from optimization. Model weights, task definitions, and scoring assertions remained fixed. Costs are estimated from recorded token counts and published prices.
What happens on the 20 learning tasks?
Every optimized open-weight agent reached or exceeded at least one frontier baseline.
Harness optimization moves all three open-weight agents into the performance range of frontier agents while keeping inference cost far lower.
But these are the tasks used during optimization. The more important question is whether the gains generalize.
The gains generalize to unseen tasks
We applied each optimized harness, unchanged, across all 100 finance tasks. Here are the detailed results (as illustrated in the header figure above):
The aggregate movement is the key result:
- Open-weight agents, default harness: 13.3% average pass rate, $3.38 average cost.
- Open-weight agents, optimized harness: 38.7%, $1.64.
- Frontier agents, default harness: 37.7%, $49.92.
And the result is not driven only by the 20 learning tasks.
On the 80 held-out tasks, optimized DeepSeek, MiMo, and Nemotron passed 45%, 33%, and 30%, versus 20%, 15%, and 8% before optimization. Their frontier counterparts scored 49% for Opus 5, 36% for Gemini 3.5 Flash, and 28% for GPT-5.5.
The harness did not simply memorize training tasks. It learned how to better use each model.
Why were the open-weight agents failing, before harness optimization?
One clue came from the benchmark's 50-model-call inference budget. Across the nine agents, runs that hit the limit passed only 9% of the time, versus 32% for runs that finished within budget.
Under the default harness, Opus 5 and GPT-5.5 never reached the limit while Gemini reached the limit only on 8 tasks. On the other hand, DeepSeek did so on 37 tasks, MiMo on 15, and Nemotron on 18.
But simply raising the limit would miss the point. The three open-weight models were wasting their budgets in different ways.
- DeepSeek: too much search. DeepSeek ran a median of 32 unique API searches per task, compared with 7 for Opus 5, and repeatedly explored the tool surface instead of committing to a plan. RELAI introduced explicit search limits, added a service-inventory tool, reduced unnecessary narration, and let the model reference previous tool outputs rather than reproduce them. This reduced searches 32 → 10, median model calls 38 → 17 and lowered inference cost by 78%.
- Nemotron: one action at a time. Nemotron issued exactly one tool call per tool-calling turn, even though both the default and optimized prompts instructed it to parallelize calls. It also spent a median of nine turns searching before its first action. Prompting alone did not change the one-call behavior, so RELAI adapted around it: it made search more targeted, required earlier planning, and made each turn more useful. Unique searches fell from 15 to 6, turns before first action from 9 to 5, and average score nearly doubled. This is an important example: the optimizer did not force the model to behave like another model. It adapted the harness around how this model actually behaved.
- MiMo: too many shallow steps. MiMo had the opposite problem. It generated relatively little reasoning per model call and compensated with many steps. RELAI gave it a shorter bounded workflow, capped search results at five, capped completions at 4,096 tokens, and reused prior tool outputs by reference. The result: 16 → 8 searches, 23 → 13 median model calls and 15 → 0 tasks hitting the limit. Output per model call increased from 371 to 611 tokens: fewer steps, but deeper ones. MiMo's total cost actually rose 36%, from $0.44 to $0.60, because of changes in cache utilization. But pass rate increased by 22 points, and cost per successful task still fell 50%.
Three models. Three failure modes. Three different fixes.
Cost per successful task
An agent that fails cheaply is not necessarily cheap. A more useful quantity is what it costs to actually complete a task successfully:
Cost per successful task falls sharply for all three open-weight agents after harness optimization.
Much of the cross-model cost difference comes from model pricing—the open-weight models are substantially cheaper per token.
So the stronger evidence for harness optimization is the within-model change: average open-weight pass rate increased from 13.3% to 38.7%, while average inference cost fell from $3.38 to $1.64.
Harness optimization is not simply about using fewer tokens. It is about spending inference where it matters.
The takeaway
Open-weight vs. frontier is not just a question of model capability. It is also a question of agent economics, control, and sovereignty.
Open-weight models can offer dramatically better economics, greater control over deployment and data, and less dependence on a small number of model providers. But realizing those advantages requires more than swapping one model for another.
The harness has to be designed around the model.
Our results suggest that a meaningful part of the apparent gap between open-weight and frontier agents can come from how effectively the surrounding system uses the model—how it provides context, manages tools, allocates inference, and adapts to model-specific behaviors.
That changes the deployment question. Instead of asking only,“Which model is strongest?”, teams should also ask:“What is the best agent we can build around the model we want to own and deploy?”
For enterprises, this can fundamentally change the economics of agentic AI. And at a broader level, it makes open-weight models more viable as a foundation for AI systems where cost, control, privacy, and technological sovereignty matter.
Before reaching for a stronger model, it may be worth optimizing the system around the one you already have.
RELAI's continual-learning workflow is designed to do exactly that: optimize the prompts, scaffolding, tools, and inference behavior around an agent from its own experience.
If you’d like to replicate these results with your own agent, meet our team.
