In collaboration with Wenxiao Wang
The standard way to reduce the inference cost of an AI agent is to make it do less: use a cheaper model, limit the number of turns, shorten the context, or reduce the reasoning budget. These approaches can work, but they usually trade quality for cost.
The question we asked was:
Can harness optimization improve both quality and inference cost at the same time?
At RELAI, we approach this by making inference cost part of the optimization objective itself. Rather than constraining the agent upfront, we optimize its behavior for both task quality and cost.
RELAI’s verifiable continual learning uses the agent’s own trajectories to improve how it uses tools, manages context, and carries information across a task. The goal is not to force every run to use less computation, but to remove computation that is not helping.
We tested this on three agents using GPT-5.5, Claude Opus 5, and Gemini 3.5 Flash on AutomationBench.
- On the 20 finance tasks used for learning, cost per successful task fell by62%, 44%, and 55%, while pass rate increased by 25, 25, and 20 points, respectively.
- We then evaluated the optimized agents on the full 100-task finance domain, including 80 unseen tasks. The gains transferred: cost per successful task fell by 44%, 26%, and 22%, while pass rate increased by3, 10, and 5 points.
Across all three model families, the result was the same: higher task quality at lower inference cost.
The token traces show where the savings came from. The agents did not simply stop earlier or produce less. Much of the reduction came from carrying substantially less unnecessary context through their trajectories.
Setup
We used AutomationBench v1.0.5, Zapier's benchmark for workflow-automation agents. We focused on its finance domain: 100 tasks requiring an agent to operate across mocked business systems such as accounting platforms, spreadsheets, email, and CRMs. Tasks include workflows such as reconciling invoices, applying vendor credits, and handling expense-policy exceptions.
Each task is evaluated against the resulting system state. We report both a continuous score, which gives partial credit, and a strict pass rate, which counts only perfectly completed tasks.
For each model independently, we:
- Started from AutomationBench's default harness.
- Used RELAI on the first 20 finance tasks, optimizing jointly for task score and inference cost.
- Evaluated the initial and optimized agents on those 20 learning tasks as well as the full 100 tasks in the finance domain.
We used 200 learning rollouts for GPT-5.5 and Claude Opus 5, and 100 for Gemini 3.5 Flash.
Improving cost and quality on the learning tasks
We first evaluated the agents on the 20 finance tasks used during learning. As the cost–quality plot at the top shows, all three optimized agents move up and to the left: higher pass rate at lower total inference cost.
The improvements were substantial. Pass rate increased from 30% to 55% for GPT-5.5,45% to 70% for Claude Opus 5, and 30% to 50% for Gemini 3.5 Flash. At the same time, total inference cost fell from $9.42 to $6.48,$9.00 to $7.81, and $6.38 to $4.82, respectively.
A useful way to combine these two effects is to ask: how much inference cost does it take, on average, to complete one task successfully? This metric—total inference cost divided by the number of perfectly completed tasks—captures quality and cost in a single number.
Cost per successful task on the 20 learning tasks. RELAI reduces the inference cost required to complete a task successfully across all three models.
The same pattern holds under partial-credit scoring: average score increased by 14.9, 5.9, and 12.5 percentage points across the three models.
Detailed results on the 20 learning tasks from AutomationBench
On the tasks used for learning, all three agents therefore moved in the desired direction on both dimensions: better task performance at lower inference cost. The harder question is whether those gains carry over to tasks the agents never learned from.
The optimized agents generalize to unseen tasks
The first 20 tasks show that RELAI can jointly improve quality and inference cost on the tasks it learns from. The more important question is whether those improvements transfer.
We froze each optimized harness and evaluated it on the full 100-task finance domain, including 80 tasks that were never used during learning. The gains were smaller, as expected, but all three agents still improved on both dimensions.
Across the full finance domain, all three optimized agents again move up and to the left: higher pass rate at lower inference cost.
Pass rate increased from 27% to 30% for GPT-5.5, 49% to 59% for Claude Opus 5, and 37% to 42% for Gemini 3.5 Flash. Average score also increased for all three models.
At the same time, cost per successful task fell by 44%, 26%, and 22%, respectively.
Total spend divided by tasks solved across the full finance domain. Every model roughly halves the price of a completed task.
So the improvements were not confined to the optimization set. The learned harness changes transferred to unseen tasks, improving quality and inference cost simultaneously.
Most of the savings came from carrying less context
The token traces help explain what RELAI changed.
Input tokens and average context per model call across all 100 tasks. RELAI reduces the amount of context processed by every model, with the largest reductions for GPT-5.5 and Gemini 3.5 Flash.
Across the full 100-task evaluation, input-token usage fell:
- 65% for GPT-5.5
- 10% for Claude Opus 5
- 49% for Gemini 3.5 Flash
Average context per model call also decreased for every model:
- GPT-5.5: 19.8k → 8.5k
- Claude Opus 5: 17.7k → 14.1k
- Gemini 3.5 Flash: 16.4k → 8.6k
The reduction was not simply the result of agents taking fewer steps. GPT-5.5 and Gemini made somewhat fewer model and tool calls after optimization, but those reductions were far smaller than the reductions in input tokens.
Claude is an even cleaner example. After optimization, the Claude agent made 13% more model calls and 17% more tool calls than before—and still reduced total inference cost by about 11%. It took more steps, but carried less context into each one.
GPT-5.5 shows the same phenomenon from another angle. Its output-token generation increased by 30%, even as its input-token consumption fell by 65%.
So these agents did not become less expensive simply because they stopped earlier or generated less. They were processing less unnecessary history.
Agentic systems naturally accumulate context as they operate: tool outputs, intermediate results, previous searches, errors, and earlier decisions. If unnecessary information enters that history, it may be reread by the model across many later calls. The resulting cost compounds over the trajectory. RELAI's optimized harnesses reduced this accumulation. They carried less context forward while preserving the computation needed to continue working on the task.
Expensive runs became less expensive
Mean inference cost does not tell the whole story. In production, some of the most problematic behavior comes from the expensive tail of the distribution: unusually long trajectories, very large contexts, latency spikes, and unexpectedly expensive sessions.
We therefore looked at the distribution of task-level cost.
Per-task inference cost across all 100 tasks, sorted from most to least expensive. RELAI reduces the high-cost tail across all three models, with the largest savings on the most expensive runs.
At the 90th percentile, task cost fell:
- GPT-5.5: $1.37 → $0.75
- Claude Opus 5: $0.77 → $0.65
- Gemini 3.5 Flash: $0.62 → $0.54
The effect was also broader than a few extreme trajectories.
Of the 100 tasks:
- 69 became cheaper for GPT-5.5
- 63 became cheaper for Claude Opus 5
- 66 became cheaper for Gemini 3.5 Flash
Median task cost also decreased for all three models. So the aggregate savings were not driven by fixing a small number of pathological runs. Optimization shifted a substantial portion of each model's cost distribution.
Continual learning can learn efficiency, not just quality
Continual learning for AI agents is usually discussed in terms of quality. An agent encounters failures or feedback. Those experiences provide a learning signal. The agent changes, and future trajectories solve more tasks.
But a trajectory contains another useful signal: how computation was spent while producing the outcome.
A failed or inefficient trajectory may show that the agent searched repeatedly, selected unnecessary tools, accumulated irrelevant context, or continued executing after it had stopped making useful progress.
Those behaviors affect quality and inference cost at the same time. This creates an opportunity to optimize the two jointly. Instead of imposing a global rule such as “use fewer tokens” or “make fewer tool calls,” continual learning can use real trajectories to identify where additional computation helps and where it is being wasted.
That is what we see in these experiments. RELAI did not produce a uniformly smaller computation budget. It changed the behavior of the agent and, in particular, how computation was allocated across the trajectory. The result was higher quality at lower inference cost.
Limitations
These experiments cover one benchmark and one domain. AutomationBench's finance tasks share tools and structure, so transfer from the 20 learning tasks to the remaining 80 may be easier than transfer across unrelated domains.
The magnitude of the gains also varies across models. An agent that already manages context and tools efficiently should have less recoverable waste.
Finally, we did not tune the learning budget. We used 200 rollouts for GPT-5.5 and Claude Opus 5 and 100 for Gemini 3.5 Flash. Interestingly, Gemini achieved the largest full-domain score improvement despite using half the learning budget, but understanding that relationship requires a systematic budget sweep.
Use RELAI to improve the cost–quality frontier of your AI agents
RELAI’s continual-learning workflow now directly supports improving agent quality while reducing inference cost. Rather than treating cost reduction as a separate compression or post-processing step, RELAI optimizes task quality and inference cost jointly.
We offer three modes that determine how aggressively inference cost is weighted: Conservative prioritizes quality and takes only low-risk cost reductions; Balanced keeps quality as the primary objective while actively reducing inference cost; and Aggressive places greater weight on reducing inference cost.
You can try RELAI’s public preview at relai.ai.
