Evals-driven improvement
for AI agents.

RELAI is an open-source plugin for coding agents. It turns requirements, traces and expert feedback into executable evals, then helps improve your agent against them.

claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loop
Open in Claude Code

Run in your terminal, then start a new Claude Code session.

View on GitHub
RELAI loop: discover maps behavior, eval builds and runs evals, and improve makes measured changes before the loop repeats.loopdiscovermap the behaviorevalbuild and run evalsimprovemeasured changes

Build evals first, then improve your agent against them.

The same eval-first discipline supports production agents, business workflows and software factories.

01

Agent applications and business workflows

Build evals from traces, or generate them from the agent’s code, documentation and risks. Improve prompts, tools, workflow, memory or model against the resulting suite.

Use RELAI’s discover skill to map this agent’s behavior and where it can fail

02

Coding agents and software factories

Turn coding sessions, pull requests and git history into tasks any coding agent can run. Compare models, instructions, skills and agent setups on your code.

Use RELAI’s coding-evals skill to mine ../api for tasks, then compare our coding agents

The hard part is the eval.

RELAI turns what the agent should do, where it fails and what your team has learned into executable evaluation work.

01

Discover the behavior

Turn requirements, risks, traces and open questions into the EvalsWiki: readable, versioned files that guide what to test and why.

02

Build and freeze the eval suite

Create realistic evals in Harbor’s open task format, with environments and graders. Review and freeze the suite, then record a baseline before comparing changes.

03

Improve against the baseline

Generate harness improvements—prompts, tools, workflows and model settings—that meet your objective and pass the suite’s regression checks.

04

Repeat as evidence arrives

Bring new traces, expert answers and failures into the next discovery, evaluation and improvement cycle.

Measured impact

RELAI has helped us turn hard use cases into evals, and evals into measurable improvements in the agents we ship to customers.

C3.aiNikhil KrishnanCTO & Chief AI Officer, C3.ai
+3–10

Pass-rate points gained across three frontier models.

AutomationBench study
22–44%

Lower inference cost per successful task across the same models, excluding optimization cost.

AutomationBench study
79.2%

Pass rate after the first optimization round; baseline 62.5%.

Terminal-Bench 2.0 study

Applying RELAI to a production agent? Our team can help with professional services. Contact us

Own the intelligence your agents accumulate.

Your EvalsWiki and Harbor tasks live as readable, versioned files in your repository. That knowledge compounds with every iteration.

  • Versioned with your code.
  • Reusable across models and agent setups.
  • Portable in the open-source Harbor format, or runnable with your existing infrastructure.
relai/
├── EvalsWiki/
│   ├── requirements.md
│   ├── risks.md
│   └── open-questions.md
├── evals/
│   ├── tasks/
│   └── graders/
├── STATUS.md
└── README.md

# Versioned with the repository
# Reusable across models
# Runnable with your infrastructure

Set up RELAI in your desktop app.

Open your coding agent. Choose the project for RELAI. Review and confirm setup.

Open in Claude Code

Copies installation commands and opens your app.

Then ask your coding agent:

For an agentMap this agent’s behavior and build the evals.
For a software factoryTurn our recent pull requests into coding-agent evals.

RELAI creates a versioned relai/ folder containing the knowledge, evals and current status.