Evals-driven improvement
for AI agents.

An open-source plugin for Claude Code, Codex and other coding agents. RELAI turns requirements, traces and expert feedback into executable evals, then helps improve your agent against them.

RELAI · from requirements to measurable improvement agent demo

A real RELAI workflow, from the first request to evaluated results.

Evals are the hard part.
RELAI builds them.

From real evidence

Your traces, code, docs and risks.

Realistic

The real agent, end to end, in a container.

Trustworthy

Frozen graders. Changes kept only with evidence.

Two ways to use it.

01

Agent applications and business workflows

Build evals from traces, or generate them from the agent’s code, documentation and risks. Improve prompts, tools, workflow, memory or model against the resulting suite.

Use RELAI's discover skillto find where this agent can fail

02

Coding agents and software factories

Turn coding sessions, pull requests and git history into tasks any coding agent can run. Compare models, instructions, skills and agent setups on your code.

Use RELAI's coding-evals skillto build 20 evals from <your-repo>

Discover. Eval. Improve. Repeat.

01

Discover the behavior

Turn requirements, risks, traces and open questions into the EvalsWiki: readable, versioned files that guide what to test and why.

02

Build and freeze the eval suite

Create realistic evals in Harbor’s open task format, with environments and graders. Review and freeze the suite, then record a baseline before comparing changes.

03

Improve against the baseline

Generate harness improvements—prompts, tools, workflows and model settings—that meet your objective and pass the suite’s regression checks.

04

Repeat as evidence arrives

Bring new traces, expert answers and failures into the next discovery, evaluation and improvement cycle.

Open source. On your terms.

  • Apache 2.0Read, fork and extend every line.
  • In your repoEvals and results are plain, versioned files.
  • Your keysRuns locally with the models you choose.
  • No lock-inStandard Harbor tasks, no server, no account.
your-agent/
relai/
├── wiki/        # the EvalsWiki
├── tests/       # Harbor tasks
├── records/     # reviews and decisions
├── changes.md   # every change tried
└── STATUS.md    # what's next

RELAI has helped us turn hard use cases into evals, and evals into measurable improvements in the agents we ship to customers.

C3.ai
Nikhil KrishnanCTO & Chief AI Officer, C3.ai
+3–10 pts

pass-rate points

AutomationBench
22–44%

lower cost per task

AutomationBench
62.5→79.2%

pass rate

Terminal-Bench 2.0

Start with one request.

claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loop

Run in your terminal, then start a new Claude Code session.

Open in your desktop app Opens Claude Code with a setup prompt.