From real evidence
Your traces, code, docs and risks.
An open-source plugin for Claude Code, Codex and other coding agents. RELAI turns requirements, traces and expert feedback into executable evals, then helps improve your agent against them.
claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loopYour traces, code, docs and risks.
The real agent, end to end, in a container.
Frozen graders. Changes kept only with evidence.
Build evals from traces, or generate them from the agent’s code, documentation and risks. Improve prompts, tools, workflow, memory or model against the resulting suite.
Use RELAI's discover skillto find where this agent can fail
Turn coding sessions, pull requests and git history into tasks any coding agent can run. Compare models, instructions, skills and agent setups on your code.
Use RELAI's coding-evals skillto build 20 evals from <your-repo>
Turn requirements, risks, traces and open questions into the EvalsWiki: readable, versioned files that guide what to test and why.
Create realistic evals in Harbor’s open task format, with environments and graders. Review and freeze the suite, then record a baseline before comparing changes.
Generate harness improvements—prompts, tools, workflows and model settings—that meet your objective and pass the suite’s regression checks.
New evidence -traces, expert answers and failures- starts the next cycle.
relai/ ├── wiki/ # the EvalsWiki ├── tests/ # Harbor tasks ├── records/ # reviews and decisions ├── changes.md # every change tried └── STATUS.md # what's next
RELAI has helped us turn hard use cases into evals, and evals into measurable improvements in the agents we ship to customers.
claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loopRun in your terminal, then start a new Claude Code session.