From real evidence
Your traces, code, docs and risks.
An open-source plugin for Claude Code, Codex and other coding agents. RELAI turns requirements, traces and expert feedback into executable evals, then helps improve your agent against them.
A real RELAI workflow, from the first request to evaluated results.
Your traces, code, docs and risks.
The real agent, end to end, in a container.
Frozen graders. Changes kept only with evidence.
Build evals from traces, or generate them from the agent’s code, documentation and risks. Improve prompts, tools, workflow, memory or model against the resulting suite.
Use RELAI's discover skillto find where this agent can fail
Turn coding sessions, pull requests and git history into tasks any coding agent can run. Compare models, instructions, skills and agent setups on your code.
Use RELAI's coding-evals skillto build 20 evals from <your-repo>
Turn requirements, risks, traces and open questions into the EvalsWiki: readable, versioned files that guide what to test and why.
Create realistic evals in Harbor’s open task format, with environments and graders. Review and freeze the suite, then record a baseline before comparing changes.
Generate harness improvements—prompts, tools, workflows and model settings—that meet your objective and pass the suite’s regression checks.
Bring new traces, expert answers and failures into the next discovery, evaluation and improvement cycle.
your-agent/relai/ ├── wiki/ # the EvalsWiki ├── tests/ # Harbor tasks ├── records/ # reviews and decisions ├── changes.md # every change tried └── STATUS.md # what's next
RELAI has helped us turn hard use cases into evals, and evals into measurable improvements in the agents we ship to customers.
claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loopRun in your terminal, then start a new Claude Code session.
codex plugin marketplace add relai-ai/agent-eval-loop
codex plugin add relai@agent-eval-loopStart a new Codex session. Review and trust the optional hooks when prompted.
git clone https://github.com/relai-ai/agent-eval-loop.git ~/agent-eval-loop
mkdir -p .cursor/skills
cp -R ~/agent-eval-loop/skills/* .cursor/skills/Requires Git. Run from your project, then reload Cursor. If already cloned, reuse ~/agent-eval-loop.
git clone https://github.com/relai-ai/agent-eval-loop.git ~/agent-eval-loop
mkdir -p .github/skills
cp -R ~/agent-eval-loop/skills/* .github/skills/Requires Git. Run from your project, then reload GitHub Copilot. If already cloned, reuse ~/agent-eval-loop.
npx skills add relai-ai/agent-eval-loop --skill '*'Cursor, Gemini CLI, Copilot and more. Run from your project with Node.js installed.