Getting Started

Ruhroh helps you improve coding-agent loops by making each run inspectable and each comparison defensible. The fastest way to understand it is to run the credential-free example, inspect the saved evidence, and then replace one piece of the loop with your own task, agent, or reviewer.

That Run -> Inspect -> Compare -> Improve cycle is loop engineering.

1. 1
   Run

   Execute a realistic software task.

2. 2
   Inspect

   Review the outcome and its evidence.

3. 3
   Compare

   Repeat under controlled conditions.

4. 4
   Improve

   Change the loop and run again.

What The Sample Demonstrates

The public sample contains two runs of the same newsletter task. One delivers the required three stories and passes. The other creates a page with only one story and fails with goal_mismatch.

The resulting 50% pass rate is visible, but it is not publishable: the suite requires five runs, an evaluator-quality warning remains, and one run is recommended for review. Open the compare report, review queue, or claim index to see the result, evidence, and blockers as a reader would.

1. Install And Scaffold

Install Ruhroh in the project where you want to evaluate coding-agent work:

pnpm dlx @kestrel-agents/ruhroh demo

demo is the live first-run experience. It uses OpenRouter, prompts for an API key when OPENROUTER_API_KEY is not already set, installs pinned Aider tooling under .ruhroh/tools/, runs the bundled bookmark-manager task, evaluates the delivered app, and writes ruhroh-report.html plus local run evidence under .ruhroh/runs/.

The intended unscoped command is pnpm dlx ruhroh demo. Until that npm package name is available for this project, use the scoped package command above.

Use init when you are ready to add Ruhroh to a project or want the no-credentials fixture path:

pnpm add -D @kestrel-agents/ruhroh
pnpm exec ruhroh init

init creates a local ruhroh/ directory containing:

• a realistic example task and version-locked suite;
• a no-credentials fixture agent command;
• a no-credentials fixture reviewer command;
• the public schemas and a starter README.

It is safe to rerun when files are unchanged and refuses to overwrite local edits.

2. Connect The Fixture Loop

Point Ruhroh at the generated example agent and reviewer:

export RUHROH_RUN_AGENT_COMMAND="$PWD/ruhroh/adapters/fixture-newsletter/run.sh"
export RUHROH_EVAL_COMMAND="$PWD/ruhroh/evaluators/fixture-newsletter/run.sh"
pnpm exec ruhroh first-run

first-run is read-only. It checks the scaffold, task, suite, exported commands, and Harbor installation, then prints the exact next action. A missing requirement remains visible as a blocker instead of failing later during an expensive run.

When Harbor is unavailable, this command can still verify that the local files and commands are ready for a dry-run preview:

pnpm exec ruhroh first-run --allow-dry-run --json

A dry run previews the Harbor command. It does not execute an agent, produce a real ruhroh-loop-result.json, or count as a completed loop. See Local Fixture Run for the full readiness matrix and Troubleshooting when a check remains blocked.

3. Run The Example

When first-run reports that the full loop is ready, execute the fixture:

pnpm exec ruhroh run \
  --scenario-dir ruhroh/scenarios \
  --scenario simple-newsletter \
  --adapter custom-shell

Ruhroh generates the Harbor task, starts the selected agent command, runs the reviewer against the finished workspace, and preserves the run evidence. It also writes a run plan so later reports can compare the completed result with the intended task, adapter, sample, and seed.

4. Inspect What Happened

Ask workflow what evidence exists and what the next useful step is:

pnpm exec ruhroh workflow --html ruhroh-workflow.html

For a saved run directory, the inspection commands answer different questions:

Question	Command
Did this run deliver the outcome?	ruhroh report <run>
Did the reviewer support its judgment?	ruhroh eval-quality <run>
Does a person need to inspect anything?	ruhroh review <results>
What changes across repeated runs?	ruhroh compare <results>
Is the aggregate ready to share?	ruhroh publish-check <results>

See Evidence Files for the reviewer path and the Report Gallery for generated examples of each view.

5. Replace One Piece

The fixture proves the plumbing. The next useful step depends on what you want to improve:

Evaluate your agentUse an existing project, connect a live coding-agent command, and preserve its run metadata.Evaluate your workWrite a realistic task with outcome rules and evidence requirements.Evaluate your reviewerReplace the fixture judgment and calibrate it against known pass, fail, and review cases.

Change one part first. Keeping the task, reviewer, and execution conditions stable makes it easier to understand whether a new agent or prompt actually improved the result.

6. Collect A Comparable Cohort

Once your task, agent connector, and reviewer are ready, freeze the task set in a suite and create the intended run matrix before collection:

pnpm exec ruhroh plan \
  --scenario-dir ruhroh/scenarios \
  --suite-dir ruhroh/suites \
  --suite <suite-id> \
  --adapter ./ruhroh/adapters/<agent>/run.sh \
  --runs 5

Use the same selection with ruhroh run, then compare the completed result root against .generated/ruhroh/ruhroh-run-plan.json. Ruhroh will surface missing samples, mixed versions, weak reviewer evidence, low sample counts, and other conditions that make the comparison difficult to defend.

Follow Publish a Benchmark Result for the full task-to-publication workflow. Use the CLI Reference for all flags, JSON contracts, and exit codes.

Choose The Next Guide

Goal	Guide
Add Ruhroh to a real codebase	Add to Existing Project
Author a task and reviewer	Publish a Benchmark Result
Understand Ruhroh terminology	Core Concepts
Diagnose one or more runs	Evidence Files
Understand statistical and governance rules	Benchmark Methodology
Create a portable result	Publish Claims