Ruhroh workflow guide
A read-only path from the built-in local example to evidence-backed comparison and audit-ready publication.
Next Action
Publish an audit-ready benchmark result: bundle-validation: Publication packet is valid but embedded publish-check or ready-to-publish checks are blocked.
pnpm exec ruhroh publish-check sample-results --suite-dir ruhroh/suites --suite ruhroh-sample --run-plan ruhroh-run-plan.json --bundle ruhroh-publication --verify-sources
Docs: publish-claims, claim-registry, result-json-reference, benchmark-methodology
Project Paths
| Name | Path |
|---|---|
| Scenario directory | ./ruhroh/scenarios |
| Suite directory | ./ruhroh/suites |
| Generated directory | .generated/ruhroh |
First local fixture loop
Status: ready
A local fixture result artifact exists and can be inspected before authoring or publishing benchmarks.
| Check | Status | Details |
|---|---|---|
| fixture-result | ok | 2 ruhroh-loop-result.json file(s) found for first-loop inspection |
Commands
| Command |
|---|
pnpm exec ruhroh report sample-results/run-one/ruhroh-loop-result.json --html ruhroh-report.html |
Docs: getting-started, local-fixture-run, troubleshooting, artifacts, report-gallery
Author scenario and suite
Status: ready
A versioned scenario and suite are present for local benchmark authoring.
| Check | Status | Details |
|---|---|---|
| scenario-dir | ok | ./ruhroh/scenarios exists |
| suite-dir | ok | ./ruhroh/suites exists |
| selected-scenario | ok | ./ruhroh/scenarios/simple-newsletter exists |
| selected-suite | ok | ./ruhroh/suites/ruhroh-sample exists |
Commands
| Command |
|---|
pnpm exec ruhroh new-scenario my-task --scenario-dir ruhroh/scenarios |
pnpm exec ruhroh new-suite local-smoke --scenario-dir ruhroh/scenarios --suite-dir ruhroh/suites --scenario my-task |
pnpm exec ruhroh validate --scenario-dir ruhroh/scenarios --suite-dir ruhroh/suites --suite local-smoke |
Docs: concepts, write-a-scenario, scenario-format, benchmark-suites
Make evaluator quality operational
Status: ready
Reviewer wiring and calibration evidence are ready for repeated benchmark runs.
| Check | Status | Details |
|---|---|---|
| evaluator-command | ok | local evaluator exists at ruhroh/evaluators/fixture-newsletter/run.sh |
| calibration-cases | ok | 3 evaluator calibration case(s) are defined for simple-newsletter |
| calibration-report | ok | Calibration report passed 3 case(s). |
Commands
| Command |
|---|
pnpm exec ruhroh calibrate-evaluator --scenario-dir ruhroh/scenarios --scenario simple-newsletter --generated-dir .generated/ruhroh --json |
pnpm exec ruhroh new-evaluator local-evaluator --template hybrid |
export RUHROH_EVAL_COMMAND="$PWD/ruhroh/evaluators/local-evaluator/run.sh" |
Docs: write-an-evaluator, evaluator-cookbook, human-review
Preflight benchmark pack credibility
Status: ready
Scenario versions, calibration coverage, contamination review, and reward-hacking review are ready for collection.
| Check | Status | Details |
|---|---|---|
| strict-pack-inspection | ok | Pack inspection passed: 1 scenario(s), 1 suite(s), calibration and risk-review gates satisfied. |
Commands
| Command |
|---|
pnpm exec ruhroh inspect-pack --scenario-dir ruhroh/scenarios --suite-dir ruhroh/suites --require-calibrated --require-risk-reviewed --html ruhroh-pack-inspection.html --json |
Docs: benchmark-pack-registry, benchmark-methodology, scenario-evolution, evaluator-cookbook
Plan repeated agent runs
Status: ready
A run plan exists for repeatable adapter and scenario coverage.
| Check | Status | Details |
|---|---|---|
| run-plan | ok | ./ruhroh-run-plan.json exists |
Commands
| Command |
|---|
pnpm exec ruhroh plan --scenario-dir ruhroh/scenarios --suite-dir ruhroh/suites --suite ruhroh-sample --adapter custom-shell --runs 5 --json |
Docs: benchmark-methodology, benchmark-suites, cli-reference
Compare agents with evidence
Status: ready
Run results and a plan are available for evidence-backed comparison.
| Check | Status | Details |
|---|---|---|
| run-results | ok | 2 ruhroh-loop-result.json file(s) found |
| run-plan | ok | ./ruhroh-run-plan.json exists |
Commands
| Command |
|---|
pnpm exec ruhroh compare sample-results --suite-dir ruhroh/suites --suite ruhroh-sample --run-plan ruhroh-run-plan.json --html ruhroh-compare.html |
pnpm exec ruhroh review sample-results --json |
pnpm exec ruhroh eval-quality sample-results --html ruhroh-eval-quality.html --json |
Docs: artifacts, report-gallery, benchmark-methodology
Publish an audit-ready benchmark result
Status: needs_action
Validate the publication packet and claim index before making external claims.
| Check | Status | Details |
|---|---|---|
| publish-inputs | ok | Run results and run plan are available for publish-check. |
| claim-or-bundle | ok | publication packet inventory exists |
| bundle-validation | failed | Publication packet is valid but embedded publish-check or ready-to-publish checks are blocked. |
| registry-index | failed | claim-index.json exists but registry is blocked: blocked claim ./ruhroh-publication: agent-a: suite minimum runs or scenario coverage not satisfied |
Commands
| Command |
|---|
pnpm exec ruhroh publish-check sample-results --suite-dir ruhroh/suites --suite ruhroh-sample --run-plan ruhroh-run-plan.json --bundle ruhroh-publication --verify-sources |
pnpm exec ruhroh validate-bundle ruhroh-publication --json |
pnpm exec ruhroh claim-index ruhroh-publication --html ruhroh-claims.html --json > claim-index.json |
Docs: publish-claims, claim-registry, result-json-reference, benchmark-methodology
Workflow Docs
- getting-started
- local-fixture-run
- concepts
- write-a-scenario
- write-an-evaluator
- write-an-adapter
- benchmark-suites
- benchmark-pack-registry
- publish-claims
- claim-registry