Ruhroh compare

1 task/agent result group

run plan 2 planned samples

benchmark suite ruhroh-sample@0.1.0; minimum runs=5

Publish And Evidence Overview

Ready to publishno

Scopesuite

Runs1/2 passing

Result files2

Evidence files24 tracked

Planned runsclean

Human review0 required, 1 recommended

Readiness blockers3

2 hashed result files; 24 named evidence files; 0 missing evidence files; 2 planned samples; benchmark suite ruhroh-sample@0.1.0

Ready To Publish?

Scope	Publishable	Blockers	Advisories
suite	no	agent-a: suite minimum runs or scenario coverage not satisfied; simple-newsletter/agent-a: fewer than 5 runs; treat pass rate and pass@k as directional; simple-newsletter/agent-a: eval-quality warnings present	agent-a: simple-newsletter has 2/5 required runs; agent-a: scenario-level warnings present; simple-newsletter/agent-a: eval command evidence includes non-zero exit codes (1); simple-newsletter/agent-a/sample-newsletter-agent-a-2: review recommended

Side-by-Side Results

Task	agent-a
simple-newsletter	50% (1/2); CI 9.5%-90.5%; review 0; warnings 2

What Needs Attention

Task	Agent connector	Non-passing runs	Failure buckets	Review required	Reviewer warnings	Missing evidence warnings	Statistical warnings	Next action
simple-newsletter	agent-a	1/2	goal_mismatch=1	0	eval command evidence includes non-zero exit codes=1	none	fewer than 5 runs; treat pass rate and pass@k as directional	Strengthen reviewer evidence before citing the score.

Cost and Efficiency

Task	Agent connector	Usage coverage	Cost coverage	Total cost	Mean cost	Cost per pass	Token coverage	Total tokens	Tokens per pass
simple-newsletter	agent-a	2/2 runs	2/2 runs	$0.05	$0.025	$0.05	2/2 runs	1600	1600

Benchmark Suite Coverage By Agent

Agent connector	Tasks	Missing tasks	Runs per task	Runs	Pass rate	95% CI	Mean task pass rate	Min runs	Warnings
agent-a	1/1	unknown	simple-newsletter=2	2	50% (1/2)	9.5%-90.5%	50%	not satisfied	simple-newsletter has 2/5 required runs; scenario-level warnings present

Human Review Queue

Priority	Task	Agent connector	Run id	Score	Review	Failure	Reasons	Transcripts	Event logs
recommended	simple-newsletter	agent-a	sample-newsletter-agent-a-2	0	failed	goal_mismatch	non-passing run: goal_mismatch; eval command evidence includes non-zero exit codes; unmet criterion: Newsletter must include three sample stories.	./sample-results/run-two/transcript.log	./sample-results/run-two/events.jsonl

Evidence Browser

Open the saved evidence for each run without digging through the full file table.

simple-newsletter / agent-a / sample-newsletter-agent-a-1

simple-newsletter/agent-a/1-of-2

Stack: target=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs|protocol=fixture-command; recommended=yes

result: ./sample-results/run-one/ruhroh-loop-result.json
how this run was produced: ./sample-results/run-one/ruhroh-run-manifest.json
review result: ./sample-results/run-one/ruhroh-loop-eval.json
review input: ./sample-results/run-one/ruhroh-loop-eval-input.json
agent journey: ./sample-results/run-one/ruhroh-loop-journey.json
agent turns: ./sample-results/run-one/ruhroh-loop-iterations.jsonl
transcript: ./sample-results/run-one/transcript.log
events: ./sample-results/run-one/events.jsonl
project summary: ./sample-results/run-one/ruhroh-workspace-summary.json
project archive: ./sample-results/run-one/ruhroh-workspace.tar.gz

simple-newsletter / agent-a / sample-newsletter-agent-a-2

simple-newsletter/agent-a/2-of-2

result: ./sample-results/run-two/ruhroh-loop-result.json
how this run was produced: ./sample-results/run-two/ruhroh-run-manifest.json
review result: ./sample-results/run-two/ruhroh-loop-eval.json
review input: ./sample-results/run-two/ruhroh-loop-eval-input.json
agent journey: ./sample-results/run-two/ruhroh-loop-journey.json
agent turns: ./sample-results/run-two/ruhroh-loop-iterations.jsonl
transcript: ./sample-results/run-two/transcript.log
events: ./sample-results/run-two/events.jsonl
project summary: ./sample-results/run-two/ruhroh-workspace-summary.json
project archive: ./sample-results/run-two/ruhroh-workspace.tar.gz

Saved Result Evidence

Task	Agent connector	Benchmark target	Run id	Sample id	Task version	Result JSON	SHA-256	Named evidence files
simple-newsletter	agent-a	target=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs\|protocol=fixture-command; recommended=yes	sample-newsletter-agent-a-1	simple-newsletter/agent-a/1-of-2	1.0.0	./sample-results/run-one/ruhroh-loop-result.json	19aafd3de86aea748b915e7bbdffcba7ef769873a843baca2a960c6cd195a568	evalInput: available ./sample-results/run-one/ruhroh-loop-eval-input.json 3763a8caaa6ef3a89cdbc9aab8335e5aab51837d717e007ac6bdb83676239c1e evalResult: available ./sample-results/run-one/ruhroh-loop-eval.json 076c551c4114b48d017aebf98480d339dc92b72ca3a4fb164a2f4074fc6eea93 events: available ./sample-results/run-one/events.jsonl 354f962620ba81e40eee8813d3ad33cddbd01e7c40f9bf10166f843d7f7c7060 eventsTarball: available ./sample-results/run-one/ruhroh-loop-events.tar.gz 2002b38a24ad7ff391d9b87433e1df4e852ebc937edf13e5bd95b44798e8b1e5 implementationRuns: available ./sample-results/run-one/ruhroh-loop-iterations.jsonl 791fecab95e947282d21a5293f8b1a127310161277ac9efba6c50bc7d257d867 journey: available ./sample-results/run-one/ruhroh-loop-journey.json cd414b655de7de51f329079a8f9f8caaf6f4b417b2e99180e43acefa61d2e2f2 result: available ./sample-results/run-one/ruhroh-loop-result.json 19aafd3de86aea748b915e7bbdffcba7ef769873a843baca2a960c6cd195a568 runManifest: available ./sample-results/run-one/ruhroh-run-manifest.json ddb29446d073b45540bb387b8bd8d751938580e5cdacce6f4c82ca692492eedc transcript: available ./sample-results/run-one/transcript.log b64339f315ee62b69bca610f6689b7ba7e5172c6b15147fbed0ad2c28f02139a transcriptsTarball: available ./sample-results/run-one/ruhroh-loop-transcripts.tar.gz 454c0113d5058433b39a1ae789695420ce3cf42c8706ddc22778b671e55a8146 workspaceSummary: available ./sample-results/run-one/ruhroh-workspace-summary.json ddfc18e19275262cd383cbc3d744e8048a5c9366cf2faaa5bcad065b609e6caa workspaceTarball: available ./sample-results/run-one/ruhroh-workspace.tar.gz 3c3f9efb11335aa9ef5106a20811c02a379548e6f7c7cedb4ccede370c2a1ca8
simple-newsletter	agent-a	target=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs\|protocol=fixture-command; recommended=yes	sample-newsletter-agent-a-2	simple-newsletter/agent-a/2-of-2	1.0.0	./sample-results/run-two/ruhroh-loop-result.json	db0fd51fa1f598a701be9d2fee78c1f21a63b5ef6b6319a48c01a7fefced4bb0	evalInput: available ./sample-results/run-two/ruhroh-loop-eval-input.json 3763a8caaa6ef3a89cdbc9aab8335e5aab51837d717e007ac6bdb83676239c1e evalResult: available ./sample-results/run-two/ruhroh-loop-eval.json d319f036ff5f3331b31819d61f5f182b4bf8943e542523e9962b847a906e9fa8 events: available ./sample-results/run-two/events.jsonl c866b4a17696f13dbce7e7b63e649de021acbf7e9cd6a55e26b3c44240de8fc9 eventsTarball: available ./sample-results/run-two/ruhroh-loop-events.tar.gz 3f5e5b3c5e59edbefa9782469149a61fc6eb18b9eebdff1f990b614013b24f37 implementationRuns: available ./sample-results/run-two/ruhroh-loop-iterations.jsonl 3d9b3814142e91963857753f9de32e03cac88db3d54ce487446ffda00b8cfd15 journey: available ./sample-results/run-two/ruhroh-loop-journey.json ede885c0cdce22f1b632cf6ac4d2b46beaedb1a1aa745ef8b622dcc667094609 result: available ./sample-results/run-two/ruhroh-loop-result.json db0fd51fa1f598a701be9d2fee78c1f21a63b5ef6b6319a48c01a7fefced4bb0 runManifest: available ./sample-results/run-two/ruhroh-run-manifest.json 42c07b94425dd558bf6ffe6f6f99ade0a59868f72c0df9a502966b5633ce86df transcript: available ./sample-results/run-two/transcript.log fc147d4635e80448bfd305b3aec71b5f0514de1c27b64057ab044fd3b03d3ed0 transcriptsTarball: available ./sample-results/run-two/ruhroh-loop-transcripts.tar.gz 68eeb595f2849fd813675e5846e2bbb76cdf61566a8c7b6c953005fbea4aa4e2 workspaceSummary: available ./sample-results/run-two/ruhroh-workspace-summary.json ddfc18e19275262cd383cbc3d744e8048a5c9366cf2faaa5bcad065b609e6caa workspaceTarball: available ./sample-results/run-two/ruhroh-workspace.tar.gz 39f79c358adad823b4d80be383fcadb1187f09bdd8f92722d30ca8eab5e59ff9

Task	Agent connector	Runs	Pass rate	95% CI	pass@k	Mean score	Mean score CI	Cohort	Median duration	Iterations	Failures	Review	Reviewer warnings	Missing evidence warnings	Usage	Warnings
simple-newsletter	agent-a	2	50% (1/2)	9.5%-90.5%	pass@1=50%, pass@2=100%	0.5	0-1	sample=simple-newsletter/agent-a/1-of-2\|simple-newsletter/agent-a/2-of-2, sampleSeed=docs-seed-one\|docs-seed-two, scenarioVersion=1.0.0, adapterVersion=docs-sample-v1, stream=native-stack, target=docs-fixture-native-stack, harness=fixture-newsletter@docs-sample-v1, providerPath=docs/fixture-command, agentModel=docs/sample-agent@2026-07-08, agentCanonicalModel=docs/sample-agent@2026-07-08, agentPrompt=docs-v1, evalModel=docs/docs-fixture-evaluator@2026-07-08, evalPrompt=docs-eval-v1, judge=hybrid/docs-fixture-evaluator@2026-07-08, env=sha256:1111111111111111111111111111111111111111111111111111111111111111	60000ms	1=2	none=1, goal_mismatch=1	0	eval command evidence includes non-zero exit codes=1	none	runsWithUsage=2, runsWithCost=2, runsWithTokens=2, totalCostUsd=$0.05, meanCostUsd=$0.025, costPerPass=$0.05, totalTokens=1600, meanTotalTokens=800, tokensPerPass=1600	fewer than 5 runs; treat pass rate and pass@k as directional