Ruhroh compare

1 task/agent result group

run plan 2 planned samples

benchmark suite ruhroh-sample@0.1.0; minimum runs=5

Publish And Evidence Overview

Ready to publishno
Scopesuite
Runs1/2 passing
Result files2
Evidence files24 tracked
Planned runsclean
Human review0 required, 1 recommended
Readiness blockers3

2 hashed result files; 24 named evidence files; 0 missing evidence files; 2 planned samples; benchmark suite ruhroh-sample@0.1.0

Ready To Publish?

ScopePublishableBlockersAdvisories
suitenoagent-a: suite minimum runs or scenario coverage not satisfied; simple-newsletter/agent-a: fewer than 5 runs; treat pass rate and pass@k as directional; simple-newsletter/agent-a: eval-quality warnings presentagent-a: simple-newsletter has 2/5 required runs; agent-a: scenario-level warnings present; simple-newsletter/agent-a: eval command evidence includes non-zero exit codes (1); simple-newsletter/agent-a/sample-newsletter-agent-a-2: review recommended

Side-by-Side Results

Taskagent-a
simple-newsletter50% (1/2); CI 9.5%-90.5%; review 0; warnings 2

What Needs Attention

TaskAgent connectorNon-passing runsFailure bucketsReview requiredReviewer warningsMissing evidence warningsStatistical warningsNext action
simple-newsletteragent-a1/2goal_mismatch=10eval command evidence includes non-zero exit codes=1nonefewer than 5 runs; treat pass rate and pass@k as directionalStrengthen reviewer evidence before citing the score.

Cost and Efficiency

TaskAgent connectorUsage coverageCost coverageTotal costMean costCost per passToken coverageTotal tokensTokens per pass
simple-newsletteragent-a2/2 runs2/2 runs$0.05$0.025$0.052/2 runs16001600

Benchmark Suite Coverage By Agent

Agent connectorTasksMissing tasksRuns per taskRunsPass rate95% CIMean task pass rateMin runsWarnings
agent-a1/1unknownsimple-newsletter=2250% (1/2)9.5%-90.5%50%not satisfiedsimple-newsletter has 2/5 required runs; scenario-level warnings present

Human Review Queue

PriorityTaskAgent connectorRun idScoreReviewFailureReasonsTranscriptsEvent logs
recommendedsimple-newsletteragent-asample-newsletter-agent-a-20failedgoal_mismatchnon-passing run: goal_mismatch; eval command evidence includes non-zero exit codes; unmet criterion: Newsletter must include three sample stories../sources/results/run-2/transcript.log./sources/results/run-2/events.jsonl

Evidence Browser

Open the saved evidence for each run without digging through the full file table.

simple-newsletter / agent-a / sample-newsletter-agent-a-1

simple-newsletter/agent-a/1-of-2

Stack: target=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs|protocol=fixture-command; recommended=yes

simple-newsletter / agent-a / sample-newsletter-agent-a-2

simple-newsletter/agent-a/2-of-2

Stack: target=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs|protocol=fixture-command; recommended=yes

Saved Result Evidence

TaskAgent connectorBenchmark targetRun idSample idTask versionResult JSONSHA-256Named evidence files
simple-newsletteragent-atarget=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs|protocol=fixture-command; recommended=yessample-newsletter-agent-a-1simple-newsletter/agent-a/1-of-21.0.0./sources/results/run-1/ruhroh-loop-result.json19aafd3de86aea748b915e7bbdffcba7ef769873a843baca2a960c6cd195a568evalInput: available ./sources/results/run-1/ruhroh-loop-eval-input.json 3763a8caaa6ef3a89cdbc9aab8335e5aab51837d717e007ac6bdb83676239c1e
evalResult: available ./sources/results/run-1/ruhroh-loop-eval.json 076c551c4114b48d017aebf98480d339dc92b72ca3a4fb164a2f4074fc6eea93
events: available ./sources/results/run-1/events.jsonl 354f962620ba81e40eee8813d3ad33cddbd01e7c40f9bf10166f843d7f7c7060
eventsTarball: available ./sources/results/run-1/ruhroh-loop-events.tar.gz 2002b38a24ad7ff391d9b87433e1df4e852ebc937edf13e5bd95b44798e8b1e5
implementationRuns: available ./sources/results/run-1/ruhroh-loop-iterations.jsonl 791fecab95e947282d21a5293f8b1a127310161277ac9efba6c50bc7d257d867
journey: available ./sources/results/run-1/ruhroh-loop-journey.json cd414b655de7de51f329079a8f9f8caaf6f4b417b2e99180e43acefa61d2e2f2
result: available ./sources/results/run-1/ruhroh-loop-result.json 19aafd3de86aea748b915e7bbdffcba7ef769873a843baca2a960c6cd195a568
runManifest: available ./sources/results/run-1/ruhroh-run-manifest.json ddb29446d073b45540bb387b8bd8d751938580e5cdacce6f4c82ca692492eedc
transcript: available ./sources/results/run-1/transcript.log b64339f315ee62b69bca610f6689b7ba7e5172c6b15147fbed0ad2c28f02139a
transcriptsTarball: available ./sources/results/run-1/ruhroh-loop-transcripts.tar.gz 454c0113d5058433b39a1ae789695420ce3cf42c8706ddc22778b671e55a8146
workspaceSummary: available ./sources/results/run-1/ruhroh-workspace-summary.json ddfc18e19275262cd383cbc3d744e8048a5c9366cf2faaa5bcad065b609e6caa
workspaceTarball: available ./sources/results/run-1/ruhroh-workspace.tar.gz 3c3f9efb11335aa9ef5106a20811c02a379548e6f7c7cedb4ccede370c2a1ca8
simple-newsletteragent-atarget=docs-fixture-native-stack; stream=native-stack; harness=fixture-newsletter@docs-sample-v1; requested=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); actual=docs/sample-agent (protocol=fixture-command, prompt=docs-v1); providerPath=provider=docs|protocol=fixture-command; recommended=yessample-newsletter-agent-a-2simple-newsletter/agent-a/2-of-21.0.0./sources/results/run-2/ruhroh-loop-result.jsondb0fd51fa1f598a701be9d2fee78c1f21a63b5ef6b6319a48c01a7fefced4bb0evalInput: available ./sources/results/run-2/ruhroh-loop-eval-input.json 3763a8caaa6ef3a89cdbc9aab8335e5aab51837d717e007ac6bdb83676239c1e
evalResult: available ./sources/results/run-2/ruhroh-loop-eval.json d319f036ff5f3331b31819d61f5f182b4bf8943e542523e9962b847a906e9fa8
events: available ./sources/results/run-2/events.jsonl c866b4a17696f13dbce7e7b63e649de021acbf7e9cd6a55e26b3c44240de8fc9
eventsTarball: available ./sources/results/run-2/ruhroh-loop-events.tar.gz 3f5e5b3c5e59edbefa9782469149a61fc6eb18b9eebdff1f990b614013b24f37
implementationRuns: available ./sources/results/run-2/ruhroh-loop-iterations.jsonl 3d9b3814142e91963857753f9de32e03cac88db3d54ce487446ffda00b8cfd15
journey: available ./sources/results/run-2/ruhroh-loop-journey.json ede885c0cdce22f1b632cf6ac4d2b46beaedb1a1aa745ef8b622dcc667094609
result: available ./sources/results/run-2/ruhroh-loop-result.json db0fd51fa1f598a701be9d2fee78c1f21a63b5ef6b6319a48c01a7fefced4bb0
runManifest: available ./sources/results/run-2/ruhroh-run-manifest.json 42c07b94425dd558bf6ffe6f6f99ade0a59868f72c0df9a502966b5633ce86df
transcript: available ./sources/results/run-2/transcript.log fc147d4635e80448bfd305b3aec71b5f0514de1c27b64057ab044fd3b03d3ed0
transcriptsTarball: available ./sources/results/run-2/ruhroh-loop-transcripts.tar.gz 68eeb595f2849fd813675e5846e2bbb76cdf61566a8c7b6c953005fbea4aa4e2
workspaceSummary: available ./sources/results/run-2/ruhroh-workspace-summary.json ddfc18e19275262cd383cbc3d744e8048a5c9366cf2faaa5bcad065b609e6caa
workspaceTarball: available ./sources/results/run-2/ruhroh-workspace.tar.gz 39f79c358adad823b4d80be383fcadb1187f09bdd8f92722d30ca8eab5e59ff9
TaskAgent connectorRunsPass rate95% CIpass@kMean scoreMean score CICohortMedian durationIterationsFailuresReviewReviewer warningsMissing evidence warningsUsageWarnings
simple-newsletteragent-a250% (1/2)9.5%-90.5%pass@1=50%, pass@2=100%0.50-1sample=simple-newsletter/agent-a/1-of-2|simple-newsletter/agent-a/2-of-2, sampleSeed=docs-seed-one|docs-seed-two, scenarioVersion=1.0.0, adapterVersion=docs-sample-v1, stream=native-stack, target=docs-fixture-native-stack, harness=fixture-newsletter@docs-sample-v1, providerPath=docs/fixture-command, agentModel=docs/sample-agent@2026-07-08, agentCanonicalModel=docs/sample-agent@2026-07-08, agentPrompt=docs-v1, evalModel=docs/docs-fixture-evaluator@2026-07-08, evalPrompt=docs-eval-v1, judge=hybrid/docs-fixture-evaluator@2026-07-08, env=sha256:111111111111111111111111111111111111111111111111111111111111111160000ms1=2none=1, goal_mismatch=10eval command evidence includes non-zero exit codes=1nonerunsWithUsage=2, runsWithCost=2, runsWithTokens=2, totalCostUsd=$0.05, meanCostUsd=$0.025, costPerPass=$0.05, totalTokens=1600, meanTotalTokens=800, tokensPerPass=1600fewer than 5 runs; treat pass rate and pass@k as directional