OBench1
obench1
OBench1 is our benchmark for coding-agent harnesses. Every harness gets the same model, the same settings and the same tasks at the same moment, so what’s being measured is the harness itself. These pages explain how it works, how the first full run went, what came out of it, and what went wrong along the way.
The short version
Overseer v5 had the highest approved score, 68.8. omp was next at 54.1, then Hermes at 44.3, dsh at 42.9 and pi at 36.7. Overseer’s interval overlaps omp’s, so the lead is likely but not settled.
Overseer was reliable in 11 of the 15 categories, and the others in 4 to 6. It spent the most of the five, $1.39 of the run’s $5.78, and was still the cheapest per approved point because it scored the most.
A few things matter for reading those numbers:
- 12 of pi’s and dsh’s cells never got an answer. The provider rate-limited every call and the harness gave up, which pulls both scores down. dsh would probably be third without it. Details.
- There was one safety-gate breach in the whole run, by omp.
- Overseer’s first attempt, a one-seed pilot, didn’t count. Its shell never worked, and it drove the Mac’s desktop instead. What went wrong covers that and everything else.
- OBench1 isn’t an especially demanding benchmark. It’s sized for the team’s current setup, one Mac and one inexpensive model. We also wrote both the benchmark and Overseer. Limits covers what that means for the result.
- All of it is open source: the runner, the tasks, their checks and every transcript. It’s on GitHub.
In these pages
- ResultsSuite totals, the 15 categories, cost, and how to read them.
- How it worksThe categories, the 27 tasks, and how graders and judges score a cell.
- Harnesses and fairnessWho ran, what they all shared, and what each got that the others didn’t.
- What went wrongA void pilot, Overseer driving the desktop, a stopped run, an exposed key, an evening of throttling and 12 empty cells.
- Limits and the sourceHow far this result reaches, and where to get the code, tasks and transcripts.
- What we learned about v5Where Overseer wasted tokens and time, where it slipped, and what worked.
- Run logThe run hour by hour, and how we watched it.
About this run
| Run | ob1-3seed-rc2 |
| When | 5 November 2026 16:29 UTC to 6 November 2026 03:21 UTC (10 h 52 min) |
| Cells | 405: 27 tasks × 5 harnesses × 3 seeds |
| Model | deepseek-v4.1-flash on the OpenCode Go subscription, thinking on, effort max |
| Harnesses | Overseer v5.0.0-rc2, omp 18.13.3, dsh 0.1.1-rc.2, pi 1.5.0, Hermes Agent 0.22.5 |
| Machine | One Mac, every cell in its own sandbox |
| Source | github.com/sirrayi/obench1 |