OBench1
results
Suite totals
Five harnesses ran 27 tasks three times each, which makes 405 cells. Every cell used deepseek-v4.1-flash through the same metered gateway. Overseer v5 came first on approved score, with omp second.
| Harness | Approved | 95% CI | Raw | Solved | Gates | Reliable | $/point | Spend |
|---|---|---|---|---|---|---|---|---|
| Overseer v5 | 68.8 | 54.0–82.7 | 95.2 | 61/81 | 0 | 11/15 | $0.0007 | $1.39 |
| omp | 54.1 | 37.0–69.8 | 87.2 | 50/81 | 1 | 6/15 | $0.0008 | $1.21 |
| Hermes | 44.3 | 29.6–60.1 | 85.5 | 49/81 | 0 | 5/15 | $0.0011 | $1.34 |
| dsh | 42.9 | 28.1–57.8 | 74.0 | 42/81 | 0 | 4/15 | $0.0009 | $1.00 |
| pi | 36.7 | 21.7–51.5 | 71.4 | 37/81 | 0 | 5/15 | $0.0008 | $0.84 |
Overseer’s interval (54.0–82.7) and omp’s (37.0–69.8) overlap, so read the order of the top two as likely rather than settled. Hermes, dsh and pi sit close together, and their intervals overlap almost completely.
pi’s and dsh’s totals include 8 and 4 cells that never got an answer from the provider. Every call in those cells was rate-limited, and the harness gave up within about 16 seconds. Without them dsh would probably be third rather than fourth. What went wrong has the details.
The early preview
We posted a preview before the run finished, covering 22 of the 27 tasks: Overseer 72.2, omp 56.2, Hermes 43.3, dsh 41.7 and pi 34.4. With all 27 in, Overseer’s lead over omp narrowed from 16.0 points to 14.7, and pi’s cost per approved point fell from $0.0010 to $0.0008.
By category
A harness is reliable in a category when it was approved on every seed in at least 90% of that category’s tasks. Overseer was reliable in 11 of the 15 categories and the others in 4 to 6. Nobody reached the summit, which needs all 15 and no gate breach anywhere.
| Category | Overseer v5 | omp | Hermes | dsh | pi |
|---|---|---|---|---|---|
| Long-horizon execution27 tasks | 96% | 85% | 67% | 63% | 52% |
| Context management27 tasks | 100% | 100% | 100% | 96% | 100% |
| Memory across sessions3 tasks | 100% | 67% | 33% | 33% | 0% |
| Orchestration and subagents2 tasks | 100% | 50% | 50% | 0% | 0% |
| Tool use27 tasks | 100% | 100% | 100% | 96% | 96% |
| Skills and extensibility1 task | 100% | 100% | 100% | 100% | 100% |
| Token, cache and prompt efficiency27 tasks | 81% | 81% | 78% | 81% | 70% |
| Speed27 tasks | 93% | 93% | 89% | 85% | 93% |
| Instruction following27 tasks | 85% | 81% | 59% | 52% | 48% |
| Policy and repo rules22 tasks | 91% | 95% | 91% | 68% | 73% |
| Out-of-the-box problem solving3 tasks | 67% | 67% | 67% | 33% | 0% |
| Verification, critique and review11 tasks | 73% | 55% | 55% | 45% | 27% |
| Safety and trust27 tasks | 100% | 81% | 70% | 96% | 96% |
| Resilience27 tasks | 100% | 85% | 85% | 63% | 52% |
| Honest reporting3 tasks | 100% | 100% | 100% | 67% | 33% |
Each figure is the share of the category’s judged tasks where the harness was approved on all three seeds; higher is better. The darker the cell, the higher the share, and bold marks a reliable category (90% or more).
Overseer was the only harness at 100% in memory, orchestration, safety and resilience. Its weakest categories were out-of-the-box problem solving (67%), verification and review (73%), efficiency (81%) and instruction following (85%). Policy is the one category where it trails: omp has 95% to its 91%.
Verification and efficiency were hard for everyone. No harness got above 73% in verification or 81% in efficiency, and their budgets are still first guesses (see Limits).
Some categories hold very few tasks: skills has one, orchestration two, and memory, out-of-the-box and honest reporting three each. One task can move those by a third or more, so take their figures as a rough read.
Cost
Overseer spent the most in total, $1.39 of the run’s $5.78. It also earned the most points, so it was the cheapest per approved point. Most of the difference is output: Overseer writes more per call than the others, which is the first thing What we learned covers.
Spend over the run
Lower is cheaper.
Spend per approved point
Lower is better: cheap and wrong doesn’t look cheap.
Spend is metered at Go’s standard price list: $0.15, $0.003 and $0.60 per million uncached input, cached input and output tokens. The run’s last two hours fell in Go’s 01:00–04:00 UTC double-rate window, so the subscription quota it used is somewhat higher than this. All five harnesses were in that window together.
How to read the numbers
- Approved is the headline. Each task’s graders score a cell out of 100, which is the raw score. Then 15 category judges look at the evidence, and the cell’s points count only if none of them rejects it. A cell that passes its tests but leaves a process running, rewrites git history it shouldn’t, or says it finished when it didn’t gets raw points and no approved ones.
- 95% CI is the confidence interval: how far the approved score could move with a different draw of tasks. Where two intervals overlap, the gap between those harnesses isn’t settled.
- Solved counts cells where every one of the task’s checks passed, before the judges.
- Gates counts cells that failed a check marked as a safety or destructive-action gate. There was one in the whole run, by omp.
- Reliable counts categories, out of 15, as above.
- $/point is total spend divided by approved points, so cheap and wrong doesn’t look cheap. It only compares harnesses within this run, since a run with more seeds spends more per point.
Every task, check and cell result behind these numbers is in the OBench1 repository.