Overseer Docs
← Home

OBench1

results

Suite totals

Five harnesses ran 27 tasks three times each, which makes 405 cells. Every cell used deepseek-v4.1-flash through the same metered gateway. Overseer v5 came first on approved score, with omp second.

Overseer v568.8
omp54.1
Hermes44.3
dsh42.9
pi36.7
Approved score out of 100, higher is better. The thin line is the 95% interval and the ring is the raw score, before the judges.
HarnessApproved95% CIRawSolvedGatesReliable$/pointSpend
Overseer v568.854.0–82.795.261/81011/15$0.0007$1.39
omp54.137.0–69.887.250/8116/15$0.0008$1.21
Hermes44.329.6–60.185.549/8105/15$0.0011$1.34
dsh42.928.1–57.874.042/8104/15$0.0009$1.00
pi36.721.7–51.571.437/8105/15$0.0008$0.84

Overseer’s interval (54.0–82.7) and omp’s (37.0–69.8) overlap, so read the order of the top two as likely rather than settled. Hermes, dsh and pi sit close together, and their intervals overlap almost completely.

pi’s and dsh’s totals include 8 and 4 cells that never got an answer from the provider. Every call in those cells was rate-limited, and the harness gave up within about 16 seconds. Without them dsh would probably be third rather than fourth. What went wrong has the details.

The early preview

We posted a preview before the run finished, covering 22 of the 27 tasks: Overseer 72.2, omp 56.2, Hermes 43.3, dsh 41.7 and pi 34.4. With all 27 in, Overseer’s lead over omp narrowed from 16.0 points to 14.7, and pi’s cost per approved point fell from $0.0010 to $0.0008.

By category

A harness is reliable in a category when it was approved on every seed in at least 90% of that category’s tasks. Overseer was reliable in 11 of the 15 categories and the others in 4 to 6. Nobody reached the summit, which needs all 15 and no gate breach anywhere.

CategoryOverseer v5ompHermesdshpi
Long-horizon execution27 tasks96%85%67%63%52%
Context management27 tasks100%100%100%96%100%
Memory across sessions3 tasks100%67%33%33%0%
Orchestration and subagents2 tasks100%50%50%0%0%
Tool use27 tasks100%100%100%96%96%
Skills and extensibility1 task100%100%100%100%100%
Token, cache and prompt efficiency27 tasks81%81%78%81%70%
Speed27 tasks93%93%89%85%93%
Instruction following27 tasks85%81%59%52%48%
Policy and repo rules22 tasks91%95%91%68%73%
Out-of-the-box problem solving3 tasks67%67%67%33%0%
Verification, critique and review11 tasks73%55%55%45%27%
Safety and trust27 tasks100%81%70%96%96%
Resilience27 tasks100%85%85%63%52%
Honest reporting3 tasks100%100%100%67%33%

Each figure is the share of the category’s judged tasks where the harness was approved on all three seeds; higher is better. The darker the cell, the higher the share, and bold marks a reliable category (90% or more).

Overseer was the only harness at 100% in memory, orchestration, safety and resilience. Its weakest categories were out-of-the-box problem solving (67%), verification and review (73%), efficiency (81%) and instruction following (85%). Policy is the one category where it trails: omp has 95% to its 91%.

Verification and efficiency were hard for everyone. No harness got above 73% in verification or 81% in efficiency, and their budgets are still first guesses (see Limits).

Some categories hold very few tasks: skills has one, orchestration two, and memory, out-of-the-box and honest reporting three each. One task can move those by a third or more, so take their figures as a rough read.

Cost

Overseer spent the most in total, $1.39 of the run’s $5.78. It also earned the most points, so it was the cheapest per approved point. Most of the difference is output: Overseer writes more per call than the others, which is the first thing What we learned covers.

Spend over the run

Overseer v5$1.39
omp$1.21
Hermes$1.34
dsh$1.00
pi$0.84

Lower is cheaper.

Spend per approved point

Overseer v5$0.0007
omp$0.0008
Hermes$0.0011
dsh$0.0009
pi$0.0008

Lower is better: cheap and wrong doesn’t look cheap.

Spend is metered at Go’s standard price list: $0.15, $0.003 and $0.60 per million uncached input, cached input and output tokens. The run’s last two hours fell in Go’s 01:00–04:00 UTC double-rate window, so the subscription quota it used is somewhat higher than this. All five harnesses were in that window together.

How to read the numbers

Every task, check and cell result behind these numbers is in the OBench1 repository.