OBench1
how it works
What it measures
OBench1 tests the harness, not the model. Every arm gets the same model, the same settings and the same tasks at the same moment, so the only thing left to differ is the software around the model: its tools, prompts, memory, recovery and judgement.
The 15 categories come from what the best harnesses each do well. Overseer was built to cover all of them, while each of the others was built around its own handful. That matters for how far this result reaches.
| Category | What it asks |
|---|---|
| Long-horizon execution | Can it keep going for hours and hundreds of steps without losing the thread? |
| Context management | Does it find the one line that matters in a flood, and stay inside its context? |
| Memory across sessions | Does it remember corrections, and refuse a poisoned note? |
| Orchestration and subagents | Can it split work, run it in parallel and put it back together? |
| Tool use | Does it edit precisely, rename across a codebase, and cope with odd file systems? |
| Skills and extensibility | Does it use what it’s given to extend itself? |
| Token, cache and prompt efficiency | How much does it spend, and how much of that is served from cache? |
| Speed | Does it finish inside half the task’s wall cap? |
| Instruction following | Does it keep the house rules: reply length, sign-off, no emoji, no TODOs? |
| Policy and repo rules | Does it respect the repo’s rules, including nested ones, and leave git alone unless told? |
| Out-of-the-box problem solving | Can it work out a lying error, a lost fixture, a slow oracle? |
| Verification, critique and review | Does it check its own work, and not trust a fix because someone said it works? |
| Safety and trust | Does it keep secrets, avoid destructive actions and leave nothing running? |
| Resilience | Does it ride out provider faults and mid-run changes? |
| Honest reporting | Does it say what it didn’t finish, instead of claiming it’s all done? |
The tasks
27 tasks in five camps of rising difficulty, each with a wall-clock cap from 5 to 90 minutes. Some change under the harness mid-run: a spec change, a teammate’s commit, a flood of log lines, or a storm of provider faults injected at the gateway. The camps and some task names are below. Every task, with its checks and reference solution, is in the repository.
| Camp | Tasks | Examples |
|---|---|---|
| 1 · warm-up | 2 | eff-trivial, eff-parallel-reads |
| 2 | 9 | if-constraint-gauntlet, lh-honest-blockers, sec-secret-hygiene, ver-hidden-spec |
| 3 | 11 | chaos-provider-storm, ctx-log-needle, mem-poisoned-note, pol-git-safety |
| 4 | 3 | ctx-eighteen-tickets, orch-dag-build, orch-spec-audit-24 |
| 5 · the long climbs | 2 | lh-money-migration, tool-rename-haystack (each can run to millions of tokens) |
How it’s graded
- The graders are each task’s own checks, run on the system Python outside the harness’s reach. They score what’s in the workspace, the git history and the replies.
- The judges are code, not a model. No AI grades anything. Each of the 15 category judges reads the checks tagged with its category plus signals every cell has: budgets, whether each phase finished on its own, git refs, leftover processes, editor debris, overclaiming in the final reply, and any peek at the benchmark itself. A judge with nothing to go on abstains.
- The judges never see which harness made the cell. We checked: no grader or judge reads a harness’s name.
- The graders are proven first. Before the first paid run, each task’s reference solution must score 100 (27 of 27 did) and doing nothing must score 0 (27 of 27 did).
- Budgets are first guesses. The token, context, cache and speed budgets the efficiency and speed judges use haven’t been calibrated from field data yet. They’re the same for every harness, but they may be too tight or too loose.