Overseer Docs
← Home

OBench1

how it works

What it measures

OBench1 tests the harness, not the model. Every arm gets the same model, the same settings and the same tasks at the same moment, so the only thing left to differ is the software around the model: its tools, prompts, memory, recovery and judgement.

The 15 categories come from what the best harnesses each do well. Overseer was built to cover all of them, while each of the others was built around its own handful. That matters for how far this result reaches.

CategoryWhat it asks
Long-horizon executionCan it keep going for hours and hundreds of steps without losing the thread?
Context managementDoes it find the one line that matters in a flood, and stay inside its context?
Memory across sessionsDoes it remember corrections, and refuse a poisoned note?
Orchestration and subagentsCan it split work, run it in parallel and put it back together?
Tool useDoes it edit precisely, rename across a codebase, and cope with odd file systems?
Skills and extensibilityDoes it use what it’s given to extend itself?
Token, cache and prompt efficiencyHow much does it spend, and how much of that is served from cache?
SpeedDoes it finish inside half the task’s wall cap?
Instruction followingDoes it keep the house rules: reply length, sign-off, no emoji, no TODOs?
Policy and repo rulesDoes it respect the repo’s rules, including nested ones, and leave git alone unless told?
Out-of-the-box problem solvingCan it work out a lying error, a lost fixture, a slow oracle?
Verification, critique and reviewDoes it check its own work, and not trust a fix because someone said it works?
Safety and trustDoes it keep secrets, avoid destructive actions and leave nothing running?
ResilienceDoes it ride out provider faults and mid-run changes?
Honest reportingDoes it say what it didn’t finish, instead of claiming it’s all done?

The tasks

27 tasks in five camps of rising difficulty, each with a wall-clock cap from 5 to 90 minutes. Some change under the harness mid-run: a spec change, a teammate’s commit, a flood of log lines, or a storm of provider faults injected at the gateway. The camps and some task names are below. Every task, with its checks and reference solution, is in the repository.

CampTasksExamples
1 · warm-up2eff-trivial, eff-parallel-reads
29if-constraint-gauntlet, lh-honest-blockers, sec-secret-hygiene, ver-hidden-spec
311chaos-provider-storm, ctx-log-needle, mem-poisoned-note, pol-git-safety
43ctx-eighteen-tickets, orch-dag-build, orch-spec-audit-24
5 · the long climbs2lh-money-migration, tool-rename-haystack (each can run to millions of tokens)

How it’s graded