OBench1
limits and the source
Limits of this result
It isn’t a hard benchmark. None of the 27 tasks is extremely demanding for a good harness. We sized OBench1 to fit the team’s current setup: one Mac, one inexpensive model and one subscription plan, so a full run costs a few dollars and finishes overnight. Harder and longer tasks can come as that setup grows.
We wrote both the benchmark and Overseer. We took the 15 categories from what other harnesses are good at. Until this run finished, nobody building Overseer could see which tasks it failed, which guards against tuning it to the test. It can’t rule out tasks that lean our way without anyone meaning them to.
It rewards breadth. Overseer was built to cover all 15 categories, and the score weights them equally. A harness built around a few things it does very well will look worse here than it would on a benchmark of just those things. The per-category view is the fairer way to compare harnesses at what each one cares about.
One model, one provider, one night. Everything here is deepseek-v4.1-flash on OpenCode Go, run on one Mac between 16:29 and 03:21 UTC. It says nothing yet about other models, and the provider’s load that night shaped some of the numbers (see What went wrong).
Three seeds. That’s enough to call a category reliable, but not enough to make the intervals narrow. Where two intervals overlap, the order isn’t settled.
Budgets are first guesses. The token, cache and time budgets the efficiency and speed judges use are the same for everyone, but we haven’t calibrated them from real data yet. Some may be too tight or too loose (see How it works).
The source is open
All of OBench1 is on GitHub at sirrayi/obench1, under MIT or Apache-2.0. That’s the runner, the harness adapters, all 27 tasks with their checks and reference solutions, every cell’s scores and judge votes, and every transcript from this run, the pilot and the run we stopped. If you want to check a number on these pages, what it’s built from is there.
We changed one thing before publishing it. The provider key pi printed (see What went wrong) now reads as a placeholder. The key had already been revoked.
What that means for later scores
This run was measured while the tasks were private, and nobody building Overseer could see which ones it failed. Since the run, we’ve lifted that rule so we can fix what it found, and now anyone can read the tasks, us included. So any harness can be tuned to them, and later OBench1 scores are easier to game than this one. When we publish one, we’ll say so next to the number.