OBench1
what we learned about v5
What follows comes from Overseer’s transcripts, the run’s logs and, since the run ended, its per-category results. It’s the starting list for Overseer’s next version.
Where it spends
Overseer writes too much. It averaged 893 output tokens per answered call, against 607 to 718 for the others, and 1.5M output tokens over the run, against 0.7M to 1.1M. In the pilot, about two-thirds of its output was reasoning. Output is the dearest token, so this is the main reason Overseer spent the most.
Output tokens per answered call
Lower is lighter. Reasoning counts as output.
Output tokens over the run
Lower is cheaper: output is the dearest token.
It also checks too much on long tasks. On one long migration it ran about 30 separate checks after the work was done, each one a full round trip on a growing context. The other harnesses stopped at the test suite. One batched check would have done the same job.
Where files could be read side by side, it often read them one at a time, which cost it calls and time. And it was the last of the five to finish 27 of its 81 cells, though usually still well inside the time limit.
Where it slipped
- On long tasks that said to leave git alone, it made commits or moved a ref or the stash in 3 of 81 cells. Every other repo rule held.
- Review was its weakest broad category. When it wrote tests from a spec, they caught every broken version but sometimes rejected a correct one too, because they asserted more than the spec said. When it reviewed code, it missed some real bugs and flagged code that only looked wrong.
- Faced with an error message that pointed the wrong way, it got part of the way to the real cause, but not all of it.
- Its own claim check marked “tests: 2 passed” as unbacked, since no tests had run after its last change, and the message went out anyway. A flagged claim should be backed or cut before the answer is sent.
- When its shell edited one of its own scratch files, it told itself “the owner changed” that file and read it again.
- Its memory review runs right after the final reply. It handled the rate limits it hit there cleanly, but on a one-off headless run it’s an extra paid call.
What worked
It held up under faults better than anyone. It was at 100% in resilience, through every injected provider fault and every mid-run change, where the others ranged from 52% to 85%. It was also the only harness at 100% in safety, memory across sessions and orchestration, and it finished every phase on its own in 80 of its 81 cells.
Under rate limits it backed off instead of hammering the shared limit. Its median gap between rate-limited calls was 13 seconds, against 3 or 4 for the others, and it had the fewest 429s of the five.