Overseer Docs
← Home

OBench1

what we learned about v5

What follows comes from Overseer’s transcripts, the run’s logs and, since the run ended, its per-category results. It’s the starting list for Overseer’s next version.

Where it spends

Overseer writes too much. It averaged 893 output tokens per answered call, against 607 to 718 for the others, and 1.5M output tokens over the run, against 0.7M to 1.1M. In the pilot, about two-thirds of its output was reasoning. Output is the dearest token, so this is the main reason Overseer spent the most.

Output tokens per answered call

Overseer v5893
omp672
Hermes637
dsh607
pi718

Lower is lighter. Reasoning counts as output.

Output tokens over the run

Overseer v51.51M
omp1.07M
Hermes1.02M
dsh0.85M
pi0.72M

Lower is cheaper: output is the dearest token.

It also checks too much on long tasks. On one long migration it ran about 30 separate checks after the work was done, each one a full round trip on a growing context. The other harnesses stopped at the test suite. One batched check would have done the same job.

Where files could be read side by side, it often read them one at a time, which cost it calls and time. And it was the last of the five to finish 27 of its 81 cells, though usually still well inside the time limit.

Where it slipped

What worked

It held up under faults better than anyone. It was at 100% in resilience, through every injected provider fault and every mid-run change, where the others ranged from 52% to 85%. It was also the only harness at 100% in safety, memory across sessions and orchestration, and it finished every phase on its own in 80 of its 81 cells.

Under rate limits it backed off instead of hammering the shared limit. Its median gap between rate-limited calls was 13 seconds, against 3 or 4 for the others, and it had the fewest 429s of the five.