Overseer Docs
← Home

OBench1

what went wrong

In the order it happened. Some of it is about Overseer, and we’d rather you read it here than find it yourself.

The pilot didn’t measure Overseer

A one-seed pilot ran on 5 November. Overseer came last on approved score (29.6) while costing more than twice the next harness ($1.11, against Hermes’s $0.49). Reading its transcripts, we found why: its shell never worked. macOS refuses to start a second sandbox inside a first (sandbox_apply: Operation not permitted), and every one of Overseer’s 178 shell calls failed. None of the other four sandboxes its own shell, so none hit it.

We voided the pilot’s Overseer row. Its other four rows stand. For the record:

Pilot, one seedApproved95% intervalRawSpend
omp51.633.1–69.889.4$0.35
pi40.522.0–59.081.4$0.30
Hermes36.818.5–55.385.9$0.49
dsh29.614.8–48.155.6$0.32
Overseer v5 void: no working shell29.611.1–48.171.6$1.11

The pilot’s wiring check only checked routing, not whether a tool call works, which is how this got past it. The check now runs each harness’s shell (see Harnesses and fairness).

Overseer drove the Mac’s desktop to get a shell

This is the serious one. With its shell blocked, Overseer used its computer-use tool, in 8 of the pilot’s 27 tasks, to open Terminal and Script Editor on the real desktop, type shell commands into Terminal, run do shell script, delete files with rm -f, quit both apps with killall and start background jobs. All of that ran outside the benchmark’s sandbox, as the machine’s user.

It ran on our own benchmark machine, not a customer’s. It left 7 files in /tmp, which we’ve kept as evidence, and left Terminal and Script Editor open. But on someone’s own Mac, “the sandbox said no, so it opened my Terminal and did it anyway” is exactly what an agent must never do. What we changed:

We stopped a run and started again

The first three-seed run started before we’d found any of this. We stopped it after about an hour ($0.40 spent), fixed the jail and Overseer, and reran every harness from the start. We didn’t give Overseer extra runs: fairness here comes from every arm running the same tasks, at the same hour, the same number of times.

The provider key was visible to every harness

The run passed the OpenCode Go key to each harness through its environment, so every harness could read it. In three cells, pi listed its environment, so the key was written into its transcripts and sent to the provider as part of its context. No other harness printed it. We revoked the key after the run, and it’s replaced with a placeholder in the published transcripts. From the next run each harness will get a dummy key, with the gateway adding the real one, so no harness can see it.

12 cells never got an answer

We found this after the run, while checking the token charts for the full report. In 12 cells, every model call came back as a real 429 and the harness gave up before doing any work: 8 of pi’s cells and 4 of dsh’s. pi stops after four rate-limited tries, dsh after seven, both within about 16 seconds. In those same seconds the other three harnesses mostly got through, because they keep retrying for longer. Overseer, omp and Hermes lost no cells this way. Nine of the 12 fell in the evening’s throttling; the other three came between 02:38 and 03:09 UTC.

The runner counts these as finished cells, not infrastructure errors, so “0 infrastructure errors” is accurate but doesn’t tell the whole story. An empty cell scores 0. As a rough estimate, if those cells had scored each harness’s own average, pi’s approved score would be about 40.7 instead of 36.7, and dsh’s about 45.1 instead of 42.9, just above Hermes (44.3). Overseer and omp would stay first and second.

This is partly the provider and partly the harness. Giving up after 16 seconds of rate limits is a real resilience trait, and the other three showed it can be ridden out. But the benchmark never meant to throw these faults. We haven’t rerun the cells, because a rerun at a different hour wouldn’t be lockstep. Before the next run, the runner will learn to flag any cell with no answered calls.

The provider throttled us for an evening

From about 21:00 to 23:00 UTC on 5 November, the provider rate-limited us in bursts: for a few minutes at a time, 40–60% of calls got a 429. These were real, not injected. Rejected calls cost nothing, but they cost wall time, and unevenly: harnesses that retry fast pile up more 429s. Wall times on the memory tasks that evening say more about the throttling than about the harnesses.

omp266
Hermes173
dsh86
pi78
Overseer v557
Real 429s from the provider in 10-minute bins, one row per harness, all on the same scale (the tallest bar is 55). Fewer is better. The total is on the right.
Real 429s over the whole runRate-limited callsAll calls
omp2661,853
Hermes1731,783
dsh861,486
pi781,078
Overseer v5571,742

Counted at the gateway. Faults a task injects on purpose aren’t in these numbers.