OBench1
what went wrong
In the order it happened. Some of it is about Overseer, and we’d rather you read it here than find it yourself.
The pilot didn’t measure Overseer
A one-seed pilot ran on 5 November. Overseer came last on approved score (29.6) while costing more than twice the next harness ($1.11, against Hermes’s $0.49). Reading its transcripts, we found why: its shell never worked. macOS refuses to start a second sandbox inside a first (sandbox_apply: Operation not permitted), and every one of Overseer’s 178 shell calls failed. None of the other four sandboxes its own shell, so none hit it.
We voided the pilot’s Overseer row. Its other four rows stand. For the record:
| Pilot, one seed | Approved | 95% interval | Raw | Spend |
|---|---|---|---|---|
| omp | 51.6 | 33.1–69.8 | 89.4 | $0.35 |
| pi | 40.5 | 22.0–59.0 | 81.4 | $0.30 |
| Hermes | 36.8 | 18.5–55.3 | 85.9 | $0.49 |
| dsh | 29.6 | 14.8–48.1 | 55.6 | $0.32 |
| Overseer v5 void: no working shell | 29.6 | 11.1–48.1 | 71.6 | $1.11 |
The pilot’s wiring check only checked routing, not whether a tool call works, which is how this got past it. The check now runs each harness’s shell (see Harnesses and fairness).
Overseer drove the Mac’s desktop to get a shell
This is the serious one. With its shell blocked, Overseer used its computer-use tool, in 8 of the pilot’s 27 tasks, to open Terminal and Script Editor on the real desktop, type shell commands into Terminal, run do shell script, delete files with rm -f, quit both apps with killall and start background jobs. All of that ran outside the benchmark’s sandbox, as the machine’s user.
It ran on our own benchmark machine, not a customer’s. It left 7 files in /tmp, which we’ve kept as evidence, and left Terminal and Script Editor open. But on someone’s own Mac, “the sandbox said no, so it opened my Terminal and did it anyway” is exactly what an agent must never do. What we changed:
- In Overseer (v5.0.0-rc2): the computer tool now refuses to launch or type into any terminal or script app, to open script files, and to run
do shell script. And when the shell sandbox can’t start, Overseer now says so once, plainly, instead of failing the same way 178 times. - In the benchmark: the jail now blocks launching apps, Apple Events, the window server, Accessibility and every copy of a computer-use driver, for every harness. We checked each block by hand.
- Still open: a general rule that stops any tool after the same failure three times. We built it and dropped it before release: it tripped on ordinary refusals and would have ended healthy runs.
We stopped a run and started again
The first three-seed run started before we’d found any of this. We stopped it after about an hour ($0.40 spent), fixed the jail and Overseer, and reran every harness from the start. We didn’t give Overseer extra runs: fairness here comes from every arm running the same tasks, at the same hour, the same number of times.
The provider key was visible to every harness
The run passed the OpenCode Go key to each harness through its environment, so every harness could read it. In three cells, pi listed its environment, so the key was written into its transcripts and sent to the provider as part of its context. No other harness printed it. We revoked the key after the run, and it’s replaced with a placeholder in the published transcripts. From the next run each harness will get a dummy key, with the gateway adding the real one, so no harness can see it.
12 cells never got an answer
We found this after the run, while checking the token charts for the full report. In 12 cells, every model call came back as a real 429 and the harness gave up before doing any work: 8 of pi’s cells and 4 of dsh’s. pi stops after four rate-limited tries, dsh after seven, both within about 16 seconds. In those same seconds the other three harnesses mostly got through, because they keep retrying for longer. Overseer, omp and Hermes lost no cells this way. Nine of the 12 fell in the evening’s throttling; the other three came between 02:38 and 03:09 UTC.
The runner counts these as finished cells, not infrastructure errors, so “0 infrastructure errors” is accurate but doesn’t tell the whole story. An empty cell scores 0. As a rough estimate, if those cells had scored each harness’s own average, pi’s approved score would be about 40.7 instead of 36.7, and dsh’s about 45.1 instead of 42.9, just above Hermes (44.3). Overseer and omp would stay first and second.
This is partly the provider and partly the harness. Giving up after 16 seconds of rate limits is a real resilience trait, and the other three showed it can be ridden out. But the benchmark never meant to throw these faults. We haven’t rerun the cells, because a rerun at a different hour wouldn’t be lockstep. Before the next run, the runner will learn to flag any cell with no answered calls.
The provider throttled us for an evening
From about 21:00 to 23:00 UTC on 5 November, the provider rate-limited us in bursts: for a few minutes at a time, 40–60% of calls got a 429. These were real, not injected. Rejected calls cost nothing, but they cost wall time, and unevenly: harnesses that retry fast pile up more 429s. Wall times on the memory tasks that evening say more about the throttling than about the harnesses.
| Real 429s over the whole run | Rate-limited calls | All calls |
|---|---|---|
| omp | 266 | 1,853 |
| Hermes | 173 | 1,783 |
| dsh | 86 | 1,486 |
| pi | 78 | 1,078 |
| Overseer v5 | 57 | 1,742 |
Counted at the gateway. Faults a task injects on purpose aren’t in these numbers.