OBench1
harnesses and fairness
The harnesses
Each ran as it ships, apart from the settings under Same conditions and the differences we list under What still differs.
| Harness | Build | Provider entry | What it is |
|---|---|---|---|
| Overseer v5 | v5.0.0-rc2, pinned binarysha256 d93bbf46…babab68 · commit 910f9881 | opencode | Ours |
| omp | 18.13.3 | opencode-go (built in) | oh-my-pi, a pi-based coding agent |
| dsh | DeepSeek Harness 0.1.1-rc.2 | deepseek-official (base URL moved) | DeepSeek’s own desktop harness, run headless |
| pi | 1.5.0 | opencode-go (built in) | A small, fast coding agent |
| Hermes | Hermes Agent 0.22.5 | opencode-go (built in) | A general agent with tools, skills and memory |
The v5 binary is pinned: built once, its hash recorded, never changed. The same commit is tagged v5.0.0-rc2 on GitHub.
Same conditions for everyone
One model, one gateway
Every harness used the OpenCode Go subscription with deepseek-v4.1-flash: 1M tokens of context, 384K of output. Each cell gets its own gateway on this machine, and every model call goes through it. Whatever a harness asks for, the call that reaches the provider carries the same four settings. The gateway logs every request it had to change.
| Setting | Sent on every call | Why |
|---|---|---|
| model | deepseek-v4.1-flash | one model for everyone |
| thinking | on | some harnesses send no thinking field, or turn it off for side calls |
| reasoning effort | max | the provider’s top level |
| output ceiling | 384,000 tokens | the model’s own limit, so nobody’s thinking gets cut short |
Raising side calls to max effort costs some harnesses a little more than they’d spend on their own. One rule for every call is easier to check than a rule per harness.
Same context, same limits, same moment
- Context: all five plan for a 1M-token window for this model, so none compacts its history earlier than another for lack of information.
- Limits: the task’s wall cap is the only limit. Overseer ships with a 100-step and $5 budget, which the others don’t have, so we lifted both. Approvals are off for every arm.
- Lockstep: the five arms of a task and seed start together, in a launch order shuffled per task and seed, at the same process priority. They share the same provider load, machine load and hour.
- Faults: injected provider faults follow one plan per task and seed, keyed to the call number, the same for every arm.
The same jail
- Every cell runs inside a macOS sandbox. It can reach loopback only, so the gateway is the only way out.
- It can’t launch apps, send Apple Events, reach the window server or Accessibility, or run any copy of a computer-use driver. Computer use is off for every arm in this benchmark (it isn’t off in the products).
- Every cell gets a fresh home directory. No harness sees our own configs, memories, skills or sessions.
Proved before money was spent
Before the paid run, every harness ran against a fake provider on this machine, with a dummy key. The check proves each one routes through its gateway with the forced settings, and that its own shell tool works: it writes a file, then tries to launch an app, send an Apple Event and find the computer-use driver. All five wrote the file, and all three escapes were refused for all five.
What still differs
On purpose, because it’s what makes a harness what it is:
- Tools, as counted in the pilot’s wiring check: Overseer v5 offers 10, pi 4, omp 11, Hermes 19, dsh 25.
- Side calls: omp makes a short title call, dsh a small extra call, Hermes probes the endpoint for its context size, Overseer runs a memory review after its final answer.
- Recovery: each handles errors its own way. During an evening of provider throttling (see What went wrong), the median gap between one harness’s rate-limited calls was about 3 seconds for omp, dsh and Hermes, 4 for pi and 13 for Overseer.
- Prompts, temperature, memory and everything else.
What each harness got that the others didn’t
Small changes we made so each harness could run here at all. None touches grading.
| Harness | What we changed, and why |
|---|---|
| Overseer v5 |
--no-sandbox: macOS won’t start Overseer’s own shell sandbox inside the cell’s sandbox (see What went wrong). The jail is every arm’s sandbox, and no other arm sandboxes its shell.Step and cost caps lifted (see Same conditions, above). --probe off and a copied capability file: Overseer caches a check of each endpoint’s capabilities for 30 days, and a new gateway port every cell would make it re-check every cell. That saves it a little. |
| dsh | The gateway copies dsh’s session id into the header OpenCode Go routes cache hits on. pi and omp send that header themselves; without the copy, dsh’s calls wouldn’t get the same cache routing. |
| Hermes | Its ~600 MB of tool installs and cache are built once beforehand and shared between cells, so installing never counts against its wall time. It holds no memory or sessions. |
| pi, omp | Throwaway agent directories, copied per cell with only the base URL changed. |