Overseer Docs
← Home

OBench1

harnesses and fairness

The harnesses

Each ran as it ships, apart from the settings under Same conditions and the differences we list under What still differs.

HarnessBuildProvider entryWhat it is
Overseer v5v5.0.0-rc2, pinned binarysha256 d93bbf46…babab68 · commit 910f9881opencodeOurs
omp18.13.3opencode-go (built in)oh-my-pi, a pi-based coding agent
dshDeepSeek Harness 0.1.1-rc.2deepseek-official (base URL moved)DeepSeek’s own desktop harness, run headless
pi1.5.0opencode-go (built in)A small, fast coding agent
HermesHermes Agent 0.22.5opencode-go (built in)A general agent with tools, skills and memory

The v5 binary is pinned: built once, its hash recorded, never changed. The same commit is tagged v5.0.0-rc2 on GitHub.

Same conditions for everyone

One model, one gateway

Every harness used the OpenCode Go subscription with deepseek-v4.1-flash: 1M tokens of context, 384K of output. Each cell gets its own gateway on this machine, and every model call goes through it. Whatever a harness asks for, the call that reaches the provider carries the same four settings. The gateway logs every request it had to change.

SettingSent on every callWhy
modeldeepseek-v4.1-flashone model for everyone
thinkingonsome harnesses send no thinking field, or turn it off for side calls
reasoning effortmaxthe provider’s top level
output ceiling384,000 tokensthe model’s own limit, so nobody’s thinking gets cut short

Raising side calls to max effort costs some harnesses a little more than they’d spend on their own. One rule for every call is easier to check than a rule per harness.

Same context, same limits, same moment

The same jail

Proved before money was spent

Before the paid run, every harness ran against a fake provider on this machine, with a dummy key. The check proves each one routes through its gateway with the forced settings, and that its own shell tool works: it writes a file, then tries to launch an app, send an Apple Event and find the computer-use driver. All five wrote the file, and all three escapes were refused for all five.

What still differs

On purpose, because it’s what makes a harness what it is:

What each harness got that the others didn’t

Small changes we made so each harness could run here at all. None touches grading.

HarnessWhat we changed, and why
Overseer v5 --no-sandbox: macOS won’t start Overseer’s own shell sandbox inside the cell’s sandbox (see What went wrong). The jail is every arm’s sandbox, and no other arm sandboxes its shell.
Step and cost caps lifted (see Same conditions, above).
--probe off and a copied capability file: Overseer caches a check of each endpoint’s capabilities for 30 days, and a new gateway port every cell would make it re-check every cell. That saves it a little.
dshThe gateway copies dsh’s session id into the header OpenCode Go routes cache hits on. pi and omp send that header themselves; without the copy, dsh’s calls wouldn’t get the same cache routing.
HermesIts ~600 MB of tool installs and cache are built once beforehand and shared between cells, so installing never counts against its wall time. It holds no memory or sessions.
pi, ompThrowaway agent directories, copied per cell with only the base URL changed.