Benchmark task reference¶

Documentation for the benchmarks behind our robotic coding-agent work. For every task: what it asks for, its oracle demonstration, the capabilities it requires, and whether we are keeping it; and how our coding agents do on it.

Benchmarks¶

Agent runs¶

Codex + GPT-6 Luna, reasoning xhigh (ChatGPT login) · Codex CLI 0.157–0.159 + GPT-6 Luna, reasoning medium (OpenRouter), on every benchmark: success among graded trials, the mean score (partial credit, a success counts as 1.0), and what the model use would cost at list price. All runs, tasks and trial logs →

^* RoboPaint: the plain mean of the grader's final reward over the graded trials, a success at its own value. Its metric depends on the task family: IoU for lettering, kaishu and xingshu; 1 - mean ΔE / 20 for acrylic and oil.

How to contribute¶

Everything you would change lives in one folder, state/. Task pages under docs/ hold only upstream metadata and prose, so reviewing a day of someone's triage is git diff state/ — not a hundred file diffs.

Start here¶

git clone <this-repo> && cd <this-repo>

make install     # one-time: creates .venv, installs the pinned toolchain
make demos       # one-time: downloads 134 oracle demos + posters (~1.2 GB)
make edit        # start the site WITH editing enabled

make edit prints the URL it picked (the port is chosen automatically, so a colleague's copy never collides with yours). Open it, and you are on the task list.

Then:

1. Set the speed to 2× or 3× once — it is remembered across pages.
2. Filter Status to pending and pick a short task.
3. Click Edit on that row, set owner to your handle, Save. Now nobody duplicates your work.
4. Watch the demo in the row, then set status, difficulty and labels.
5. Optionally open the task page and write down why — that prose is the part a reviewer actually reads.
6. make check, then commit. Your whole session is a diff of state/.