Franka · Pick and place next to¶
Agent runs
Each mode of this task runs once per run. Est. cost is tokens at list price (data/prices.yml), never a bill; see the MolmoSpaces runs for every task of a run.
Codex CLI 0.157–0.159 + GPT-6 Luna, reasoning medium (OpenRouter)default runclosed
| Trial | Started | Agent time | Model requests | Tokens in / out | Est. cost | Grade |
|---|---|---|---|---|---|---|
| U✓ 16m | 10-03 21:24 | 16m | — | 2.9M / 22k | $0.047 | deterministic 1, grader_error 0, n_actions 165, replay_success 1, video_rendered 1 |
| L✗ 11m | 10-03 19:18 | 11m | — | 3.0M / 36k | $0.055 | deterministic 1, grader_error 0, live_success 0, n_actions 454, replay_success 0, video_rendered 1 |
Task instruction (upstream)
Pick up the lepidopteran and place it next to the brown rectangular bread.
What MolmoSpaces states about this task
| Success criteria | 1. the horizontal gap between the two objects' axis-aligned bounding boxes is between 0 and 6 cm; both rest on the same supporting body; and the reference object has moved at most 15 cm and rotated at most 60 degrees from its start pose 2. judged at the end of the episode: the trajectory's last row (privileged), done (standard) or 455 control steps (the benchmark's 30 s horizon), whichever comes first3. unlimited (privileged): both fresh-process replays of the handed-in trajectory end in the same state, and the check holds on it 4. limited (standard): the one episode (no reset) is recorded by the service and replays to the same state; the check holds on it live and in the replay |
| Family | pick-and-place-next-to |
| Robot | Franka FR3 with a Robotiq 2F-85 gripper on a fixed base (the DROID setup) |
| Category | Franka |
| Instance | molmospaces-bench-v2/20260415, package procthor-objaverse/FrankaPickandPlaceNextToHardBench/FrankaPickandPlaceNextToHardBench_20260305_json_benchmark, episode 5 |
| Deliverable | /app/output/trajectory.npz with actions: float64 (T, 8), one row per control step, the targets of upstream's joint-position controllers |
| Reference Solution | None is shipped. Upstream's scripted experts (molmo_spaces/policy/solvers) are reference solutions and are not in the image; neither are grasp files nor the public MolmoBot trajectories (agent egress: the model APIs only). |
| Limited Mode | Standard-mode twin of molmospaces-pick-and-place-next-to-i00-privileged (the same frozen MolmoSpaces episode): Pick up the lepidopteran and place it next to the brown rectangular bread. The agent gets only the eai-standard/2.2 client (docs/STANDARD_MODE_2_2.md); the simulator runs in the sim sidecar (environment/docker-compose.yaml), which owns the episode, serves cameras, proprioception and upstream's kinematic model, and records every executed row. The collect hook (environment/sim/finalize.sh) ends the episode, lets the service exit, replays the trajectory in two fresh processes and writes final.json; the verifier grades those artifacts in a separate sandbox (tests/Dockerfile). |
| Oracle | none — no reference solution (MolmoSpaces' planners and grasp files are not shipped); positive example graded 1 by the separate verifier: molmospaces-pick-and-place-next__3W4hmcz (run codex-gpt6_luna-medium, batch molmospaces-luna-1003; https://embodied-agent-interface-v2-internal.github.io/runs/molmospaces/codex-gpt6_luna-medium-openrouter/pick_and_place_next_to/unlimited/); human review in PR #47 |
| Base Image | ghcr.io/mll-lab-nu/eai-molmospaces:0.2.0 |
| Agent Budget | 3600 s of wall clock per mode |
| Task Dirs | molmospaces-pick-and-place-next-to-i00-privileged, molmospaces-pick-and-place-next-to-i00-standard |
From https://github.com/allenai/molmospaces @ molmo-spaces 0.2.9 (benchmark molmospaces-bench-v2/20260415), as defined in our task definitions @ 030f55607.
Tags¶
Why this task is interesting¶
A flat moth, about 2 cm thick, has to end up beside a loaf of bread: on the same surface, within 6 cm. The flat object is hard to grasp from above, and "next to" is judged on bounding boxes, so a few centimetres decide.
Capability notes¶
Not yet written.
Oracle demo review¶
No demo.
Discussion¶
The family's own difficulty, the 6 cm placement, decides it. Upstream's check also accepts the object while it is still held just above the counter; kept as upstream defines it. (@williamzhangNU)