redstone-bench

Site: https://redstone-bench.adamholter.chatgpt.site

Can a model design redstone that works, and how small can it make it? Every answer is built and run in a real vanilla Minecraft Java 26.3 server, tick by tick. Nothing is judged by a model.

Tasks (16)

family	tasks	what must happen
door	door_2x2, door_3x3, door_4x4	one lever opens and closes a piston door a player can walk through
mechanical	mech_extender2, mech_extender3	one lever pushes a block 2 or 3 cells up out of a floor and pulls it back
logic	logic_not, logic_xor, logic_full_adder, logic_adder2, logic_seg7	levers in, lamps out, every row of a truth table, in several orders
memory	mem_tff, mem_rs_latch, mem_counter2	lamps must remember: toggle, set/reset, count lever presses
timing	time_clock20, time_clock120, time_pulse30	a lamp blinks with an exact period, or lights for an exact number of ticks

Score for every task: the volume of the bounding box around everything the build ever occupies, lower is better. Ties go to fewer ticks (door: close plus open; circuit: delay from lever change to correct lamps). Door and extender levers are not counted (the piston-door community's rule); circuit levers and lamps are.

Task files are in tasks/. The non-door ones are generated by bench/make_tasks.py.

How a run works

sh serve.sh                                    # the judge: vanilla 26.3 server, RCON 127.0.0.1:25592 (setup: .claude/skills/contribute)
python3 bench/run.py --task logic_xor --model terra --effort medium     # one run (40 tries, 60 minutes)
python3 bench/batch.py                         # the whole model x task matrix; safe to kill and restart
python3 bench/batch.py --audit                 # what really finished, read from disk
python3 bench/export_site.py                   # replay builds into site/data (3D states, animations, time series)
(cd site && node render_all.mjs)               # thumbnails; the site is served on http://127.0.0.1:7461

The model runs as an agent in its own CLI (Codex, on the subscription) in a scratch folder with no network. It reads TASK.md, writes a block list as JSON, and calls python3 verify.py submission.json. That hands the file to the runner, which places the blocks in the server, flips the levers, steps the game one tick at a time, and returns pass/fail with the reason and the world state. The best valid build across its verifier calls is its score.

What each run saves (runs/<model>-<effort>/<task>/)

• result.json: best build, wall time, time to first valid and to best, token usage, CLI version, git commit, the model's last message, any shell command that touched a path outside the workspace, and attempt_log: every verifier call with its wall-clock second, the shell step it happened on, and the outcome (the time series)
• timeline.json: every CLI event with the second it arrived (each shell command, each message)
• TASK.md, task.json: exactly what the model was given
• attempts/NNN.json + NNN.result.json: every submission and the verifier's full answer
• transcript.jsonl: the CLI's full event stream
• workspace/: the scripts and notes the model wrote
• error.json instead of result.json when the CLI died without finishing a turn (quota, auth); the batch retries those

runs/_v1/ holds the first round (older prompt, 15 to 40 tries); runs/_void_* are rounds thrown out for harness bugs.

Site (site/)

Static page: leaderboard, task gallery, score table, and an explorer per task with a 3D view (deepslate, MIT, with Minecraft's own block models and textures from misode/mcmeta), a tick-by-tick animation recorded from the server, and each run's score over time. The block textures are Mojang's and are not covered by this repo's license.

Verifier notes

• bench/verify.py <task.json> <submission.json> [--capture] runs one check by hand. bench/render.py draws a build.
• Blocks are placed with /setblock in list order (a lamp placed after its lit torch stays dark until a neighbour changes; the task text says so). "placement": "raw" places exact states with no updates (for machines described mid-state).
• A lever flip is /setblock plus a block update around the block it is mounted on, which is what a real flip does. Plain /setblock alone skips that second update, so a lever on a block did not power things next to that block. Runs from before this fix have a rescored.json with their verdicts under the current verifier.
• Each build starts with a flush of ticks left over from the previous one; results repeat exactly.
• baselines/ are hand-made working builds (baselines/scores.json has their sizes). seeds/door_3x3.json is the 56-volume door from vjeux/redstonepedia.

Not done

• No 4x4 seed: the 99-volume human record has no public file.
• No position test (redstonepedia checks 32 world offsets) and no random-tick test.
• logic_seg7, logic_adder2, mem_counter2, time_clock120, mech_extender3 have no hand-made baseline; a task counts as proven solvable only once some run or baseline passes it (see docs/RESULTS.md).

Hosting

bash site-host/sync.sh builds site-host/dist for a static host: the page, data, renders and icons, keeping only model settings with every task run. It publishes nothing. site/icons.mjs makes the icons from favicon.svg; site/og.html is the share image source.

A run that loses the judge (server down for over four minutes) or runs out of disk is thrown out and retried by the batch.

Public site: https://redstone-bench.adamholter.chatgpt.site (ChatGPT Sites project appgprj_6af37b7255ac8191a1df85b1eb863aba, public). Redeploy: export, render, bash site-host/sync.sh, then a Codex agent (prompt: site-host/deploy-prompt.txt, run with codex exec -m gpt-6-luna -C site-host -) with the Sites plugin republishes site-host/dist to that same project (site-host is its own git checkout with .openai/hosting.json).

Contributing

See CONTRIBUTING.md and the agent skill at .claude/skills/contribute/SKILL.md.

License

MIT, see LICENSE. Excludes Minecraft's textures and block models (Mojang) and third-party files under site/vendor and third_party, which keep their own licenses.