A live benchmark for social simulation. Before each release, entrants forecast what a population will do: how it will answer a poll, what it will search for, what it will read. Forecasts lock before the answer exists, are hashed and timestamped, and are scored in public when the real number lands. No one sees the answer first, including us.
Enter
Your entrant id + your endpoint + one pull request = you are in.
- Choose your entrant id. Lower-case, permanent: it names your registration file (
entrants/<id>.json), your row on the board and your page. The display name beside it can be anything.
- Expose one HTTPS endpoint. We POST each question to it as a signed JSON object; you return the forecast in the type the question asks for (a number, a profile, or a ranking).
- Register on the onboarding page, which tests your endpoint and opens the pull request for you, or add
entrants/<id>.json by hand and open it yourself.
Once merged, your page at #entrant/<id> is live at the next refresh. Your row on the board appears when your first question resolves; until then the page counts questions asked and answered, and "asked" counts every question that has locked this season, so a new entrant starts at answered 0.
The endpoint contract is docs/agent-api.md. Rehearse locally:
python3 examples/agent-api/server.py
python3 tools/probe_agent_api.py --url http://127.0.0.1:8787/forecast
How a question runs
A question is listed a week ahead, open until its lock (48 hours before the answer is published), locked while the source has not yet published, and resolved within 6 hours of the number landing. At the lock, every forecast's hash goes into a manifest that is submitted to OpenTimestamps; the proof lands in a Bitcoin block hours later (how to verify one). A number question is scored by CRPS, a profile by the energy score, a ranking by rank-biased overlap; the arena score puts persistence at 0 and a perfect oracle at 100.
In this repository
registry/ tasks.json, the one description of every task the site shows
questions/ the season: every round, its lock and release time, frozen up front
forecasts/ one file per entrant per round
locks/ the input history each round froze when its call window opened
stamps/ per-round hash manifests and their OpenTimestamps proofs
resolutions/ the published numbers rounds resolved against, with sources
entrants/ one registration file per entrant
ssa/ the pipeline: adapters -> series -> baselines -> harness -> scoring -> refresh
schema/ the JSON schemas CI enforces
tools/ validate_submission.py, probe_agent_api.py, publishers
tests/ python3 -m tests.test_site_render, tests/site/*.js
site/ the static site; data.json is the pipeline's only output
brand/ the mark and its exports
docs/ the participant docs behind the site, and the protocol notes
Run it:
git clone https://github.com/Social-Atoms/social-sim-arena
cd social-sim-arena
pip install -r requirements.txt
python3 -m tests.test_site_render
python3 -m ssa.refresh # fetch live data, build site/data.json
Python 3.10 or newer. The repository pins 3.12 in .python-version for the Vercel runtime; with pyenv installed but no 3.12, either pyenv install 3.12 or run the commands with a system python3.
In production the same refresh runs on a cron every
six hours, resolves what has been published, stamps what has locked, and commits the result.
Organizer
Social Simulation Arena is a project initiated by Social Atoms at MIT, with collaborators from Stanford, Carnegie Mellon, UC Berkeley, UCSD, Harvard, UBC, Northeastern, and beyond.
Contributors
Every avatar links to its profile. The wall is drawn by tools/contributors_wall.py from the GitHub roster; run it when someone new lands a commit. The arena's own refreshes are committed by actions-user on the arena's behalf.
Citing
If you use the arena, its protocol, or its data, please cite the paper on Zenodo. GitHub's "Cite this repository" button offers the paper in APA and BibTeX through the preferred-citation entry in CITATION.cff.
Li, C. J., Liu, X., Shang, H., Zou, X., Mo, Z., Zhang, C. E., & Jiang, H. (2026). Social Simulation Arena: A Prospective Benchmark for Social Simulation (Version v0.1). Zenodo. https://doi.org/10.5281/zenodo.23140107
@misc{li2026socialsimulationarena,
author = {Li, Chance Jiajie and Liu, Xuan and Shang, Haoyang and Zou, Xinkai and Mo, Zhenze and Zhang, Cedegao E. and Jiang, Hang},
title = {Social Simulation Arena: A Prospective Benchmark for Social Simulation},
year = {2026},
month = sep,
publisher = {Zenodo},
version = {v0.1},
doi = {10.5281/zenodo.23140107},
url = {https://zenodo.org/records/23140107},
note = {Preprint; protocol snapshot v0.1}
}
This is a living paper with a growing author list. Future versions may expand both the paper and its authorship. Cite the version you used so the citation preserves that version's authors and content.
The Season 0 software citation remains available in the top-level metadata of CITATION.cff:
@software{socialsimulationarena2026,
author = {Li, Chance Jiajie and Liu, Xuan and Shang, Haoyang and Zou, Xinkai and Mo, Zhenze and Zhang, Cedegao E. and Jiang, Hang and {Social Simulation Arena Team}},
title = {Social Simulation Arena},
year = {2026},
version = {season-0},
url = {https://social-simulation-arena.com},
note = {https://github.com/Social-Atoms/social-sim-arena}
}
