RealtimeCUA Bench
Computer use with shared state, asynchronous change, and real deadlines.
Model leaderboard and task videos · Native video-learning tasks · Replay data and provenance
Project website · English task catalogue · Original catalogue
RealtimeCUA studies whether a computer-use agent can notice changing requirements, preserve concurrent edits, and act before an opportunity disappears. The environment continues to advance while the agent reasons or waits.
This release contains 32 implemented task packages: 16 English browser tasks, nine Chinese browser tasks, the original native Blender rocket, and six native video-learning tasks. Three Chinese livestream tasks require separately supplied source video. The website publishes 52 catalogue entries: 32 implementations and 20 existing OSWorld candidates. The full 73-entry source catalogue, including planned concepts, remains in data/ for provenance; planned entries are excluded from the website.
Included tasks
| Task ID | Scenario | English version | Timing / evaluation |
|---|
| rcua_doc_001 | Quarterly report: three timed checks | 0.2.0-en.1 | Three 24-second save windows; 100-second total |
| rcua_doc_002 | Procurement plan: three budget reviews | 0.2.0-en.1 | Budget changes and withdrawal; three checkpoints |
| rcua_doc_003 | Release plan: concurrent edits and timed merges | 0.2.0-en.1 | Preserve changing time and owner; merge stale revisions |
| rcua_doc_004 | Co-writing: The Old Bookshop on a Rainy Night | 0.1.1-en.1 | Three writing windows; 120 seconds; trusted semantic review |
| rcua_doc_005 | A changing problem: revise every answer | 0.1.2-en.1 | Four answer-triggered versions; 45/35-second response windows; 180 seconds |
| rcua_ticket_001 | Ticket release: a one-second inventory window | 0.4.0-en.1 | Sales open after 60 seconds; stock exhausted within one second |
| rcua_condition_001_market | Prediction markets: five live games, ten brief opportunities | 0.2.0-en.1 | Ten 15-30 second trade windows that close when the game or price changes; 360 seconds; also a paused-world mode |
| rcua_ticket_002_food |
| rcua_game_003_pipeflow | PipeFlow: route fluid before it arrives | 0.1.0-en.1 | Three levels; 15-second planning; realtime / paused |
| rcua_game_004_portalsnake | Portal Snake: collect fruit through linked portals | 0.1.0-en.1 | 160 seconds; realtime / paused |
| rcua_stream_002_trademonitor | Trading supervisor: stop faulty strategy orders | 0.1.0-en.1 | Ten incidents; realtime / paused and specification variants |
| rcua_ticket_003_classreg | Class registration: scarce seats and outages | 0.1.0-en.1 | Registration opens at 30 seconds; closes at 180 seconds |
The sixteen English directories include English instructions, a local application, frozen scenario, setup/cleanup, trusted replay evaluator, OSWorld-style adapter, checksums, and a matching application bundle. Downloadable complete English ZIPs are in docs/downloads.
Quick start
Requires Python 3.10+ and a modern browser. The task runtime uses only the Python standard library, except the Node.js tasks (rcua_condition_001_market, rcua_doc_006_finmodel, rcua_stream_001_chat): their servers are pre-bundled Node.js programs that need Node.js 18+ locally, and in an OSWorld VM their adapters download the official Node.js build from nodejs.org (pinned to its published SHA-256; OSWorld's setup downloads it on the host and uploads it, so the VM needs no internet). No binary is stored in this repository. Their graders and tests are standard-library Python.
git clone https://github.com/Rongjiehuang/RealtimeCUABench.git
cd RealtimeCUABench/tasks/rcua_doc_001
python3 initial_setup.py --run-dir /tmp/rcua-report-trial01 --port 0
Open the URL printed by the command. The first browser-ready event starts the server clock. Page reloads, inference, and waiting do not pause or reset the episode. Use a new run directory for each trial.
Run the trusted evaluator and stop the task-owned server:
python3 reward.py --record /tmp/rcua-report-trial01/record.json
python3 initial_setup.py --stop --registry /tmp/rcua-report-trial01/launch.json
The English browser tasks follow this interface. Native tasks and the newer Chinese tracks have their own setup requirements; follow the README in each task directory. The ticket task creates a simulated order only; it involves no actual booking or payment. Tasks with a paused-world mode (rcua_condition_001_market, rcua_doc_006_finmodel, rcua_game_001_tetris, rcua_game_002_fruit, rcua_game_003_pipeflow, rcua_game_004_portalsnake, rcua_stream_001_chat, rcua_stream_002_trademonitor, rcua_ticket_002_food, rcua_ticket_003_classreg) also take --mode realtime (default) or --mode paused; a paused mode has its own OSWorld task id ending in _paused, and its results are reported separately.
Observation, action, and scoring
- Agent: screenshots and ordinary mouse/keyboard interaction. The agent may not inspect DOM/CDP, call task APIs, or read/write server configuration, journals, or grading files.
- Trusted setup/evaluation: may inspect configuration and server records. Keep these outside the evaluated agent environment.
- Clock: one server monotonic timeline, armed once by browser readiness. Collaborator and inventory events advance independently of agent calls.
- Document tasks: grade actual saved versions and submission, not a model claim of success. A correct final answer cannot repair an earlier missed checkpoint. Preserve collaborator edits and avoid forced overwrites.
- Story task: mechanical checks require a new paragraph of at least 40 English words in each window. A final numeric score additionally requires an evidence-backed trusted semantic review. Without review, the evaluator returns
score: null and status: semantic_review_required. See story review.
- Ticket task: replay the inventory and order journal; verify exact session, tier, attendee, quantity, budget, and timing. Retries are idempotent.
- Market task: re-judge every order against the frozen 3 Hz game and market state table at its submission tick; +1 per opportunity captured with the exact ticket, -2 per other order, score = min(1, max(0, raw) / 10). The paused-world mode (
rcua_condition_001_market_paused) freezes the server clock between inputs; its results are reported separately.
- Spreadsheet task: rebuild the workbook from the journal and score each published report on the fields it froze (+1 right, -1 per broken guarded cell), 100/6 points per report, score = points / 100. The paused mode (
rcua_doc_006_finmodel_paused) freezes workbook time between inputs; its results are reported separately.
- Tetris task: replay the page's journal (spawns, inputs, locks, line clears) against the final board; score = lines cleared / 40. Chrome DevTools traffic from the agent is a hard fail. The paused mode (
rcua_game_001_tetris_paused) runs game time only during input; its results are reported separately.
- Fruit task: replay the page's journal (launches, swipes, slices, bomb hits, exits) against the final state; score = max(0, fruit sliced - 5 x bombs touched) / 100. Chrome DevTools traffic from the agent is a hard fail. The paused mode (
rcua_game_002_fruit_paused) runs game time only during input; its results are reported separately.
These collaborative mocks use custom paragraph-revision semantics; they do not reproduce Google Docs OT/CRDT internals. The collaborator is a server script, not a human or another GUI model. Journal hashes support consistency checks; they are not a security boundary against an agent with server-file access.
English localization and historical results
The sixteen English task packages have English UI, scenario messages, instructions, and grader matching strings. The newly integrated Chinese packages retain their source language; English catalogue descriptions explain those versions and do not define localized benchmark variants. Timing, numerical targets, event order, IDs, and conflict mechanisms are retained. The story length requirement changes from 40 Chinese characters to 40 English words. Versions therefore use an explicit -en.1 suffix. See localization and provenance.
Unmodified source packages are preserved in upstream. Their original-language contents and model results are historical artifacts. Aggregate historical results are transcribed from the source catalogue and do not measure the English variants. New Chinese model pilots are preserved separately in eval_runs/, results/ and the rocket evidence page. These do not establish English model performance. No real OSWorld VM evaluation is claimed.
Validation
python3 -m pip install -r requirements-dev.txt
python3 -m pytest tests -q
python3 -m unittest discover -s tasks/rcua_ticket_001/tests -v
python3 -m unittest discover -s tasks/rcua_condition_001_market/tests -v
python3 -m unittest discover -s tasks/rcua_doc_006_finmodel/tests -v
python3 -m unittest discover -s tasks/rcua_game_001_tetris/tests -v
python3 -m unittest discover -s tasks/rcua_game_002_fruit/tests -v
python3 -m unittest discover -s tasks/rcua_stream_001_chat/tests -v
python3 -m unittest discover -s tasks/rcua_ticket_002_food/tests -v
python3 scripts/check_integrated_tasks.py
python3 -m pytest tasks/rcua_video_blender_001/tests -q # Linux
python3 scripts/check_rocket_variant.py # Linux, 600s diagnostic
python3 -m unittest discover -s tasks/rcua_game_003_pipeflow/tests -v
python3 -m unittest discover -s tasks/rcua_game_004_portalsnake/tests -v
python3 -m unittest discover -s tasks/rcua_stream_002_trademonitor/tests -v
python3 -m unittest discover -s tasks/rcua_ticket_003_classreg/tests -v
python3 scripts/verify_repository.py
python3 scripts/verify_replays.py
python3 scripts/verify_video_learning.py
python3 scripts/check_native_saveguards.py # Linux
The tests cover timely saves, missed checkpoints, stale-revision merge, overwrite failure, preservation of collaborator content, all four English arithmetic revisions, accepted discount conventions, English word counts, review evidence binding, inventory contention, idempotency, and invalid journals. Engine tests use injected clocks; the ticket suite also checks background progression with a real clock. Browser smoke checks are recorded separately in validation.
OSWorld V2 adapters
These are project-authored tasks, not official OSWorld tasks. Use an existing OSWorld V2 checkout and provider VM; neither is included.
python3 scripts/install_osworld.py /absolute/path/to/OSWorld-V2
The installer refuses conflicting files unless their bytes already match. It copies task entries, frozen graders, selection lists, and asset bundles. Set OSWORLD_FILE_BASE_URL to the printed absolute asset directory and use the generated evaluation_examples/test_realtimecua.json with the V2 runner. See integration. The adapters have been checked for packaging consistency but have not been exercised in a real OSWorld VM.
Website
The bilingual website (English / Chinese) is a buildless static site in docs. It publishes 52 catalogue entries in both languages, a persistent language switch, bilingual search, category/source/status filters, expandable instructions, historical-result boundaries, and complete downloads for all 32 primary tasks. The native video-learning page includes tutorials and 14 model videos; three livestream implementations are marked as requiring local source media. Both catalogue languages describe the same versioned packages; each package retains its stated task language. Use ?lang=zh or ?lang=en to share a language-specific view. GitHub Pages serves main → /docs.
python3 -m http.server 8000 --directory docs --bind 127.0.0.1
Visit http://127.0.0.1:8000/. Task servers run separately: GitHub Pages hosts the catalogue and downloads, while Python runs the live task backends locally.
Repository layout
tasks/ All task packages, including language and time-budget variants
data/ Catalogue, package registry, integration provenance and validation
docs/ Bilingual GitHub Pages site, replays and English/Chinese downloads
scripts/ Deterministic packaging, integration, and repository checks
tests/ Document contracts and English localization regression tests
upstream/ Unmodified original-language source archives
Original catalogue snapshot: 2026-10-19. English release: 2026-10-24. Branch and rocket integration: 2026-10-28. Latest branch / video-learning import: 2026-11-01. Source attribution and redistribution scope are described in NOTICE.
Integrated source tracks (28 October 2026)
The main, aczx and ry-rcua-tasks histories are integrated. Exact input commits and the rocket archive hash are recorded in data/integration.json. The October input is retained for provenance. The November update integrates aczx@0b13646 and ry-rcua-tasks@cb92943; data/update_2026_11_01.json records the new input commits. The current source catalogue contains 73 scenarios, and the website publishes 32 implemented packages and 20 existing OSWorld candidates.
| Original Chinese package | Scope | Maturity / evidence |
|---|
| rcua_ticket_002 | TicketVault sale, 90 frozen episodes / three tiers | Reference ticket track; model and oracle samples in eval_runs/ |
| rcua_live_001 | Livestream response and restraint | Historical frozen package, superseded upstream; retained for the 17-task replay cohort |
| rcua_meeting_001 | Captions-only meeting response | Early, actively being deepened; no model evaluation |
| rcua_present_001 | Streaming presentation requests | Early, actively being deepened; no model evaluation |
| rcua_notes_001 | Shared live minutes | Early, actively being deepened; no model evaluation |
| rcua_video_blender_001 | Native Blender rocket, 210-second formal deadline | v0.1.9; rubric, verified source and model replay page |
| rcua_game_001 | Chinese server-driven fruit slicing | 120-second standalone track; separate from the English fruit task |
| rcua_live_002 |
The 600-second Blender diagnostic is a separate budget variant of the rocket task. Both 483-file source snapshots are unchanged from the verified original archive, including all tutorial frames, the native projects, geometry grader and manifests. Linux, Blender 4.2 LTS and NumPy are required for trusted native evaluation. Host-side references and future frames must remain outside the solver environment. This task has no validated OSWorld VM adapter.
rcua_ticket_001_zh preserves the Chinese branch variant; tasks/rcua_ticket_001 remains the English release. They share one catalogue identity. Branch maturity notes preserve the latest branch README and its limitations. The old site/ snapshot is archival; GitHub Pages still serves main → /docs.
Run the older Chinese browser tasks from their directory under tasks/ with python3 initial_setup.py --run-dir /tmp/rcua-new-trial01 --port 0. For the tiered ticket track, use --tier t2 --episode-id ep00. Follow the rocket README for its native Linux setup. Downloads and source directories are enumerated in data/task_packages.json.
The immutable diagnostic snapshot inherited four assertions with hardcoded 210-second expectations. scripts/check_rocket_variant.py runs its full suite in a temporary copy with test literals derived from the actual 600-second deadline. The original snapshot, runtime and downloadable archive remain byte-exact.
Rebuild the published bilingual catalogues with python3 scripts/build_site_catalogue.py. The site excludes planned entries, supplementary concepts and categories with no published entries; the complete source catalogues remain in data/.
All task directories now live directly under tasks/. The Chinese ticket alternative is tasks/rcua_ticket_001_zh/, alongside the English tasks/rcua_ticket_001/. The extended Blender budget is tasks/rcua_video_blender_001_diag600/; the 300-second Godot learning variant is tasks/rcua_learn_godot_002_budget300/. Source files and scenario IDs are preserved; data/task_packages.json records primary packages and additional variants.
GPT / Haiku replay leaderboard
docs/leaderboard/ contains 34 downloaded MP4s for 17 task variants at fixed commit 13f1b74, paired GPT-6.1 Sol / Claude Haiku 5.5 scores, per-task results, instructions and source frame metadata. Every video SHA-256 was checked against the source collection. These include 12 English task versions, four Chinese primary tasks and the Chinese one-second ticket alternative. Each task/model has one valid local GUI trial; the equal-weight mean of published task scores is not a success-rate estimate. API routing and reasoning defaults differ. Story-task raw semantic-null results are retained with the source's explicitly derived strict-failure zeros. These runs do not establish native OSWorld VM compatibility.
Older rocket and TicketVault pilots are available separately in the same page and excluded from the 17-task aggregate. Catalogue entries link to replay evidence when available; newly added implementations without a published video are not represented as model evaluations. Run python3 scripts/verify_replays.py to check hashes, versions, instructions, scores, aggregates and coverage. Preview the site with python3 -m http.server 8000 --directory docs and open /leaderboard/.
Native video learning (1 November 2026)
The video-learning page mirrors the current six-task source collection, its tutorials, 14 SHA-verified model MP4s, raw results, corrected evaluations, publication audits and encoding manifests. Complete native packages remain under tasks/; the registry also retains the separate Godot 300-second package.
| Package | Current evaluation version | Protocol |
|---|
| rcua_learn_blender_002 | 0.4.0-localvideo-en.1-saveguard.1 | Exhibition stand; immutable 120-second pickup and 600-second final |
| rcua_learn_calc_001 | 0.7.0-video-only-self-notes-en.1 | Uninterrupted video lesson and self-written notes |
| rcua_learn_impress_001 | 0.4.0-localvideo-en.15 | Editable slide layout; 120 / 600-second pickups |
| rcua_learn_inkscape_001 | 0.4.0-localvideo-en.2-saveguard.1 | Editable vector work; 120 / 600-second pickups |
| rcua_learn_kicad_001 | 0.4.0-localvideo-en.2-saveguard.1 | Native circuit project; 120 / 600-second pickups |
| rcua_learn_godot_002 | 0.2.1-fullscreen-budget600-en.1 | Continuous fullscreen lesson; 600 seconds (300-second variant retained separately) |
Model-run versions and evaluation versions are displayed separately: several current scores regrade the same previously saved project, rather than recording another model run. These local pilots are separate from the fixed 17-task leaderboard and are not included in its mean. Tutorial playback rules differ by task; the review player's controls do not define benchmark permissions. Native applications and Linux desktop dependencies are described in each HOST_SETUP.md; application binaries are not bundled, and native OSWorld VM compatibility has not been validated.
Source import hashes, normalized task / replay metadata and original source metadata preserve provenance. scripts/verify_video_learning.py checks all 963 imported source files and the 91 mirrored evidence files, including every selected model-video hash.
