AirOpsEnv v3

---

AI agents run JetBlue's operations control center through a real storm day with its passengers on board, from the three desks real control centers have. The ops controller runs the flights and aircraft with the FAA, the crew scheduler runs the pilots and flight attendants, as in v2, and the new customer desk looks after the passengers. Every flight carries its booked passengers, about 86,000 trips on a storm date, some connecting at JFK, Boston, Fort Lauderdale, Orlando or San Juan, generated from public records: each flight's load from BTS T-100 and who connects where from the DOT's ticket survey of the date's own period. When a flight is cancelled or a connection missed, its passengers are rebooked the way JetBlue's contract says: on JetBlue only, the earliest arrival first, for up to five days, then a refund. The customer desk sees whose connections are at risk and where a cancellation's passengers would go, and asks ops to hold a flight for them, not to cancel one, or to cancel one whose passengers are better off rebooked now; ops decides. The day is one of v2's 24 storm dates at JFK or Boston from 2022 to 2026, the whole network from 04:00, played watch by watch on AgentEnv's virtual clock and graded once, on what really happened, with no judge, under a cost that follows every passenger to their final destination: against the best plan found in hindsight under that cost.

Contents: The customer desk · Passengers · Grading under P · The v3 board · Run v3 · How v3 was validated · Watch a v3 run · The dataset and the Space · AirOpsEnv v2 · AirOpsEnv v1 · Repository · Licence and attribution

The customer desk

	Ops controller	Crew scheduler	Customer desk
Sees	v2's picture, now with each flight's passengers (booked, connecting in and out) and the day's passenger costs	v2's picture	the connections at risk in the next 4 h; the flights ops might cancel in the next 6 h, with where their passengers would go if cancelled now (the same day, a later day, a refund) and how many have no later seat today; any flight's passengers by destination
Does	as v2; answers the customer desk's requests	as v2	looks passengers up
Asks ops		HOLD, CANCEL, CANCEL_PAIR, TEXT	HOLD (a departure, for its connecting passengers), PROTECT (don't cancel a flight), SUGGEST_CANCEL (cancel one), TEXT
Can't	move or assign crews; rebook passengers	cancel or hold flights; rebook passengers	cancel, hold or crew flights; rebook passengers

• The customer desk proposes; ops decides, as in real control centers, where the passenger desk can say "shouldn't" and crew and aircraft can say "can't". An accepted HOLD is ops' hold, as the crew desk's is; PROTECT and SUGGEST_CANCEL are advice, which ops carries out, if at all, with its own tools. Ops sends the crew desk COVER and TEXT, and the customer desk TEXT.
• No desk rebooks. The passenger system rebooks automatically, on JetBlue only, the earliest arrival first; holds, cancellations and their timing decide where the passengers go.
• Turns. Each hourly watch the crew desk takes its turn, then the customer desk, then ops; then the crew desk again if ops sent it a request or a text, and the customer desk if ops texted it. The env runs the turns, so every run replays exactly.
• One agent (mode one) runs all three desks from one page and their tools, with no channel, for comparison.

Desk	Tools
ops	ops_flights, ops_program, ops_aircraft, ops_check, ops_confirm, ops_request, ops_reply
crew	crew_flights, crew_crews, crew_reserves, crew_program, crew_options, crew_check, crew_confirm, crew_request, crew_reply
customer	cust_flights, cust_flight, cust_request
all three	end_turn, wait_turn, resign

Each desk is its own A2A agent (airops-desk, in role ops, crew, customer or both) and sees only its role's tools on the AgentEnv gateway. A day's task runs v2's steps (below) on the env airops-occ3, with the customer desk deployed and played beside the other two, and the grade's audit checks every desk.

Passengers

The pack b6-v3 is v2's 70 dates (24 eval storm dates, 44 training dates and 2 calm controls), each with its passengers and its references under the passenger cost. No airline's itineraries are public, so the passengers are generated stand-ins, labelled as such like v2's crews:

• Who flies. Each flight's booked passengers are its seats times the route's T-100 load factor that month, as in v2.
• Who connects. At JetBlue's connecting stations (JFK, BOS, FLL, MCO, SJU), a route's share of connecting passengers from each origin comes from the DOT's Origin and Destination Survey of the date's own period (the DB1B quarter to June 2025, then the DB1C month), spread over the day's scheduled JetBlue-to-JetBlue connections by DB1C's connection times, with minimum connection times of 45 minutes at JFK and Boston and 40 elsewhere. An eval date has a median of 86,148 passenger trips, 2.7% of them connecting.
• How they are rebooked, by JetBlue's own published rules: on JetBlue only, from the decision, the earliest arrival first (Contract of Carriage §37(c)); for up to five days, then a refund (its Customer Service Plan); a flight 3 h or more earlier only when no other seat is left (DOT's refund rule, 14 CFR 260.2). Crews keep their seats first. On a storm day JetBlue owes no compensation, meals or hotels, so none is charged; a night away costs the passenger a lodging night.
• Not modelled, and labelled: international connections (the public survey and the on-time data are domestic only), rebooking on other airlines, compensation on storm days, rebooking by hand at the desk, and passengers who choose a refund.

FIDELITY.md says what is real data and what is generated, the rules' sources, every check against the record and every miss.

Grading under P

The day's cost P, in passenger-minutes: each passenger's arrival delay at their final destination, rebooking included (a refunded passenger's wait); 66 for each cancelled flight's passenger, the refund exposure, more when the cancellation comes under 2 hours ahead; a lodging night for each night a passenger waits; the fare of each passenger refunded; and v2's aircraft and crew costs. v2 charged delay at every leg and a flat 523 per cancelled passenger.

Reference	What it is
best plan	the cheapest plan under P from a fixed set: v2's best plan, B0, the passenger-aware aircraft plans played through the standard crew desk, and a repair of the best of them; replayed exactly with a legal crew on every flight; it scores 1.0
bound	a capacity-aware lower bound under P, published per date: 54% to 79% under the best plan on the eval dates (median 66%), most of it the solver not finishing in its 30 minutes
B0	standard crew-desk practice with the aircraft doing nothing; 0.308 on 20 of the 24 eval dates, more on the 4 where it comes within 10% of the best plan (a mean of 0.386)
doing nothing	the planned crews by the trip rule, no desk at all (a mean of 0.329)
v2's best plan	v2's best plan of aircraft and crews, blind to passengers, played with them: at least 5% over the best plan on 17 of the 24 eval dates (a mean of 0.602)
the airline	JetBlue's own decisions, replayed with the generated crews and passengers (a mean of 0.239)

The reward is v2's, under P: 0.2 + 0.8 exp(−g/0.5), with g = max(0, P − best) / max(B0 − best, 0.1 best); reward_L is the same against the bound. A run is scored only if its audit passes, as in v2: every watch's news on time, every call in its desk's turn, no decision reaching back, each desk in its role, a legal crew on every flight, the cost not under the bound, and every desk present. No judge. The best plan is a best-found plan, not a proven optimum, and the bound is weak, so most of the published gap is the bound's.

The v3 board

GPT-6.1 Sol (openai/gpt-6.1-sol, reasoning effort high) played the 24 eval dates once in each mode, registered before the first run: the code (f232962, the pack with it), the dates, the caps ($3 a desk, $6 for one agent) and the budget. Five more models then played the same 24 dates with three desks, once a date, at reasoning effort high: GPT-6 Astra (openai/gpt-6-astra), Claude Opus 5.5 (anthropic/claude-opus-5-5), Claude Sonnet 5.5 (anthropic/claude-sonnet-5-5), GPT-5.6 Terra (openai/gpt-5.6-terra) and Kimi K3 (fireworks_ai/kimi-k3). The models, the dates, the caps, the retries and how the models are compared were registered before the first of them ran; Sol isn't run again, its three desks are its board's 24 runs. A desk's cap buys what $3 bought a Sol desk: Sol's median desk's tokens at the model's prices, rounded up to the next $0.50. The five boards ran at 6e5ec17 on 22 October 2026, from 05:17 to 07:59 UTC, on five r7i.24xlarge machines, one a model, within a budget of $3,000 for model spend and cloud together. All 168 board runs scored, every one replays exactly through the env's calls on Linux, as the boards ran, and passes the audit, and every reward and reward_L recomputes from the graded costs and the pack's best plan, bound and B0.

	Model	Mean reward (95% CI)	reward_L (95% CI)	Below best	Spend	Cap a desk	Median day
1	GPT-6 Astra	0.742 (0.673–0.813)	0.377 (0.356–0.406)	6	$544.24	$23.50	58 min, $21.33
2	GPT-6.1 Sol	0.721 (0.633–0.811)	0.372 (0.351–0.398)	7	$71.38	$3	70 min, $2.79
3	Claude Opus 5.5	0.626 (0.544–0.711)	0.357 (0.340–0.376)	3	$261.46	$9.50	43 min, $10.40
4	Claude Sonnet 5.5	0.556 (0.476–0.644)	0.347 (0.330–0.367)	2	$72.28	$5	27 min, $3.23
5	GPT-5.6 Terra	0.515 (0.443–0.594)	0.337 (0.324–0.353)	1	$84.41	$5	52 min, $3.70
6	Kimi K3	0.426 (0.366–0.491)	0.326 (0.310–0.342)	0	$150.51	$7.50	57 min, $6.01
	GPT-6.1 Sol, one agent at all three desks	0.677 (0.592–0.766)	0.366 (0.346–0.391)	5	$35.66	$6	44 min, $1.52
	Standard crew desk (B0)	0.386
	v2's best plan, blind to passengers

Rows 1 to 6 are each model's three desks, talking, ranked by their mean. Below best: the runs that came in under the best plan found. Median day: a day's wall time and spend at the median.

• GPT-6 Astra leads; GPT-6.1 Sol and Claude Opus 5.5 are not separated from it: their 95% intervals overlap Astra's. Claude Sonnet 5.5, GPT-5.6 Terra and Kimi K3 lie below it, their intervals wholly under Astra's. reward_L, which the best plan doesn't cap, ranks the six in the same order.
• The best plan's ceiling compresses the top. A run under the best plan found scores 1.0 whichever model played it and however far under it came: Astra's did on 6 dates and Sol's on 7, Claude Opus 5.5's on 3, Claude Sonnet 5.5's on 2, GPT-5.6 Terra's on 1 and Kimi K3's on none, all of them on the 7 dates where Sol's did.
• Spend doesn't follow the ranking. Astra's board cost $544.24, 7.6 times Sol's $71.38, for 0.021 more; Claude Opus 5.5's $261.46; Kimi K3's $150.51, twice Sol's, for the lowest mean; Claude Sonnet 5.5's and GPT-5.6 Terra's near Sol's. The caps stopped 5 desks, all crew desks: one of Astra's 72, three of Opus's and one of Kimi's, and none of Sol's, Sonnet's or Terra's (FIDELITY.md).
• Every run scored. GPT-5.6 Terra's 24 took 31 attempts: 7 first attempts lost a desk to Azure's rate limit (provider_error) and scored on a retry. Every attempt is kept in the dataset.
• CIs are bootstrap over the 24 dates (10,000 resamples). Spend is the harness's tokens times price, over a model's scored runs: Sol's board cost $107.04 and the pilot before it $6.20; the five boards cost $1,125.30, Terra's 7 unscored attempts ($12.39) included.

Sol's two modes:

• Three desks against one agent, paired by date: +0.044 reward (bootstrap 95% CI 0.007–0.083), three desks higher on 14 of 24 dates, lower on 5 and level on the 5 where both came in under the best plan; their days cost 1.8% of the best plan less (CI 0.5–3.1%). A small margin: see below.
• Both well above standard practice: at the median a Sol day saves 71% (three desks) or 72% (one agent) of what B0 leaves on the table and costs 7.7% more than the best plan found, and both modes' means are above v2's best plan played with its passengers (0.602).
• 12 of Sol's 48 runs came in under the best plan found: both modes on 5 dates (2023-07-13, 2023-07-29, 2024-06-23, 2024-07-15 and 2024-07-22) and three desks alone on 2 (2024-07-13 and 2025-12-30), so 7 three-desk runs and 5 one-agent runs. They score 1.0 and are flagged below_best, with reward_L, against the bound, beside them. On 4 of those 5 dates B0 itself comes within 10% of the best plan, which is weak there. As registered before the board, the best plan isn't updated from played runs, so no run's score depends on another's; a longer search for better plans is left to v3.1.

Each model's column is its three-desk run; Sol's one agent is last.

Date	Storm hubs	Flights	Passengers	Connecting	Gap, best to bound	Astra	Sol	Opus	Sonnet	Terra	Kimi	Sol, one agent
2022-12-07	JFK, BOS	767	86,500	3.5%	78.7%	0.484	0.477	0.378	0.296	0.288	0.249	0.465
2023-06-06	BOS	774	93,810	2.9%	64.5%	0.775	0.848	0.696	0.594	0.821	0.668	0.587
2023-06-16	JFK, BOS	753	91,328	2.9%	59.6%	0.686	0.548	0.481	0.434	0.620	0.432	0.515
2023-07-08	JFK, BOS	693	86,237	2.8%	54.2%	0.555	0.528	0.471	0.385	0.462	0.321	0.440
2023-07-09	JFK, BOS	733	90,919	2.8%	62.8%

The day's highest in bold, none where several runs share 1.0. * Under the best plan found, so 1.0 and flagged below_best; reward_L, below, is the fairer reading there. B0 scores 0.308 on 20 of these dates and more on 2023-07-29, 2024-06-23, 2024-07-15 and 2024-07-22. "Passengers": the day's passenger trips; "Connecting": the share of them that connect.

reward_L, against the bound:

Date	Astra	Sol	Opus	Sonnet	Terra	Kimi	Sol, one agent
2022-12-07	0.329	0.329	0.319	0.306	0.304	0.294	0.327
2023-06-06	0.345	0.348	0.341	0.336	0.347	0.340	0.335
2023-06-16	0.366	0.351	0.342	0.335	0.359	0.335	0.347
2023-07-08	0.371	0.366	0.354	0.333	0.352	0.313	0.347
2023-07-09	0.364	0.359	0.352	0.334	0.290	0.317	0.344
2023-07-13	0.383	0.380	0.343	0.357	0.354	0.345	0.385
2023-07-29	0.344	0.339	0.329	0.311	0.301	0.298	0.334
2023-12-10	0.388	0.393	0.380	0.362	0.337	0.336	0.375

What the results show, and what they don't

• They show that on these 24 storm dates GPT-6 Astra's three desks scored highest and that Claude Sonnet 5.5, GPT-5.6 Terra and Kimi K3 scored below them, their 95% intervals over the dates wholly under Astra's; and that three Sol desks scored above one Sol agent running all three, +0.044 (bootstrap CI 0.007–0.083), higher on 14 dates and lower on 5, both recovering most of what standard crew-desk practice loses.
• They don't separate GPT-6 Astra from GPT-6.1 Sol or Claude Opus 5.5. Their intervals overlap Astra's, and Astra's lead over Sol, 0.021, is a tenth of Sol's own run-to-run swing on the same day and mode (up to 0.23). The best plan's ceiling lifts the top two: Astra came in under the best plan on 6 dates and Sol on 7, scoring 1.0 there, against 0 to 3 for the other models. Nor do they show that spend buys score: Astra's board cost 7.6 times Sol's, and Kimi K3's, twice Sol's, has the lowest mean.
• They don't show that Sol's third desk or talking helps. The margin is small: a sign test on the 19 dates where the two differ gives p = 0.064, and leaving out the 7 dates with a below_best run gives +0.043 with a CI of −0.002 to 0.090. Sol's score for the same day and mode moves by up to 0.23 from one run to the next, the size of the effect, and the board has one run per date and mode. The three desks also used about twice the compute (2.0× the spend, 2.0× the input tokens and 2.4× the output tokens, more spend on every date), so the board can't separate three desks from more compute, as v2's couldn't separate two. The caps were $3 a desk against $6 for one agent; 2 crew desks reached 90% or more of their $3 cap (the highest 96%), and none ended by it.
• Nor what the customer desk adds by itself. The board has no two-desk arm under P, so the customer desk's own worth isn't measured. Sol's was busy: 1,303 requests over the 24 three-desk days (a median of 56 a day), 887 HOLDs (463 accepted by ops), 184 PROTECTs (177 accepted), 76 SUGGEST_CANCELs (24 accepted) and 156 texts.
• The scripted gate for the customer desk didn't pass. Before any model ran, a scripted customer desk built from published practice (hold an outbound up to 15 minutes when 10 or more connecting passengers would misconnect; ask ops not to cancel a flight when half its passengers have no later JetBlue seat that day) was measured against two scripted desks: it saved at least 2% on 2 of the 24 eval dates, against 12 needed, and cost more on 15. Its requests not to cancel mostly protected flights that couldn't fly, which then cancelled at their deadline, late. The premise holds (v2's best plan costs at least 5% more than the best plan under P on 17 of 24 dates), but the script didn't capture it. The customer desk ships anyway, labelled.
• Six models, one run per date each. Each model played each date once with three desks, and Sol once more with one agent, so a model's interval is over the dates, not over repeated runs, and the models are compared, as registered, by whether their intervals overlap the leader's; nothing is paired across models. Sol varies from run to run on the same day (the pilot's one-agent run on 2024-07-28 scored 1.0 on a laptop and 0.613 on a test box); more runs per date would settle what the board leaves open.

Run v3

You need Docker (running, with buildx), uv and git. Install the plugin with the framework, build the env and the desk agent and register them, and check the wiring with a day no model plays:

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/[email protected]"
agent-env airops-occ3 setup                   # builds airops-occ3 and airops-desk for this machine's Docker
agent-env airops-occ3 tasks --noplay          # writes results/bundles/airops-occ3: each eval date, joint and one
agent-env run results/bundles/airops-occ3 --task wiring-noplay

wiring-noplay loads the first eval date and runs it with no desk: the crew and passenger systems alone, then the grade. Then play a day with a model on the endpoint [model] in .agentenv/config.toml names (LiteLLM, or the Hugging Face router):

agent-env run results/bundles/airops-occ3 --task b6-20240713-0400-joint --model openai/gpt-6.1-sol
agent-env run results/bundles/airops-occ3 --task b6-20240713-0400-one --model openai/gpt-6.1-sol

Or play a day straight from the dataset, which pins this release and holds the bundle airops-occ3: every date in both modes, the very task files Sol's board ran (each desk's cost cap $3 in a three-desk run, $6 for one agent), and wiring-noplay:

agent-env hf run earakely-scale/[email protected] --bundle airops-occ3 --task wiring-noplay
agent-env hf run earakely-scale/[email protected] --bundle airops-occ3 --task b6-20240713-0400-joint \
    --model openai/gpt-6.1-sol

Realistic mode plays the same days in a world closer to the real one (v4's caps and closures, gates, aircraft that break, swaps only at the hubs, tomorrow's first departures), graded on JetBlue's own cost with JetBlue's own decisions replayed as 0.5; REALISTIC.md has the world, the cost and the references:

agent-env airops-occ3 tasks --realistic --split all   # results/bundles/airops-occ3-real: b6-v3r's 22 days
agent-env run results/bundles/airops-occ3-real --task b6-20230627-0400-joint-real --model openai/gpt-6.1-sol

A pilot plays runs in a fixed order under a budget, retries a run that failed for an infrastructure reason at most twice, and reports every scored run replayed and audited:

agent-env airops-occ3 pilot occ3-my-board --run b6-20240713-0400-joint --run b6-20240713-0400-one \
    --cap-joint 3 --cap-one 6 --budget 40 --request-margin 0.75 --parallel 2
agent-env airops-occ3 report occ3-my-board    # results/runs/occ3-my-board/report.json

tasks --split train writes the 44 training dates, --split control the calm days, --cap-joint and --cap-one set each desk's cost cap (by default $5 and $12), and --policy script plays scripted desks with no model. Sol's board ran at $3 a desk and $6 for one agent; the other models' boards ran three desks at their own caps a desk, scaled from Sol's $3 by their prices: GPT-6 Astra $23.50, Claude Opus 5.5 $9.50, Kimi K3 $7.50, and Claude Sonnet 5.5 and GPT-5.6 Terra $5. On Sol's board a three-desk day took a median 70 minutes and $2.79, one agent 44 minutes and $1.52; the other models' three-desk days took a median 27 minutes (Claude Sonnet 5.5) to 58 (GPT-6 Astra) and $3.23 (Sonnet) to $21.33 (Astra), as the board gives. Name a three-desk pilot occ3-* so that scripts/hub_dataset.py publishes it with v3, and a model's board occ3-board-<model>.

How v3 was validated

v3 was built in stages as v2 was, each checked before the next with its checks pre-registered, every change to a rule recorded with its reason before it was used, and every bar recomputed by an independent agent from the run's files. FIDELITY.md has the full report.

• The premise, without crews (step 0). Priced by P, v2's aircraft plan costs at least 5% more than a passenger-aware plan on 16 of the 24 eval dates (12 needed). It isn't the connections: the same dates pass at half and twice the connecting shares. It is pricing each cancelled passenger by their real rebooking, which favours cancelling flights whose passengers fit on other JetBlue flights.
• The passengers. The generated bookings match T-100 within 5% on all 70 dates, today and on the following days, and the connection times match DB1C's within 20 minutes at every connecting station. The connecting shares run low: 0.1 to 1.9 points under the survey, a miss (about 20% of the connections the survey implies can't be placed within 6 hours and become local passengers).
• The reference and the premise with crews (stage 2 and 2b). Every best plan replays exactly with a legal crew on every flight, and an independent re-simulation reproduced the 26 eval and control dates exactly. v2's best plan costs at least 5% more than it on 17 of 24 eval dates (12 needed).
• The misses, documented and shipped: the connecting shares (above); the scripted customer desk failed its gate (above); the bound is weak (54% to 79% under the best plan on the eval dates, and negative on the calm winter control); and the best plan's repair overran its time cap on 49 of 70 dates.
• The desks. Before any model ran, scripted desks played airops-occ3 in process on 4 training dates in both modes (scripts/occ3_checks.py): every day graded, audited and replaying byte for byte, no page or reply before the end showing a reference, the bound or a reward, the day played with every reference moved giving every page, reply and record byte for byte, and the day with every delay a desk couldn't yet know changed giving everything up to each watch byte for byte. A Sol pilot on 2 training dates scored all 4 runs for $6.20; then Sol's board scored all 48, and the five other models' boards all 120, GPT-5.6 Terra's 7 of them on a retry. All 168 replay exactly through the env's calls on Linux and pass the audit.

Watch a v3 run

The Space replays the board's 168 runs from the dataset, ahead of v2's and v1's: each model's three desks and Sol's one agent on the 24 eval dates, each card naming its model and mode. A v3 run opens in the control-center room, as v2's do, and adds the day as its passengers see it:

• The Terminal view, a camera (key 9): JetBlue's gate hall at its busiest gate, the crowd at the glass, the aircraft at the bridge and the gate's screen with its flight on time, delayed, boarding, held for its connecting passengers, cancelled or departed. ‹ › (or , and .) steps from gate to gate and to the line at the service counter, and the view moves on when a gate empties. Start here opens the featured run, Sol's three desks at JFK on 8 July 2025, in the room, and its second link goes to the same run's Terminal view at a gate held for its connecting passengers.
• The control-center room: on a three-desk day the camera holds the customer desk, ops and crew together. The customer desk's requests travel to ops as cards and pile at the desk they wait on until the answer comes back; the customer desk's own screens show the connections at risk, the passengers rebooked and waiting to fly and those stranded overnight by airport; the video wall adds a PASSENGERS panel with the missed connections, the rebooked, the stranded and the requests to ops still open.
• The airport, as in v2, with the passengers: connecting passengers walk gate to gate and the stranded leave on the hotel shuttle at the curb.
• The US map draws where the passengers off a cancelled flight or a missed connection are rebooked to, wider for more of them, and how many are stranded overnight at each airport, beside v2's crews.
• Films, Sol's three on the v0.4.1 release, cut between the terminal, the room and the airport: the line at the counter after a cancellation, a gate held for its connecting passengers, a missed connection.

The gate halls, the lounges, the counter, the passengers' walks and the shuttle are inferred, for display only: the gate area is a generic one, not JetBlue's real Terminal 5 or Terminal C.

The dataset and the Space

scripts/hub_dataset.py builds earakely-scale/AirOpsEnv-AgentEnv from the packs and the recorded runs under results/runs:

• v3's tables: v3_tasks (the 70 dates with their passengers and their references under P, the default), v3_references (every candidate the best plan was chosen from, the bound and the gap, and the best plan itself), v3_episodes (every recorded three-desk run, replayed, with its audit), v3_desks (each desk's transcript) and v3_board (one row per model and mode, the three-desk rows ranked by their mean, then B0 and Sol's two modes paired by date); the eval card results/eval-v3.json; and the bundle airops-occ3, the 70 dates in both modes and wiring-noplay, so that agent-env hf run earakely-scale/AirOpsEnv-AgentEnv --bundle airops-occ3 plays a v3 day;
• v2's, as at v0.3.x: v2_tasks, v2_runs, v2_desks, v2_board, results/eval-v2.json and the bundle airops-occ;
• v1's, as at v0.2.0: v1_tasks, results, v1_references, v1_episodes, and agentenv-hf's airops_tasks, airops_episodes and the bundle airops, the dataset's default bundle, so that agent-env hf run earakely-scale/AirOpsEnv-AgentEnv with no --bundle plays a v1 day;
• runs/, the run folders as recorded, with each day's record and each desk's transcript;
• viewer/: the versions' episodes as the viewer reads them, with their traffic, the airport stages, the radar frames, the hub twins and the list's stills.

agentenv-hf scans every file for keys and token shapes first.

python scripts/occ_films.py results/runs/occ3-board-sol --out cache/films --assets cache/occ/viewer-release --final
for m in astra opus sonnet terra kimi; do
  python scripts/occ_viewer.py results/runs/occ3-board-$m --out cache/films --assets cache/occ/viewer-release
done
python scripts/hub_dataset.py sol-v1 --occ3 occ3-board-sol --occ3 occ3-board-astra --occ3 occ3-board-opus \
    --occ3 occ3-board-sonnet --occ3 occ3-board-terra --occ3 occ3-board-kimi \
    --occ-viewer cache/films/occ3-board-sol/viewer --occ-viewer cache/films/occ3-board-astra/viewer \
    --occ-viewer cache/films/occ3-board-opus/viewer --occ-viewer cache/films/occ3-board-sonnet/viewer \
    --occ-viewer cache/films/occ3-board-terra/viewer --occ-viewer cache/films/occ3-board-kimi/viewer \
    --occ-viewer cache/occ/viewer-release --workers 6 --tag v0.5.0 \
    --out cache/hub/dataset-v0.5.0 --space-out cache/hub/space-v0.5.0
python scripts/hub_dataset.py sol-v1 --occ3 occ3-board-sol --occ3 occ3-board-astra --occ3 occ3-board-opus \
    --occ3 occ3-board-sonnet --occ3 occ3-board-terra --occ3 occ3-board-kimi \
    --occ-viewer cache/films/occ3-board-sol/viewer --occ-viewer cache/films/occ3-board-astra/viewer \
    --occ-viewer cache/films/occ3-board-opus/viewer --occ-viewer cache/films/occ3-board-sonnet/viewer \
    --occ-viewer cache/films/occ3-board-terra/viewer --occ-viewer cache/films/occ3-board-kimi/viewer \
    --occ-viewer cache/occ/viewer-release --workers 6 --out cache/hub/dataset-v0.5.0 \
    --repo earakely-scale/AirOpsEnv-AgentEnv --tag v0.5.0

occ_films.py builds each scored run of Sol's board into a viewer episode with its passengers, replayed for each watch's projections, films it and picks its list still, into cache/films/occ3-board-sol/ (the episodes under viewer/); occ_viewer.py does the same for another model's board without filming it, into cache/films/occ3-board-<model>/viewer. --assets names a folder with the airport stages, hub twins and radar frames, such as v2's. --occ-viewer takes v3's folders, Sol's first and then the others in the board's order, then v2's (scripts/occ_episodes.py --out), and lists them in that order. hub_dataset.py replays every recorded control-center day, v2's and v3's, through the env's calls (a few minutes a day, --workers at once, kept under cache/occ/replays/ so a rebuild replays only what changed). Without --occ it takes every folder under results/runs with a pilot.json for v2, and without --occ3 every such folder named occ3-* for v3; v2's retry folder counts as v2's board. Each occ3-board-<model> folder is a v3 board of its own, ranked on v3_board by its three-desk mean, and occ3-board-sol's two modes are the ones paired; the card names each run no attempt scored, and why, and --occ3-running MODEL names a model whose board is still running.

The replay Space is static: hub/space/ and the viewer (web/) under viewer/, reading the episodes straight from the dataset's resolve URLs at the tag, which huggingface.co serves with CORS headers. --space-out assembles it and --space-repo pushes it.

AirOpsEnv v2: the ops control center

v2, released in v0.3.0, runs the same network from two desks, the ops controller and the crew scheduler, with no passengers on board: a cancelled passenger costs a flat 523 passenger-minutes. It ships unchanged in this release: agent-env airops-occ, the bundle airops-occ, the pack b6-v2 and its tables in the dataset.

AI agents run JetBlue's operations control center through a real storm day, split between two desks the way airline control centers split the work. The ops controller runs the flights and aircraft with the FAA: it cancels flights and round trips, holds them, and claims and swaps arrival slots in the FAA's ground delay programs. The crew scheduler runs the pilots and flight attendants: it calls reserves, swaps crews between trips, deadheads them as passengers and rests them in hotels, with pilots under FAR 117 and flight attendants under 14 CFR 121.467. Neither can do the other's job, so they talk, through typed requests the other desk accepts or declines (hold this flight, cancel that round trip, cover this one) and free text. The day is a real one: one of 24 storm dates at JFK or Boston from 2022 to 2026, the whole network from 04:00, about 700 flights, 220 aircraft and 900 crews and reserves, with the FAA Command Center's ground stops and ground delay programs arriving as they were issued. It plays out on AgentEnv's virtual clock, watch by watch, and is graded once, on the day as it really happened, with no judge: against the best plan of aircraft and crews together that a solver found knowing the whole day, next to the lower bound it proved. Every recorded run replays exactly through the env and passes its audit, and replays in 3D in the Space: the control-center room where the desks decide, and the airport where it plays out.

Contents: The control center · How a day is played · The days · Grading · The Sol board · Watch a run · Run it · How it was validated

The control center

	Ops controller	Crew scheduler
Sees	the network as projected now on what is known: every flight, the aircraft, the FAA's programs and slots, the cost; each flight's crew status from the crew system (legal, times out at HH:MM, waits for its crew until HH:MM, no crew)	the same picture, plus every crew: its trip, duty clock and rest, the reserves at each base, the hotels; the crew system's own alerts
Does	cancel, cancel a round trip, hold to a time, claim an FAA slot, swap two flights' slots, release a hold or claim	assign a crew to a flight, call a short-call reserve, swap two crews' remaining trips at a station, deadhead a crew, rest a crew in a hotel, release a crew it moved
Asks the other desk	COVER (crew this flight by a time), TEXT	HOLD (a flight until its crew is ready), CANCEL, CANCEL_PAIR (a round trip), TEXT
Can't	move or assign crews	cancel or hold flights

• The channel is part of the env. A request is a tool call; the env logs it with its sim time, its answer and the actions that followed, and carries out an accepted HOLD or CANCEL as the ops desk's decision. A request lapses when its flight leaves, is cancelled or its time passes; a newer one on the same flight replaces it.
• Turns. Each hourly watch, from 04:00 until the last flight's deadline, the crew desk takes its turn, then ops, then the crew desk again if ops asked it something. A turn opens on the desk's page (the day as projected now, about 6,000 characters) and ends with end_turn, whose result is the next page. The env runs the turns, so every run replays exactly.
• The crew system crews every flight the crew desk leaves alone, by standard practice (its own crew, else a reserve, a swap, a deadhead or the 2 h pilot extension), and can't cancel: where it is stuck it tells the crew desk, and the flight waits for a crew or its deadline. Aircraft swaps happen on their own: aircraft of one type at a station are interchangeable.
• What the desks know. A flight's own delay becomes known 2 hours before its scheduled departure; FAA advisories arrive as issued, and the FAA's CDM refuses a slot claim outside the program's pool. A hold can start no earlier than the freeze line, an hour ahead, and flights due out before it are committed.
• One agent (mode one) runs both desks from one page and both tool sets, with no channel: the same day, the same grading, for comparison.

Desk	Tools
ops	ops_flights, ops_program, ops_aircraft, ops_check, ops_confirm, ops_request, ops_reply
crew	crew_flights, crew_crews, crew_reserves, crew_program, crew_options, crew_check, crew_confirm, crew_request, crew_reply
both	end_turn, wait_turn, resign; a *_check tries changes on the projected day without committing them, a *_confirm commits them

Each desk is its own A2A agent (airops-desk, in role ops, crew or both) and sees only its role's tools on the AgentEnv gateway. The harness plays a model on Anthropic Messages, OpenAI Responses or OpenAI-compatible chat, under a cost cap per desk, or a fixed script with no model.

How a day is played

flowchart TD
  subgraph setup["1 · Set up the day"]
    direction LR
    deploy["<b>deploy_env</b><br/>airops-occ behind<br/>the AgentEnv gateway"] --> load["<b>apply_server_config</b><br/>occ-load: the day<br/>frozen at 04:00"] --> roles["<b>modify_env_tool_access</b><br/>each desk's role<br/>sees its own tools"] --> watches["<b>register_env_triggers</b><br/>one trigger per watch<br/>with FAA advisories"] --> agents["<b>deploy_agent</b> × 2<br/>airops-desk as<br/>crew and as ops"] --> clock["<b>sync_env_clock</b><br/>04:00, rate 0"]
  end
  subgraph play["2 · prompt_agent play-crew-desk and play-ops-desk, concurrently"]
    direction LR
    crew["crew turn<br/>crew_check · crew_confirm<br/>crew_request · crew_reply"] --> ops["ops turn<br/>ops_check · ops_confirm<br/>ops_request · ops_reply"] --> reply["crew reply turn<br/>if ops asked"] --> next{{"end_turn opens<br/>the next watch: an hour on;<br/>its trigger brings the<br/>FAA's advisories"}} --> crew
  end
  subgraph finish["3 · Grade the day"]
    direction LR
    endday["<b>apply_server_config</b><br/>end-day: the rest of the day<br/>runs on the desks' decisions"] --> grade["<b>env_outcome_verifier</b><br/>reward vs the best plan;<br/>audit A1–A7"]
  end
  setup --> play -->|"both desks done"| finish

The audit scores a run only if every watch's news arrived on time (A1), every call was made in its desk's turn (A2), no decision reached back before its minute (A3), each desk kept to its role (A4), every flight flew with a legal crew (A5), the day's cost is not under the proven bound (A6), and both desks were present to the end, or stood down (A7).

The days

The pack b6-v2 is 70 of JetBlue's real days, each its whole network from 04:00 Eastern:

Split	Dates	What
eval	24	storm dates, drawn 12 per storm hub (8 summer and 4 winter each), every one from its own storm event
train	44	storm dates of events with no eval date, so no storm is in both
control	2	calm days: 2024-07-03 and 2025-01-15, no FAA program at JFK, Boston, Newark or LaGuardia

drawn_for in v2_tasks is the hub a date was drawn for, hub the airport the viewer shows; a date can be a storm date at both. An eval date has a median of 696 flights (494 to 774) on 218 aircraft to some 85 airports, flown by 364 pilot crews and 356 cabin crews on generated trips, with 82 pilot and about 100 cabin reserves at the bases. A storm date is one where the FAA's programs at JFK or Boston controlled or held at least 5 JetBlue flights, from 19 summer and winter months between December 2022 and July 2026.

• Real, as recorded. Every JetBlue flight and tail from the BTS on-time records, with its scheduled times, its own delay and block time; aircraft types and seats from Form 41; where each aircraft really was at 04:00; the FAA Command Center's ground stops and ground delay programs at JFK and Boston as issued, and as they ran; other carriers' flights at the two hubs; the rules (FAR 117's Tables A and B, the 2 h extension and 10 h rest; 121.467's 14 h scheduled duty and 10 h rest; 121.391's cabin minimums); the crew bases by date; the costs (the FAA's rates for aircraft and pilot time, BLS wages for flight attendants, GSA hotel rates, JetBlue's average fare for a deadhead seat).
• Modelled. The FAA's slots: arrival slots shared out by schedule over every carrier's arrivals plus an unseen stream, with EDCTs, compression and ground-stop holds. The crews: JetBlue's pairings are not public, so a cost-optimizing builder generates trips from the published schedule, separately for pilots and flight attendants, and reserves are set at the middle of a sourced industry band.
• Not modelled, and labelled as such: union contract rules, multi-day limits (crews start the day rested), passengers' connections and rebooking, maintenance, dispatch, gates, and other airlines as actors.

FIDELITY.md says what is real data, what is modelled and how each layer was checked against the record, with every miss.

Grading

The day's cost, in passenger-minutes: each passenger's arrival delay; each cancelled passenger (523, more when the cancellation comes under 2 hours before departure); the FAA's rates for aircraft time on the ground and in the air, and tonight's ferries; crew deadheads (pay and the seats they take), hotels away from base, and crews left out of place for tomorrow's trips.

Reference	What it is
best plan	the cheapest day of aircraft and crews together a solver found knowing the whole day, replayed exactly with a legal crew on every flight; it scores 1.0
bound	a proven lower bound on any day's cost; the gap between them is published per date (eval dates 2.4% to 13.9%, median 5.1%)
B0	standard crew-desk practice with the aircraft doing nothing: each flight's planned crew, else the cheapest of a reserve, a swap, a deadhead or the 2 h pilot extension, cancelling only when no legal crew can arrive in time; it scores 0.308

The reward is 0.2 + 0.8 exp(−g/0.5), with g = max(0, cost − best) / max(B0 − best, 0.1 best): 1.0 at the best plan, 0.308 at B0, and never below 0.2. reward_L is the same against the bound, a lower reading where the best plan is weak. The avoidable share is the part of B0's excess over the best plan a run saved. Since no aircraft-and-crew optimum can be proven at this size, a run can come in under the best plan; it then scores 1.0 and is flagged.

The Sol board

GPT-6.1 Sol (openai/gpt-6.1-sol, reasoning effort high) on the 24 eval dates, once in each mode, pre-registered before the first run: the code, the pack's hash, the dates, their order, the caps and the budget. Every run scored, replays exactly through the env's calls and passes the audit, and an independent recheck recomputed every reward from the graded costs.

Mode	Mean reward (95% CI)	reward_L (95% CI)	Over the best plan, median	Avoidable cost saved, median	Spend
Two desks, talking	0.598 (0.545–0.658)	0.523 (0.484–0.564)	12.3%	60%	$48.33
One agent at both desks	0.550 (0.497–0.609)	0.487 (0.450–0.526)	13.8%	57%	$24.54
Standard crew desk (B0)	0.308			0%

• Two desks against one agent, paired by date: +0.049 reward (95% CI 0.024–0.075), two desks higher on 18 of 24 dates; their days cost 2.9% of the best plan less (CI 1.4–4.6%).
• Both far above standard practice, both far from the best plan: at the median a Sol day saves about 60% of what B0 leaves on the table and costs 12 to 14% more than the best plan. There is room to grow.
• CIs are bootstrap over the 24 dates (10,000 resamples). Spend is the harness's tokens times price; the whole board cost $73.94 with a superseded attempt, the pilot before it $6.88.

Date	Storm hubs	Flights	Under FAA programs	JetBlue cancelled	Gap, best to bound	Two desks	One agent	Two desks, reward_L	One agent, reward_L
2022-12-07	JFK, BOS	767	32	6	4.7%	0.465	0.408	0.426	0.385
2023-06-06	BOS	774	5	2	5.8%	0.710	0.595	0.525	0.471
2023-06-16	JFK, BOS	753	31	47	5.7%	0.576	0.510	0.518	0.469
2023-07-08	JFK, BOS	693	27	26	4.7%	0.594	0.495	0.556	0.472
2023-07-09	JFK, BOS	733	87	211	4.4%	0.471	0.482	0.453	0.463
2023-07-13	JFK, BOS	750	12	15	13.8%	1.000 *	0.967	0.578	0.537
2023-07-29

The higher of the two in bold. B0 scores 0.308 on every date. * Under the best plan, so 1.0 and flagged; reward_L is the fairer reading. "Under FAA programs": JetBlue flights the programs at JFK and Boston controlled or held. 2026-07-09: the registered retry.

What the results show, and what they don't

• They show that on these 24 storm dates two Sol desks beat one Sol agent running both, by a margin whose CI excludes zero, and that both recover most of what standard crew-desk practice loses.
• They don't show that talking is what helps. The two desks used about twice the compute: 1.9× the spend, 2.0× the input tokens and 2.5× the output tokens, more on every date (the caps were $5 a desk against $8 for one agent, and no desk came near its cap). The board can't separate two desks from more compute. The run that can, two desks with the channel off, and each desk against a scripted partner, come next.
• The scripted gate for talking didn't pass. Before any model ran, scripted desks measured whether coordination is worth at least 5 points of the avoidable cost on 60% of eval dates: it was on 3 of 24. Scripted desks capture about 14% of the room with or without the channel. There is room (at least 10% to gain on 21 of 24 eval dates, median 30%), and whether talking unlocks it for model desks is open.
• One model, one run per date and mode. More models come once v2's board has settled.
• Flagged: on 2023-07-13 the two desks came in 1.5% under the best plan (reward 1.0), whose bound is 13.8% lower; reward_L, 0.578, is the fairer reading there. On 2026-07-09 the first two-desk attempt lost the crew desk's connection at 17:00 and was scored as if it had stood down; the harness now records that as an error, and the registered retry (0.464) replaces it on the board. The attempt stays recorded and flagged in the dataset.

Watch a run

The Space replays all 54 recorded control-center days (the board's 48 and the pilot's 6) from the dataset, and v1's runs after them. Each replays the run's own day, watch by watch:

• The control-center room, in 3D: tiered desk rows facing a video wall of the network map, the radar and the FAA's programs, with the two desks at their pods in the front row. Each request rises at the desk that sends it and travels to the other as a card, then its answer comes back; each desk's words at the end of its turn appear at its pod.
• The airport, in 3D as in v1, now with its crews: pilots and flight attendants in uniform walk from the crew room to their gates, ride the apron bus to remote stands, board as passengers when deadheading, and take the hotel van when their duty ends away from base; before pushback a first officer walks round each departing aircraft. An aircraft waiting for its crew sits at the gate tagged "waiting for crew", and tags show the crew clock ("crew times out 22:40"). Every airline's flights move on their real times; the task's flights follow the desks' decisions. The crew camera (key 7) stays with a crew.
• The US map: the radar and the aircraft in the air, and the crews: resting in hotels away from base, the reserves still on call at each base, crews riding as passengers.
• Expert panes: each crew on duty against its legal limit (FAR 117 or 121.467), the full log of requests between the desks with their answers, and the page each desk read at its turn.
• Films cut between the room when the desks decide or talk and the airport when it plays out.

The room is modelled on public photos of airline system operations centers; it, the crew room, the crews' walking paths, the first officers' walk-rounds, the apron bus and the hotel-van routes are display-only (none of them is public data), like v1's stands and taxi routes. Neither the env nor the grade sees them.

python scripts/occ_episodes.py occ-board-sol --replay --out cache/occ/viewer --space-out cache/occ/space
python -m http.server 8742 --directory cache/occ/space      # http://127.0.0.1:8742/
python scripts/record.py --episode cache/occ/viewer/episodes/NAME.json --director --out cache/films/NAME.mp4

occ_episodes.py replays each recorded day through the env's calls, checks it, and exports it as the viewer's episode JSON (web/episode.schema.json) with the hub's traffic, its crews and the radar; it needs the raw inputs scripts/build_pack.py downloads and the airport stages (scripts/build_stage.py BOS JFK). record.py --director films it, cutting between the room and the airport.

Run it

You need Docker (running, with buildx), uv and git. Install the plugin with the framework, build the env and the desk agent and register them, and check the wiring with a day no model plays:

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/[email protected]"
agent-env airops-occ setup                    # builds airops-occ and airops-desk for this machine's Docker
agent-env airops-occ tasks --noplay           # writes results/bundles/airops-occ: each eval date, joint and one
agent-env run results/bundles/airops-occ --task wiring-noplay

wiring-noplay loads the first eval date and runs it with no desk: the crew system alone, then the grade. (v0.3.1 fixes a clean install of v2's CLI: v0.3.0 left numpy, pandas, highspy and airportsdata, which airops-occ needs, to extras.) Then play a day with a model on the endpoint [model] in .agentenv/config.toml names (LiteLLM, or the Hugging Face router):

agent-env run results/bundles/airops-occ --task b6-20240713-0400-joint --model openai/gpt-6.1-sol
agent-env run results/bundles/airops-occ --task b6-20240713-0400-one --model openai/gpt-6.1-sol

Or play a day straight from the dataset, which pins this release and holds the bundle airops-occ: every date in both modes, the very task files the board ran (each desk's cost cap $5 in a two-desk run, $8 for one agent), and wiring-noplay. The dataset's default bundle is v1's airops, so name this one:

agent-env hf run earakely-scale/[email protected] --bundle airops-occ --task wiring-noplay
agent-env hf run earakely-scale/[email protected] --bundle airops-occ --task b6-20240713-0400-joint \
    --model openai/gpt-6.1-sol

hf run downloads the dataset card and bundles/ at the tag's commit, checks that the plugin the card's agentenv table names is installed, shows what it will run and asks first; agent-env airops-occ setup must have built the env and the desk agent.

A pilot plays runs in a fixed order under a budget, retries a run that failed for an infrastructure reason at most twice, and reports every scored run replayed and audited; this is how the board ran:

agent-env airops-occ pilot my-board --run b6-20240713-0400-joint --run b6-20240713-0400-one \
    --cap-joint 5 --cap-one 8 --budget 25 --request-margin 0.75 --parallel 2
agent-env airops-occ report my-board          # results/runs/my-board/report.json

tasks --split train writes the 44 training dates, --split control the calm days, --cap sets each desk's cost cap, and --policy script plays the scripted desks with no model. A two-desk Sol day took a median of 37 minutes and $2; one agent, 20 minutes and $1.

How it was validated

v2 was built in stages, each checked against the real record before the next, with its checks pre-registered before it ran; a check that failed was reported and the call to ship or stop made once, without fix loops. FIDELITY.md has the full report.

• The world without crews. JetBlue's own decisions, replayed in the world, fly what JetBlue flew: 1 forced cancellation in 116,986 flights over 181 storm dates. The modelled FAA program delay of other carriers matches their recorded delay within 0.8–1.25×, pooled per hub, out of sample on stage 1's eval dates (JFK 0.95, Boston 1.10). Every hindsight anchor was proven optimal.
• The crews. JetBlue's real day, crewed by the generated trips and reserves, leaves 264 of 116,517 flown flights (0.23%) without a legal crew. Dates where the crew model fails (more than 1% of flights uncrewed in the replay, or crews adding more than 2% to a calm day) were filtered out by a rule fixed before the run: 100 of 180 storm dates remain, enough for the eval and train splits. Every best plan replays exactly with a legal crew on every flight, and every bound is valid.
• The desks. Scripted desks through the real harness at no model cost replayed exactly; then a 6-run pilot and the 48-run board, all scored, replayed and audited.
• The misses, documented and shipped: the calm-day check passed on 85.6% of dates against a 90% bar (the per-date cap was not rescaled for whole days); on one calm control doing nothing ran 8.9% over the optimum because of one flight's real 733-minute delay; the standard crew desk missed the controls' 3% margin (+4.0% and +17.4%); the crew filter kept somewhat smaller storms (median 20.5 JetBlue cancellations against 28 for the dropped dates), with similar room for the crew desk; and the scripted coordination gate failed (above).

AirOpsEnv v1: one desk at one hub

v1, released in v0.2.0, is one operations desk at one hub in the 3 to 6 hours of a storm day where planning matters most, with aircraft only. It ships unchanged in this release: agent-env airops, the bundle airops, the pack b6-v1 and its tables in the dataset.

The task

An agent runs the JetBlue operations desk at one of its hubs, JFK or Boston, on a real day that went wrong: thunderstorms, snow or wind, and the FAA ground stops and ground delay programs that followed, cut how many JetBlue flights could land and take off each hour. Watch by watch, on AgentEnv's virtual clock, it cancels, restores and holds flights so that passengers lose as little time as possible. The FAA Command Center's advisories, the hub's METARs and TAFs and the forecasters' discussion reach it as they were issued; each flight's delay and each hour's slot cap become known as they would have. After the last watch the plan is flown on the day as it really happened and graded, with no judge, against the plan a MIP solver proves optimal in hindsight.

tool	what it does
get_situation()	the time, the freeze line, the slot caps as known for the next 8 hours, the FAA programs in effect, the latest METAR and TAF, every flight (its status, its delay once known, the plan, where the plan has it leave and land on the day as known, the aircraft it takes) and where the aircraft are
check_plan(changes)	runs changes over the plan on the day as known without committing them; one of the watch's 3 planning calls
confirm_changes(changes)	commits them; also one of the watch's 3 planning calls
advance()	ends the watch: the clock moves an hour and flights leave as planned

A change is {"flight": "f12", "action": "cancel" | "restore" | "hold", "not_before": "HH:MM" | null}, every field required and not_before nullable, so the schemas pass OpenAI's strict mode.

• Watches. The first watch is an hour before the disruption window; a watch every hour after it, until every delay and cap is known (or no flight is left to decide). Time only passes when the planner calls advance.
• What the desk knows. A flight's own delay (anything but waiting for its aircraft) is known 2 hours before its scheduled departure, assumed zero before. An hour's slot cap at an airport is known 4 hours ahead; before that JetBlue's own schedule for the hour stands in. News arrives at the first watch at or after it was issued.
• The freeze. Nothing is frozen at the first watch. From the second on, flights the plan has leaving before the freeze line, an hour after the watch, are frozen and stay frozen; the env refuses changes to them, and holds any other flight the plan would have leaving before the line to it (held_to_freeze_line), so no decision reaches back past it.
• Execution. The env dispatches the aircraft: a flight takes the first-ready aircraft of its family at its origin, leaves when its delay, its hold, the aircraft's turn and both hours' caps allow, and is cancelled if it can't leave within 6 hours of its schedule. At the end of the day aircraft are ferried to where tomorrow needs them.

flowchart TD
  subgraph setup["1 · Set up the day"]
    direction LR
    deploy["<b>deploy_env</b><br/>airops-day behind<br/>the AgentEnv gateway"] --> load["<b>apply_server_config</b><br/>live-load: the day<br/>as known at the first watch"] --> hide["<b>modify_env_tool_access</b><br/>hide airops_notice<br/>from the agent"] --> watches["<b>register_env_triggers</b><br/>one trigger per watch<br/>with news"] --> agent["<b>deploy_agent</b><br/>airops-llm"] --> clock["<b>sync_env_clock</b><br/>the first watch, rate 0"]
  end
  subgraph play["2 · prompt_agent play: one conversation, watch by watch"]
    direction LR
    plan["get_situation<br/>check_plan · confirm_changes<br/>3 planning calls a watch;<br/>flights leaving within<br/>the hour freeze"] --> advance["<b>advance</b><br/>the clock moves an hour;<br/>flights leave as planned"] --> trigger{{"trigger watch-k<br/>calls airops_notice:<br/>FAA advisories, METARs,<br/>TAFs, the forecast discussion"}} -->|"news arrives with<br/>the next tool result"| plan
  end
  subgraph finish["3 · Grade the day"]
    direction LR
    endday["<b>apply_server_config</b><br/>end-day: the rest of the day<br/>runs on the final plan"] --> grade["<b>env_outcome_verifier</b><br/>reward vs the hindsight optimum;<br/>audit: every watch's news on time"]
  end
  setup --> play -->|"after the last watch: done"| finish

The data

A task is one hub window on a real day of the pack b6-v1 (JetBlue, JFK and Boston), built from 13 months of 2022 to 2025: 20 eval days, 3 control days (no cancellation in scope) and 34 training days, with 59 to 194 flights on 30 to 59 aircraft and 10 to 22 watches. The eval days span 7 months of thunderstorms and winter storms, 12 at JFK and 8 at Boston. Each day's window is the 3 to 6 hours where planning matters most: the largest gap between doing nothing and the hindsight optimum, among windows where every flight has its aircraft, weather and air traffic control caused at least 80% of the cancellations, and 30 to 80 aircraft are in play. An eval day also needs doing nothing to cost at least 10% over the optimum, the follower to land within 15% of it, and the optimum proven within 60 seconds.

Everything comes from public U.S. records: the flights, tails and delays from BTS on-time data, passengers from T-100, aircraft from the FAA registry and Form 41, the FAA Command Center's advisories, and the METARs, TAFs and forecast discussions archived by the Iowa Environmental Mesonet. scripts/build_pack.py builds the pack into data/b6-v1/, where each source has its SOURCE.md; DESIGN.md is the contract between the modules.

Grading

The final plan is flown on the truth (core.sim.execute). Its cost, in passenger-minutes, is each passenger's arrival delay, plus 523 per passenger on a cancelled flight, plus the ferries. The reward is 0.2 + 0.8 exp(−g/0.5), g = max(0, cost − anchor) / (anchor − floor + 0.02 anchor): 1.0 at the anchor, the hindsight optimum, and never below 0.2, since the env never strands a flight. A day that never finished or whose news broke the reveal schedule fails the verifier and is not scored.

Reference	What it is
floor	the time-line MIP without aircraft routing: each flight alone at its earliest time under the caps
hindsight (anchor)	the time-line MIP (HiGHS, 1-minute arcs) on the truth, proven to 0.01%
follower	the MIP re-solved at every watch on the day as known (persistence forecast of unknown delays and caps), freezing as the env does, its final plan flown on the truth
blind	the follower with no forecast
naive	the empty plan: every flight as early as it can
airline	JetBlue's own cancellations and departure times as the plan (shown, never the anchor)

scripts/references.py computes them into each task line; a day without them can be played but not scored. The follower's watch solves prove their optimum and then pick, among the optimal plans, the one with the earliest weighted departures, so the references come out the same on every run and machine.

Results

GPT-6.1 Sol on 5 eval days, one run each through the real DAG on local Docker (sol-v1, $2.99 in all). Every run finished, passed the verifier's audit, and replays byte for byte through the env's day; each replays in 3D in the Space.

Day	Flights	Watches	GPT-6.1 Sol	Follower	Blind follower	Doing nothing	JetBlue
b6-bos-20250216-1500	81	12	0.672	0.937	0.937	0.366	0.224
b6-jfk-20230714-1400	91	13	0.441	0.782	0.229	0.212	0.200
b6-jfk-20240804-0700	149	20	0.382	0.901	0.627	0.213	0.202
b6-jfk-20250709-1500	76	12	0.324	0.614	0.743	0.224	0.200
b6-bos-20240107-0600	188	19	0.276	0.417	0.436	0.220	0.202
Mean			0.419	0.730	0.594	0.247	0.206

Sol beats doing nothing and JetBlue's own decisions on every day, and stays well below the re-planning reference: on average its day costs 18.9% over the hindsight optimum. JetBlue's real day scores near the floor of the reward because it cancelled far more than the passenger-centric cost favours, with crews and information this model leaves out; it is shown, never the anchor. One run per day, so these are a first look, not a board.

Watch a run

The viewer replays a recorded run on the day it was played, from its first watch to the end of the day's activity at the hub, often past midnight:

• The airport in 3D fills the screen: Boston Logan or JFK modelled from OpenStreetMap (runways, taxiways, aprons, stands, terminals, jet bridges, the tower) over USGS aerial photos of the land around, with Boston Harbor or Jamaica Bay as water. The sun follows the clock and the weather follows the hub's METARs: rain, snow, drizzle, mist and the cloud deck. Every airline's flights at the hub land, taxi and take off on their real BTS times, other airlines' aircraft in neutral grey; JetBlue's are white with a navy tail. The task's aircraft follow the agent's plan, not JetBlue's day, and carry tags: the flight, where it goes or comes from, and whether it is on time, delayed, held or cancelled (a cancelled flight's aircraft stays at its stand). JetBlue aircraft outside the task fly their real times, untagged. The camera watches from the tower, a gate, a runway threshold or across the water, or follows an aircraft you click.
• The US map sits in a corner: the NEXRAD radar and the aircraft in the air. Click it to swap it with the airport.
• The departures board lists the task's flights at the hub as the plan has them.
• The desk tells each watch in plain English: the FAA's programs, the weather, what the agent changed and what the day as known now costs. A model's own words appear when its transcript has them; Sol's runs are tool calls only.
• Expert mode shows the desk as the agent saw it: the advisories, METARs and TAFs as issued, the slot caps, and every tool call with its arguments and result.

The stands, the taxi routes and the runway in use are the viewer's inferences, for display only: BTS records none of them, and neither the env nor the grade sees them.

agent-env airops view sol-v1                  # http://127.0.0.1:8238/web/
agent-env airops record sol-v1 --task b6-bos-20250216-1500 --director --seconds 66 \
    --out cache/films/sol.mp4 --webp cache/films/sol.webp

view replays every scored run of the sweeps through the env's day (agentenv_airops.episodes): each recorded tool call is played back in order and its output checked byte for byte against the recording, the watch's news delivered after its advance as the triggers deliver it. A run that doesn't replay (recorded before the env changed) is listed and left out. Each replay is exported as the viewer's Episode JSON (agentenv_airops.export, web/episode.schema.json): the schedule, the final plan flown on the truth, each watch as the desk knew it (its news, the caps, the changes, the plan's cost on the day as known), the tool calls and the truth's caps; and, when they are in cache/, the hub's 3D stage (scripts/build_stage.py BOS JFK, with the build extra) with the day's traffic, the NEXRAD radar frames (scripts/prebake_radar.py) and the hub's OpenStreetMap twin (scripts/build_twin.py). The traffic (scripts/prebake_traffic.py --sweep sol-v1) is every airline's movements at the hub on their real times, from the BTS on-time data and the FAA registry that scripts/build_pack.py downloads into cache/inputs/; each episode gets it as a sidecar with the run's plan laid over it (DESIGN.md gives its format). record films one with the checkout's scripts/record.py (headless Chrome and ffmpeg). --director cuts the day into shots on the 3D airport, in the order they happened: the hub across the water, the FAA's program reaching the desk, a decision landing on the board, a held flight pushing back, taxiing and taking off, and the day's score; --thumb and --still write the list's thumbnail and a frame without cards. --rollout films every watch instead, as PortSim's clip runs its week: each opens on the desk with the model's planning calls and what it did, then the field moves on, then the score. The dataset card's clip is --rollout --camera '{"from": [-720, 780, 55], "at": [-770, 470, 4], "fov": 50} --size 1600x900 --webp-width 1024 --webp-fps 10 --webp-quality 60 on the Boston run. Without either, the clock sweeps the day watch by watch on the map, and --cut MINUTE:VIEW switches the camera to tower, gate, threshold, harbour, follow or network at a minute of the day.

scripts/fixture_episode.py makes the same episode for the follower: its per-watch plans played through the env's tools and exported, under cache/episodes/.

Run it yourself

You need Docker (running, with buildx), uv and git. Install the plugin with the framework, build the env and agent images and register them, and check the wiring with a task that calls no model:

uv tool install agentenv-framework \
    --with "agentenv-airops @ git+https://github.com/earakely-scale/[email protected]"
agent-env airops setup --agent        # builds airops-day and airops-llm for this machine's Docker
agent-env run airops --task wiring-noplay

wiring-noplay deploys the env, loads the first eval day, registers its triggers and arms the clock, then ends the day without an agent: the env delivers every watch's news itself, the day flies with no change (the naive reference) and the verifier scores it. Then play a day with a model on the endpoint [model] in .agentenv/config.toml names (LiteLLM, or the Hugging Face router):

agent-env run airops --task b6-bos-20250216-1500 --model openai/gpt-6.1-sol

Or play one straight from the dataset, which pins this release:

agent-env hf run earakely-scale/[email protected] --task b6-bos-20250216-1500 \
    --model anthropic/claude-sonnet-5-5

hf run downloads the dataset card and bundles/ at the tag's commit, checks that the plugin the card's agentenv table names is installed, shows what it will run and asks first.

From a checkout, uv sync --extra dev --extra viewer (or pip install -e '.[dev,viewer]') and the same commands. --platform linux/amd64 on setup builds for remote sandboxes.

Sweeps

A sweep plays models over days, k reps each, one agent-env run per attempt, under a spend cap:

agent-env airops sweep run --name pilot --models anthropic/claude-sonnet-5-5,openai/gpt-6.1-sol \
    --tasks eval --k 1 --episode-cap-usd 2 --cap-usd 50 --parallel 4
agent-env airops sweep report pilot           # results/airops.md

• --tasks is eval, control, train, all (fewest watches first, so a spend stop drops the longest days) or comma-separated task ids; --pack points at another pack, which must be the one the env image holds.
• Each attempt is a row of results/runs/<name>/results.jsonl: the outcome, the verifier's reward, the day's cost and counts, the agent's end reason, turns, tool calls, tokens, spend and the cost of its priced tokens; its transcript goes to transcripts/ and its log to logs/.
• An attempt that fails is retried up to twice; one stopped at its episode cap or refused by the provider is final. No attempt starts that could take the spend past --cap-usd with every running attempt at its cap, and an attempt that recorded no spend counts at the cap. A missing model endpoint or price stops new attempts. Running the sweep again plays the runs that aren't final; Ctrl-C tears the running attempts down.
• The report gives each model's mean reward over days (a day's value is the mean of its reps) with a 95% bootstrap CI, next to the follower's, the blind follower's, naive's and the airline's rewards on the same days, then a table per day, the unscored runs and the runs whose agent reward differs from the verifier's.

Simplifications

AirOpsEnv v1 is a model of a day, not an airline's operations system. What it leaves out or assumes:

• One airline, two hubs, one window. Only JetBlue's flights touching JFK or Boston in a 3 to 6 hour window (and the rest of the day of the aircraft that fly them) are in play. Other carriers appear only through the caps.
• Caps are a share, not the FAA's rate. An hour's cap is JetBlue's ration-by-schedule share of the movements all carriers actually made at the airport, never below what JetBlue's in-scope flights moved there. The truth caps are what happened under the airline's own decisions; a different plan would not have changed them.
• Delays are exogenous. A flight's own delay is its departure delay less the late-aircraft part; a cancelled flight gets the 75th percentile of its neighbours'. Before it is known, the desk sees no delay at all.
• Aircraft are interchangeable within a family at an airport. No tail pinning, crews, maintenance, gates or curfews; turn times are each family's 5th percentile of the day's observed turns.
• Passengers are counts. Load factors from T-100, no connections or rebooking: a cancelled passenger costs a flat 523 minutes (NEXTOR's average re-accommodation delay plus refund exposure at the FAA's value of time), and a ferry 92 × (30 + miles / 8), an assumed block time at A4A's cost per block minute.
• Diversions count as cancellations in the airline's day; the env has no diversions.
• The freeze-line hold (above) is the env's, not DESIGN.md's: without it a planner could hold a flight past the line and release it once the truth is known.
• The weather is text. The planner reads METARs, TAFs, the forecast discussion and the FAA's advisories; the radar and the 3D weather in the viewer are for people.
• The viewer's airport is for display. The stands, the taxi routes and the runway in use are inferred (BTS records none of them); other airlines' traffic and JetBlue's aircraft outside the task are drawn on their real BTS times and play no part in the env.
• The anchor is hindsight. It knows every delay and cap in advance, so 1.0 is out of reach online; the follower, re-planning on the day as known, is the online reference. No excuse is given for news after a flight's freeze.

Repository

Path	What
src/agentenv_airops/programs/pax.py, programs/cost.py	v3's passengers: the generator, the day's rebooking pass, cost P, the MIP's passenger pricing and the capacity-aware floor
src/agentenv_airops/occ/{day3,customer,tasks3,live3,cli3}.py	v3's env airops-occ3: the three-desk day, the customer desk's page, lookups and requests, the task DAG and agent-env airops-occ3
scripts/pax_shares.py, data/pax/	v3's survey extracts: JetBlue's connecting flows per period and station, DB1C's connection times, the median fares
scripts/build_occ_pack.py --v3, data/b6-v3/	v3's pack: v2's dates with their passengers and their references under P
scripts/pax_step0.py, pax_stage2.py, pax_stage2_box.py, pax_bound.py, pax_stage2b.py	v3's validation runs (FIDELITY.md)
src/agentenv_airops/programs/	v2's world: the whole-network task, the FAA's programs and slots, the costs, the crews (trips, legality, the crew sim, the replays, the standard crew desk), the best joint plan and its bound, crew_params.json with every crew parameter's source
src/agentenv_airops/occ/	v2's env airops-occ: the day, the desks' pages and tools, the turns and the channel, the grade and the audit, the task DAG, pilots, replay, and the viewer's control-center episode
agents/airops-desk/	a desk as an A2A agent: ops, crew or both, on Anthropic Messages, OpenAI Responses or OpenAI-compatible chat, or a script
scripts/build_occ_pack.py, data/b6-v2/	v2's pack: each date's task, its crews frozen at 04:00 and its references
scripts/world_check.py, crew_check.py, occ_check.py, occ_z1.py, occ_z2.py	v2's validation runs (FIDELITY.md)

uv sync --extra dev --extra viewer --extra build
pytest                                   # offline; -m docker, -m browser and -m network run the rest
PYTHONPATH=tests python scripts/live_e2e.py   # a zero-spend v1 day through the real DAG on local Docker

Licence and attribution

Code: Apache-2.0 (LICENSE). The data: BTS, FAA and NWS records are U.S. Government works in the public domain; the Iowa Environmental Mesonet archives the METARs, TAFs, forecast discussions and NEXRAD composites (public domain; attribution appreciated, and given); airport names, coordinates and time zones come from airportsdata (MIT). v3's passengers come from BTS T-100 and the DOT's Airline Origin and Destination Survey (DB1B and DB1C), U.S. Government works; its rebooking rules are facts of JetBlue's Contract of Carriage and Customer Service Plan and DOT's refund rule (14 CFR part 260). v2's crew rules come from the eCFR (14 CFR parts 117 and 121) and the Federal Register, its costs from the FAA's economic values (Table 4-7), the BLS, and GSA and DoD per diem rates, all U.S. Government works; its crew bases, fleet configurations and deadhead fare are facts cited from JetBlue's 10-K filings and press releases, ALPA and Airline Pilot Central, each with its URL in src/agentenv_airops/programs/crew_params.json. The airport stages and the hub twins are © OpenStreetMap contributors (ODbL 1.0); the aerial photos under the stages are USGS The National Map orthoimagery (public domain). Both are built into cache/ and never committed; beyond the stage the viewer loads coarser USGS imagery and hydrography tiles from The National Map as it runs. The US map is Natural Earth relief (public domain) under OpenMapTiles vector tiles © OpenStreetMap contributors, served by OpenFreeMap. The 3D scene code in web/stage/ adapts PortSimEnv v1 (FineEnvs, Apache-2.0), named on the first line of each adapted file, and runs on Three.js (MIT); the map runs on MapLibre GL JS (BSD-3-Clause), and the control-center room's video wall on topojson-client, us-atlas and world-atlas (ISC), all vendored under web/vendor/ with their licences. See NOTICE and each data/*/sources/*/SOURCE.md. Neither JetBlue, the FAA, the BTS, the NWS, the USGS, the IEM nor any union is affiliated with this project or endorses it.