simhall
A virtual GPU hall for testing bring-up tooling without hardware. One ground-truth model serves the
devices a bring-up tool talks to, with fault injection and technician actions: Redfish BMCs, LLDP tables
on Arista (eAPI) and NVIDIA (NVUE) leaves, DCGM exporters, and a minimal NetBox API. Apache-2.0.
pip install simhall turnup-check
simhall serve --golden hall-b & # 127.0.0.1:8900, any token
turnup-check --site hall-b --intent http://127.0.0.1:8900/netbox
reading intent 288 links · 18 nodes · 2 leaves
polling lldp · redfish · dcgm
✓ 286 / 288 links match intent
✓ 17 / 18 nodes reachable, firmware off baseline
✗ mismatch node7 mlx5_3 → leaf4/22
expected leaf4/21
fix move cable one port up
✗ degraded node11 mlx5_5 → leaf4/83 200G
expected leaf4/83 up 400G
fix check transceivers and cable CBL-12-11-05 at leaf4/83 (200G negotiated, intent 400G)
✗ firmware node4 bmc 1.09.2
expected 1.12.0
fix update bmc on node4 to 1.12.0
✗ gpu node14 gpu2 XID 79
expected healthy
fix drain node14 and open an RMA packet for GPU 2
✗ offline node17 redfish unreachable: timeout after 3 s
fix check BMC reachability and credentials for node17
elapsed 3.2 s
The golden hall hall-b is a hall handed over with every light green: five faults of four kinds, one node
each. A technician moves the cable one port up, reseats the degraded one and applies the firmware update, and
the same command prints 288 / 288 links match intent, firmware baseline ok and the two findings that
stay open (the GPU with the XID, the BMC that does not answer) until the trays are replaced:
simhall act '{"kind":"seat_cable","label":"CBL-12-07-03","b_interface":"swp21"}'
simhall act '{"kind":"seat_cable","label":"CBL-12-11-05"}'
simhall act '{"kind":"update_firmware","device":"node4","component":"bmc","version":"1.12.0"}'
Everything here is synthetic: no real hostnames, IPs, serial numbers or topologies. The intent it emits
carries simulated: true, so a simulated run never counts as a real one.
Why
Bring-up tooling is tested on the hardware it is meant to verify, which is the hardware nobody can spare.
simhall gives a tool like turnup-check a hall that behaves
like the real protocols at every layer, where any fault can be injected, observed through every view at
once, and cleared again byte for byte. The test suite of turnup-check runs thousands of random halls and
fault sets against it in-process.
Install
pip install simhall # Python 3.12+
pip install git+https://github.com/turnup-dev/simhall # or straight from this repository
From a checkout: pip install -e '.[dev]', then simhall --help. Dependencies: fastapi, uvicorn,
pydantic v2, httpx, pyyaml, click; pynetbox only for simhall seed-netbox (.[netbox]).
Faults and what a tool should see
| Fault | Observable effect | Expected turnup-check finding |
|---|
wrong_port(cable, to_device, to_interface) | neighbor appears on the target port, intended port down; node Redfish port shows the target | wrong_port |
missing_link(cable) | intended port down, no neighbor anywhere; node port NoLink | missing_link |
swap_pair(a, b) | the two cables' leaf ports are exchanged | 2 x wrong_port |
link_degraded(cable, 200 | "down") | speed 200G on both ends, or link down with an aging LLDP entry | link_degraded |
unreachable_node(node, timeout | http_500 | refused) | Redfish and/or DCGM answer per mode; links stay up | unreachable_node |
firmware_off_baseline(node, component, version) | the inventory member's version changes | firmware_off_baseline |
gpu_unhealthy, gpu_missing, nvlink_degraded, node_health_warning | DCGM metrics and Redfish health change accordingly | the matching finding |
pdu_outlet_off(node) | BMC and exporter refused, host off, every link down | unreachable_node + missing_link per port |
coolant_leak, partial_redfish, lldp_port_id_mac | leak detectors Critical; FirmwareInventory 404; port ids as MACs | none, by design |
Actions (seat_cable, unseat_cable, power_on, power_off, replace_node, update_firmware,
scan_qr) change the hall the way a technician would and retire the faults they physically resolve.
Faults are applied on top of the base state and clearing one restores the views exactly, in any order.
The complete tables and the rules are in docs/faults.md.
Python API
from simhall import HallSpec, Hall, Fault, Action
hall = Hall.generate(HallSpec(site="hall-b", racks=1, nodes_per_rack=18, ports_per_node=16,
leaves_per_rack=2, switch_vendor="nvidia", seed=1))
hall.inject(Fault(kind="wrong_port", cable="CBL-12-07-03", to_device="leaf4", to_interface="swp22"))
hall.lldp_table("leaf4") # what the leaf would say
hall.redfish_tree("node7") # what the BMC would say
hall.act(Action(kind="seat_cable", label="CBL-12-07-03", b_interface="swp21"))
create_app(hall) serves that very object over HTTP, so in-process actions are visible to a tool polling
it. The model, the cabling rule and how state is derived: docs/model.md.
Documentation
License
Apache-2.0. See LICENSE.
