The circuit breaker your Solana protocol's own admin can't turn off.
Bulkhead is a small Solana program that holds a protocol's token accounts on the protocol's behalf, and rate-limits every payout from inside the transaction. A stolen admin key, a buggy payout path or malicious new code can still order a drain. Bulkhead decides how fast it is paid.
The attack · How it works · Verify it · Devnet · FAQ · Docs
On 1 April 2026, Drift lost $285M in about twelve minutes. Drift had withdrawal limits. The attacker, holding admin power, raised them. Drift had removed its own timelock five days earlier.
A model, not a replay (reference/drift_model.py): Chainalysis per-asset amounts and an assumed $550M TVL, with vaults proportional to what was drained. The 2-minute freeze is an assumption too: the guardian freezes in 5–20 s on a local validator, not yet measured on mainnet. Lines join modeled points.
[!WARNING]
Not independently audited. Not for real funds. Bulkhead runs on devnet. Until an audit, its upgrade key is a disclosed team key; after one, it is burned and nobody can change the program. Two internal reviews run the way audits are run, by AI agents, are in 16 and 17. Neither is an audit; see SECURITY.md.
Who it is for today: new launches and relaunches that route custody through it (treasuries, vaults, protocols without lending). Lending integrators wait for the decisions in 16 §4.
What it does to an attack
One run of cloud/e2e/run.sh on 17 Oct 2026, drawn from its own logs and played at 4× (the still: light, dark). Every time and transaction signature is in s1-run.json; timings depend on the machine. A payout's line ends when the attacker's client reports it, one RPC call after it lands, so the guardian sometimes cancels a ticket first.
Measured end to end, and reproducible with one command (E2E=1 scripts/check-programs.sh): a local validator, the programs, a 10M USDC vault, the Cloud guardian, and an attacker holding the demo protocol's admin key.
| |
|---|
| The attacker orders | 12 payouts of 100,000 USDC, each 800 ms after the last one confirmed |
| The attacker receives | 100,000 USDC. The burst is 2% of what is in the vault at each payout, so after the first 100,000 the second no longer fits. An attacker who sizes each payout exactly gets 199,000 (S8), never more than 2%. |
| The guardian | Freezes the protocol on its own, seconds after the second payout, then cancels all 11 queued payouts. With three guardians: 5.4–5.6 s after the first payout with a healthy RPC; 13–20 s with every RPC call delayed 1.5 s (09, finding 82) |
| Afterwards | 9,900,000 USDC still in the vault, 0 payouts pending, and the status page's figures equal the program's own attest |
With nobody responding at all, half a vault takes 9 days to leave instead of 12 minutes.
How it works
The protocol still decides who gets paid. Bulkhead decides how fast.
On Solana, the one control an admin or upgrade key can't get around is ownership of the token account. Bulkhead becomes that owner. Five rules follow:
- Your own money is never held. Taking back your own deposit is never throttled, frozen or priced. No oracle, no limit, no freeze.
- Everything else is rate-limited. Limits refill at 2% of the vault per hour and 5% per day. Of the payouts asked for in any hour, at most 4% are paid within that hour, and of a day's, at most 10% within that day. Anything above that waits at least 6 hours in a public queue any guardian can cancel, and is paid later, possibly in one go.
- Loosening takes 72 hours. Tightening is instant. Loosening waits 72 h, a constant no admin can lower, and any guardian can veto it.
- Guardians stop, never steal. Any guardian can freeze, veto or stop. None can move a single token.
- The upgrade key is fenced too. Floor holds the program's upgrade key. New code waits 72 h in public; one guardian can stop the program; a quorum restores it byte for byte.
- The base. The percentages apply to the least of the live balance, its 24-hour average and its lowest balance of the last 30 to 60 minutes. A deposit made just before an attack buys almost nothing, capital held for one transaction (a flash loan) buys nothing, and money that has left stops counting at once.
- Two buckets. Each rate refills a bucket of the same size, so of the payouts requested from any moment on, at most 4% are paid within the hour and 10% within the day. Above that, payouts wait at least 6 hours in a public queue that guardians can freeze and cancel.
- What the queue is not. The limits count payouts when they are asked for, not when they land. Queued payouts can come due together, so this is no cap on what leaves in a given clock hour: at the defaults one request can be 40% of the vault (due 7 days later) and tickets asked for over several days can settle in the same moment. What every such payout gets is at least 6 hours in public, in which any guardian can cancel it (
docs/09, finding 102).
- Caps. Set absolute caps in tokens too: they bound an attacker who parks their own capital in the vault first (15).
- The exception. A loan against allowlisted, Pyth-priced collateral is bounded by haircuts and Pyth, not by the limiter; a guardian's freeze stops new ones (threat model, T5; finding 90).
Two tiers, two threats
| Vault · Tier 1 | Floor · Tier 0 |
|---|
| Holds | The protocol's token accounts | The program's upgrade key |
| Stops | Money leaving too fast, whatever the cause: a stolen admin key (the Drift class), a buggy payout path, malicious new code in the protocol, even a one-transaction drain | Malicious new code: new binaries wait 72 h in public, any guardian can stop the program at once, and a quorum restores the exact bytes that were stopped |
| Does not stop | Outflow within the limits when nobody responds: up to 10% of the vault asked for and paid within a day, plus queued payouts once they are due, which can land together (finding 102). Guardians and alerts cover that. | A drain through the existing code's own admin instructions, which is how Drift was drained. Only Vault bounds that. |
| Adoption | The protocol routes deposits and withdrawals through it: easiest for new launches and relaunches | One transaction, zero code changes, for any upgradeable program (shown on a real 1 MB mainnet program: scripts/floor-demo.sh) |
Integrating
A protocol replaces two token transfers with two CPIs, signed by its own PDA ["bulkhead-caller"], which only its program can sign for. Everything else (limits, guardians, the queue) is configuration.
// deposit: the user signs as the source authority; the vault credits what arrived
invoke_signed(&deposit_ix, &accounts, &[Signer::from(&caller_seeds)])?;
// release: Bulkhead pays now, queues the rest as a public ticket, or refuses
invoke_signed(&release_ix, &accounts, &[Signer::from(&caller_seeds)])?;
let outcome = get_return_data(); // backed | instant | queued | ticket_id | eta_s
The checklist, the calibration method and the demo protocol that does exactly this (programs/harbor, 140 lines) are in 07. attest returns the live worst case to any program that asks, so a router or curator can refuse to route into a venue whose bound is too wide.
Who the guardians are. When a protocol registers, it names 4–8 guardian keys: for example its operations key, the Bulkhead Cloud bot and two security firms. Only guardians that heartbeat count. Any one of them can freeze, veto or stop, and none can move money. A live quorum of the others ejects a rogue guardian instantly (a new set has at least two keys more than its threshold, so after any two lost or rogue keys the others still reach one, unless the rogues are a quorum themselves: two in a 2-of-n set), so a stolen guardian key means denial of service until then, never theft.
Who pays. The core is free and open source. Protocols pay for Bulkhead Cloud: a hosted guardian, limit calibration on their own flows, and an attestation feed. They pay because the capital under the admin asks for it. Depositors, curators and allocators want a worst case that is enforced, not promised, and the program publishes it (attest): "at most $X now, $Y in the next hour, $Z in a day, on top of $Q already queued in public" through the unbacked lane, with absolute caps set. It excludes oracle risk on backed loans. A patient attacker who paces payouts under the bot's thresholds is bounded by the on-chain limits, not by the bot (15 A2).
Dean Little's sbpf-asm-abort showed that a program can be stopped by upgrading it to a tiny binary that always fails. Floor builds on that pattern (credited in DISCLOSURE.md). A bare abort, though:
- is pressed by whoever holds the upgrade key, the same key that can deploy anything;
- waits for the timelock under a timelocked multisig;
- is undone only by re-uploading the program.
Floor separates stopping from upgrading. Any guardian stops, new code waits 72 h, and a quorum restores the stopped code, verified byte-identical. No stop button, bare or Floor's, reaches inside a transaction, and none bounds a drain through admin instructions. That is Vault's job. Details: 06 §2. More questions: FAQ.
Why the brake has to sit where the money moves
| |
|---|
| 37 of 54 multisigs governing solgov-tracked Solana protocols have no timelock. The longest is 24 h; 16 of 43 have changed theirs. | Our census, snapshot 26 Sep 2026, spot-checked on-chain |
| In 183 DeFi hacks, 1 of 87 protocols with a pause used it within the first hour. 44% of attacks were a single transaction. | Zhou et al., IEEE S&P 2023 |
| Since 15 Sep 2026, one Solana transaction can be 4,096 bytes (was 1,232). | SIMD-0296 |
Timelocks are settings, not floors. Stop buttons act after the transaction that did the damage. Only a limit enforced inside every payout bounds an atomic attack.
Verify it yourself
Three levels, by the time they take. In seconds, with nothing to build (Node 20+ or Python 3.9+): the live devnet market, as the status page shows it, and who can upgrade any program.
node cloud/status/snapshot.mjs https://api.devnet.solana.com HK5FbaLtvNSPHj9HYbuStfVxXktkoMejJ9GEjn8WCY1b \
6w9t76W2w8q38xTvUPuDDzSwrjTKbq4dM8nAckYXScYh CBK36QrH6fzjrDYwkKHF6jeEAEfHcm8iSVN9BZrCxHSB
python3 tools/floor_check.py dRiftyHA39MWEi3m9aunc5MzRF1JYuBsbn6VPcn33UH # any mainnet program; add --rpc https://api.devnet.solana.com for a devnet one
The status page is a static file that needs a local web server (browsers refuse its module imports from file://): run python3 -m http.server -d cloud 8000 and open this link on localhost, which fills in the devnet market and the Floor-protected S10 program.
In about two minutes each (the first run downloads Rust 1.92.0, the tests' pinned nightly and Solana's platform-tools):
scripts/check.sh # the core, the reference, the tools, Cloud: needs rustup, Python 3.9+, Node 20+
scripts/check-programs.sh # the programs on the real runtime: needs the Agave CLI v4.3.0:
# sh -c "$(curl -sSfL https://release.anza.xyz/v4.3.0/install)"
E2E=1 scripts/check-programs.sh # ...plus the attacker-vs-guardian run and a deploy rehearsal, on local validators
scripts/prove.sh # Kani bounded proofs (~6 min)
scripts/mutants.sh # mutation testing (~45 min)
scripts/fuzz.sh # coverage-guided fuzzing, 9 targets (SECONDS_PER=600 for longer)
scripts/deny.sh # supply chain: RustSec advisories, licences, crate sources
python3 tools/floor_check.py <program-id> # who can upgrade any mainnet program; is it ready for SIMD-0500? (--rpc <url> for another cluster)
scripts/floor-demo.sh <program-id> # Floor on that program's real bytes, locally: register, stop, restore byte-identical
scripts/deploy-dryrun.sh # the devnet deploy, rehearsed on a fresh local validator
cloud/e2e/guardian-scenarios.sh # the guardian on a validator: hold, restart, dead RPC, approvals, recovery key (cursor: ~10 min more)
scripts/verify-deployed.sh <url> <id>=<name> # a deployed program runs exactly the code built from this checkout
scripts/verify-devnet.sh # the three devnet programs == this checkout's canonical build (run it at `3eec968`, PR #85, until 107, 115 and 116 are deployed)
scripts/build-programs-docker.sh # that canonical (Linux x86_64) build on any machine with Docker
scripts/floor-devnet.sh <url> <team.json> <dir> # Floor's full lifecycle on a live cluster, forbidden actions simulated
node cloud/deploy/vault-check.mjs <url> <market> # a live market checked without sending anything (needs its keys)
These scripts install nothing. The core has zero dependencies, and Cloud and the Python code use only their standard libraries. Every count stated in this file and in docs/ is checked against the tree by tools/claims.py on every pull request, so a number here is never older than the code.
What the checks show
| Check | Result (latest run of each) |
|---|
| Programs on the real runtime LiteSVM, mainnet feature set, mainnet token program | 182/182: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction (8,072 swaps refused, 19 Oct) |
| Bounded proofs Kani, and exhaustive checks | 20/20 harnesses verified on 22 Oct (17 for every value of their types), 734 s of verification (09 §5). 9 exhaustive tests over ~588M inputs |
| Differential an independent Python reference vs the Rust core | 6,943,561 lines identical (6 seeds × 200k cases, 27 Sep); 33 commands now, 300k cases on every pull request and 3M weekly in CI |
| Program vs model | Random steps on-chain, every outcome equal to the core's prediction: 12,000 (Vault), 300,000 with random limits and every brake (Vault), 200,000 (Floor) |
| Mutation testing cargo-mutants: 797 mutants of the core in the last full run, 15 Oct; in CI on every change to the core | Every non-equivalent mutant killed; 22 equivalent mutants, each with the reason it can't change behavior (09 §6) |
| Coverage-guided fuzzing libFuzzer, 9 targets: the byte parsers and the full-width limiter and ledger | 125M–440M executions per target in the 17 Oct run (30 min each), no crash. Region coverage of that run found two targets that never reached the code they exist for; both fixed (09 §2, finding 109) |
| Property tests G1–G5, fail closed, tightening, the SIMD-0500 gate, snapshot identity, ejection | 26/26, with coverage counters so a vacuous pass fails |
| Golden vectors every worked example in the docs, exact boundaries, the abort binaries' bytes, loader layouts |
Live on devnet
Deployed 28 Sep 2026 and upgraded in place since (same ids, market and seal), most recently on 15 Oct (Floor and Vault; Harbor on 6 Oct). Each runs exactly the canonical build of PR #85, main at 3eec968: Linux x86_64, as CI builds it and as the Solana Foundation's verifiable-build image reproduces it on any machine. The on-chain changes merged since (findings 107, 115, 116 and 132–134) are not deployed yet; CI's weekly devnet comparison says so until they are.
scripts/verify-devnet.sh # builds natively on Linux x86_64, in Docker elsewhere, then compares with devnet (at `3eec968` until the next deploy)
A Mac build orders the Vault's functions differently, so compare with that build, not a local one (09, finding 43). All three programs share one upgrade authority, a team key disclosed with the full hashes in SECURITY.md.
- The demo market. Harbor over a Vault holding 10,000,000 test USDC (protocol
26dY2HWr…). Sealed on 1 Oct, after the 72 h review that follows a seal proposal; until then no payout of any kind was possible.
- S1 ran there with
cloud/e2e/s1.sh, the script of the local end-to-end run. The admin key ordered 12 payouts of 100,000 USDC; the first paid out (the 2% burst) and 11 queued. The Cloud guardian froze the protocol and cancelled all 11, leaving 9,900,000 in the vault. It froze about 13 minutes after the first payout, not seconds as it does locally: a mistyped Compute Budget id made the node refuse every freeze until the id was corrected and the guardian restarted. A queued payout waits at least 6 h, so nothing more could leave. The guardian now drops a refused compute budget and sends the freeze without it (09, finding 47). The devnet Vault names its protocol in every event (finding 64), as this guardian requires.
- Floor S10. A target program (
CvZEMDwM…) was registered, stopped by one guardian, restored byte-identical by two, and left armed (scripts/floor-devnet.sh).
Every signature: 08.
Guarantees
| |
|---|
| G1 | Unbacked outflow ≤ min(C_s(W_s+T)/W_s, C_l(W_l+T)/W_l) within any T, even with admin and upgrade keys stolen (proof). For a vault with absolute caps set: without them, capital parked in the vault first widens the base (A1b) |
| G2 | An owner who owes nothing is never throttled, frozen or priced taking back their own deposit. (Whether the tokens are in the pool is the pool's liquidity; 03 §4.6.) |
| G3 | Backed payouts never leave an owner's ledger unsound at conservative prices |
| G4 | Tighten instantly; loosen only after ≥ 72 h, public, vetoable |
| G5 | Guardians can stop, never steal. Only live (heart-beating) guardians act. Unfreezing needs a live quorum. A live quorum of the other guardians can eject a rogue one, in any set larger than its threshold. A new set needs at least threshold + 2 keys, so any two faults, lost keys or rogue guardians, leave such a quorum unless the rogues are one themselves (two in a 2-of-n set, T7; findings 96 and 107). |
| G6 | Emergency moves go only into a Bulkhead-held lifeboat. Exits wait 72 h, and go only to a destination committed 72 h earlier. |
| G7 | Missing, stale or wide prices, unknown assets and overflow only ever reduce what is backed |
| G8 | (Floor) A new binary deploys only after ≥ 72 h without veto. While the program is stopped, a live quorum's approval of that exact frozen buffer can deploy it sooner. Any live guardian stops the program instantly; a live quorum restores the snapshot, verified byte-identical to the stopped code. |
| G9 | Bulkhead's and Floor's own upgrade authorities are burned after audit (on devnet: a team key, disclosed) |
Repository
crates/bulkhead-core/ the math: dual-bucket GCRA, EMA, value, prices, ledger, classification, governance,
Floor checks and abort binaries (zero dependencies, no_std)
crates/bulkhead-diff/ line-protocol worker for the golden tests and the differential runner
programs/floor/ Floor (Tier 0): holds a program's upgrade key; delay, veto, stop, snapshot restore
programs/vault/ Vault (Tier 1): custody, backed ledger, limiter, tickets, governance, lifeboat
programs/harbor/ a deliberately naive demo protocol whose custody is a Bulkhead vault
programs/tests/ LiteSVM: scenarios, negative tests, compute, the program-vs-model differential
cloud/ zero-dependency JavaScript: client, guardian bot, status page, end-to-end attack,
deployment and live-cluster checks
tools/floor_check.py read-only report on any upgradeable program
tools/claims.py the counts README and docs/ state, checked against the tree (scripts/check.sh)
tools/mutant_exclusions.py the mutants .cargo/mutants.toml excludes, checked against cargo-mutants (scripts/mutants.sh)
reference/ independent Python reference, differential runner, golden vectors, fixtures, Drift model
evidence/ governance census (script + committed snapshot), model output
docs/ the full design (below)
scripts/ check.sh, check-programs.sh, build-programs.sh (+ in Docker), prove.sh, mutants.sh,
fuzz.sh, deny.sh, deploy.sh (+ its dry run), verify-deployed.sh, verify-devnet.sh,
floor-demo.sh, floor-devnet.sh
fuzz/ libFuzzer targets (cargo-fuzz) and their committed seeds
.github/workflows/ CI: every pull request, and a deeper run every Monday (docs/09 §8)
The programs use Pinocchio and build for sBPF v3. scripts/build-programs.sh fails unless each binary is still deployable after SIMD-0500.
Documentation
Start here: FAQ · 07 Integration · 03 Mechanism · 04 Threat model · 17 External review
| Doc | What it answers |
|---|
| FAQ | Short answers: who pays, who the guardians are, stolen guardian keys, why not a bare abort, what the limits mean exactly |
| 00 Decision | What we build, what we refuse, and why. Kill criteria. |
| 01 Evidence | Every number, labelled measured / reported / model / assumption |
| 02 Product | Users, Floor and Vault, guarantees, UX, non-goals, glossary |
| 03 Mechanism | The algorithms, proofs, worked examples, defaults |
| 04 Threat model | Adversaries, bounds, residual risk, invariants → tests |
| 05 Program spec | Accounts, instructions, errors, measured compute, Token-2022 policy |
| 06 Floor | Tier 0: the upgrade-key floor, stop, snapshot restore, SIMD-0500, and why it beats a bare abort binary |
| 07 Integration | How a protocol integrates; calibration; Harbor |
| 08 Scenarios | S1 Drift, S4 bank run, S7 loosen, S8 one-transaction attack, S9 honest borrower, S10 Floor, S11 liquidation cascade: where each runs and what it showed |
Team
Built for the Colosseum Crypto World's Fair.
Found a flaw? Please report it privately, as SECURITY.md describes.
Credits and license
The one-transaction abort pattern is Dean Little's (sbpf-asm-abort); our binaries are generated by our own code. The verification method comes from SP-Differ. See DISCLOSURE.md.
Apache-2.0.
