Wingfoil

Wingfoil is a stream processing engine for electronic trading. Wire a strategy once — market data in, decision, order out, fill back — and the same graph backtests against a simulated venue, then runs live against a real one, without changing the wiring.

That property belongs to the engine, not the trading layer. Underneath is a blazingly fast DAG engine that runs a graph interpreted, compiled into a single monomorphized function, or as compiled islands inside an interpreted graph, off source-driven engine time that replays deterministically. The same engine serves real-time decisioning and streaming ML features.

On top sit a venue-neutral trading layer — fixed-point market data and order books, an OMS, positions, reconciliation, a kill switch and FIX order entry — and production-ready adapters for tick stores, message buses, market protocols and observability backends, so a graph plugs into real sources and sinks in a line.

9.0 replaces the engine. Coming from 8.x, start with the release notes and the migration guide.

There are 59 runnable example targets (54 directories) under crates/wingfoil/examples/.

Languages

Wire the graph in Rust or Python — the same engine underneath, the same combinator surface — and stream it to a browser over the web adapter.

	Install	Package	Docs	Source
Rust	cargo add wingfoil	crates.io	docs.rs	crates/wingfoil/
Python	pip install wingfoil	PyPI	readthedocs	crates/wingfoil-python/
TypeScript	npm install @wingfoil/client	npm	js/README.md	js/

Rust is the engine itself — all three Nitro execution tiers, every op and adapter, and #[op] to add your own. Python gets the same graph model, combinators and adapters in the wheel, with nodes written in Python and results out as a pandas frame. TypeScript is a browser client for the web adapter, decoding the wire format with the server's own code compiled to wasm. The Python wheel and the browser client both track the engine version, so one number covers all three registries.

Features

• Fast: ~0.3 ns compiled / ~12 ns interpreted of engine overhead per node cycle, from a topologically sorted DAG execution engine that visits each node once per tick.
• Nitro — three execution tiers, one wiring: interpreted, compiled, or a compiled island — all derived from the same definition, so they cannot drift. Compiled runs 4.4×–37× faster.
• Backtesting: replay historical data deterministically off source-driven engine time, then run the identical graph live. Same-instant values ride a single burst — never coalesced, never latest-wins, never dropped, in either mode.
• Adapters: production-ready integrations for KDB+, PostgreSQL, Kafka, Redis, Fluvio, etcd, ZeroMQ, FIX 4.4, , , , , , , , , and — one runnable example each.

Quick Start

A simple linear pipeline, with all nodes ticking in lock-step:

use std::time::Duration;
use wingfoil::{RunFor, RunMode};
use wingfoil::prelude::*;

fn main() {
    GraphBuilder::new()
        .ticker(Duration::from_secs(1))
        .count()
        .map(|i| format!("hello, world {i}"))
        .print()
        .build()
        .run(RunMode::RealTime, RunFor::Cycles(3))
        .unwrap();
}

The same graph from Python — run defaults to deterministic historical replay, so this one finishes instantly rather than taking three seconds:

import wingfoil as wf

g = wf.Graph()
(
    g.counter(period_nanos=1_000_000_000)   # tick every second: 1, 2, 3, …
     .map(lambda n: f"hello, world {n}")
     .print()                               # print each value, pass it through
)
g.run(cycles=3)

Either way, this output is produced:

hello, world 1
hello, world 2
hello, world 3

Order Book Example

Wingfoil lets you wire up complex business logic, splitting and recombining streams and modulating the frequency of data. Adapters make it easy to plug in real data sources and sinks. Here we load a CSV of AAPL limit orders, maintain an order book with the lobster crate, derive trades and two-way prices, and export both back to CSV:

let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);

let g = GraphBuilder::new();
let (fills, prices) = csv_read(&g, &source_path, get_time, true, None)?
    .map(move |chunk: &Burst<Message>| process_orders(chunk, &book))
    .split();

let _prices_sink = prices.filter_none().distinct().csv_write(&prices_path)?;
let _fills_sink = fills.csv_write(&fills_path)?;

g.build().run(RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)?;

The frequencies of the inputs and outputs are all different to each other — messages arrive in same-timestamp bursts, the top of book changes less often, and trades are sparser still. This output is produced:

An hour of market data — 91,998 messages — replays in about a tenth of a second. Full example.

Trading

wingfoil::trading is a venue- and asset-neutral trading layer, in two halves:

• market — the market data vocabulary (Px, Qty, Trade, BookUpdate, OrderBook) and order book maintenance. Feature market.
• execution — built on it: the order edge, an OMS, the position fold, reconciliation against the venue, a kill switch, and the Venue trait that is the swap point between venues. Feature execution.

Neither connects to anything itself. A venue integration normalises into market and implements Venue; a strategy drives the OMS and reads the venue's session back. The whole loop is four lines:

let g = GraphBuilder::new();
let inputs = Inputs::replayed(&g); // the touches, and the quoter's desired levels

// A venue that fills at the touch, one millisecond each way.
let venue = SimVenue::new(VENUE, EPOCH, &inputs.touches).with_latency(LATENCY);

// Open the OMS on it. The session is the venue's side: reports, fills, holdings.
let oms = g.oms(CONFIG, EPOCH).venue(&venue);
let reports = oms.session().reports;

// The strategy's desired quotes are the OMS's last input; `build` closes the loop.
let (requests, _pacing) = oms.desired(&inputs.desired).build();

requests.print_items("sent");
reports.print_items("heard");
g.build().run(RunMode::HistoricalFrom(START), RunFor::Duration(Duration::from_secs(1)))?;

The OMS keeps one request in flight per slot, amends a level whose price moved, and withdraws one the strategy no longer wants:

0.000_000_000 sent place 1 Aapl Bid 100 @ 189.5 (PostOnly, GoodTillCancel)
0.000_000_000 sent place 2 Aapl Ask 100 @ 189.6 (PostOnly, GoodTillCancel)
0.000_000_000 sent place 3 Msft Bid 50 @ 420.9 (PostOnly, GoodTillCancel)
0.000_000_000 sent place 4 Msft Ask 50 @ 421.1 (PostOnly, GoodTillCancel)
0.002_000_001 heard ack 1 venue 1
0.002_000_001 heard ack 2 venue 2
0.002_000_001 heard ack 3 venue 3
0.002_000_001 heard ack 4 venue 4
0.300_000_000 sent amend 1 Aapl → 100 @ 189.55
0.300_000_000 sent amend 2 Aapl → 100 @ 189.65
0.302_000_001 heard ack 1 venue 5
0.302_000_001 heard ack 2 venue 6
0.600_000_000 sent cancel 3
0.600_000_000 sent cancel 4
0.602_000_001 heard cancelled 3, 50 left
0.602_000_001 heard cancelled 4, 50 left

Between a backtest and production, the venue changes and the run mode changes. Nothing else does. SimVenue is the test venue; FixOrderVenue is the same Venue over a FIX order session, with what one venue says differently confined to its Dialect. The OMS, the strategy and everything downstream are untouched:

// Backtest: the test venue, fed by replayed touches, under `RunMode::HistoricalFrom`.
let venue = SimVenue::new(VENUE, EPOCH, &touches).with_latency(LATENCY);

// Live: the FIX session's inbound messages, status and sender, in the venue's
// dialect, under `RunMode::RealTime`.
let venue = FixOrderVenue {
    inbound: fix.data,
    status: fix.status,
    sender: fix.sender(),
    epoch,
    mass_cancel: false,
    codec: FixCodec::new(Lmax),
};

// Either way:
let oms = g.oms(CONFIG, EPOCH).venue(&venue);

The rest of the layer wires around that loop. In execution_strategy_loop the session's fills fold into a position book reconciled against the venue's holdings, and a kill switch over the day's loss and any mismatch pulls every quote when it latches. At 300ms the venue reports two lots where the fold has one; once that outlasts the grace the fold is re-based on the venue, the switch latches and the quotes are pulled, until an operator clears it:

0.300_000_000 desk net 1 entry 4999.75 mark 4999.625 pnl -6.25 — 1 position disagrees, inside the grace
0.400_000_000 desk mismatch: ours 1, venue 2 since 0.300_000_000 → re-based
0.400_000_000 desk net 2 entry 4999.6875 mark 4999.625 pnl -6.25 — agrees with the venue
0.400_000_000 switch Halted: standing {Mismatch}, latched {Mismatch}
0.400_000_000 sent cancel all
0.402_000_001 heard cancelled 3, 1 left
0.402_000_001 heard cancelled 2, 1 left
0.450_000_000 switch Open: standing {}, latched {}
0.461_000_000 sent place 4 Es Bid 1 @ 4999.75 (PostOnly, GoodTillCancel)
0.461_000_000 sent place 5 Es Ask 1 @ 5000.5 (PostOnly, GoodTillCancel)

execution_oms_quote is the loop above in full · all trading examples · the live venue: trading_e2e · the layer's map: docs/execution-layer.md.

Execution tiers

Nitro is the tier system: one wiring function, wrapped in nitro! { fn my_graph(g: &GraphBuilder) -> ... }, expands to a module offering all three tiers:

Tier	Entry point	What it is
Interpreted	fluent chaining directly, or my_graph::interpreted()	One dyn boundary per op; open world — threaded/busy-poll sources, feedback, bursts.
Compiled	my_graph::compiled(run_mode, run_for)	The whole graph monomorphized into one function, state in locals — fastest, static DAGs.
Nested (island)	my_graph::nested(&g, inputs...)	A compiled sub-graph mounted as one node of an interpreted graph — hot core compiled, edges stay open.

Semantics live once, in each op's cycle function — the tiers differ only in how the engine reaches it, so there is no duplicated execution logic behind those three doors. core/dual_mode has the rules governing what a nitro! wiring accepts.

Performance

Read the ratios, not the absolute times: these were captured on shared 4-core cloud VMs, each comparison measured back to back in the same run. Full method, caveats and per-workload tables: benches/README.md.

	Measurement
Engine overhead per node cycle	~0.3 ns compiled, ~12 ns interpreted (10×10 fan-out, 103 nodes, every node ticking every cycle)
Compiled vs interpreted	4.4×–37× faster across eight workloads
Nested island vs interpreted	2.2×–10.2× faster
Interpreted vs the legacy engine	0.56×–0.84× — the port is faster on all eight
vs rxrust / tokio async streams	~79× / ~134× faster at depth 10, and the gap grows with depth

Wingfoil visits every node once per tick, in topological order. Libraries that propagate along one path at a time re-visit shared nodes once per path — so on a branch-and-recombine graph their cost doubles with every level while Wingfoil's stays flat. core/topological_sort explains the mechanism in 40 lines.

Where the engine sits against FPGA, kernel-bypass and GC'd stacks — and what is deliberately not claimed — is in where wingfoil currently sits, which ends with the four projects that move that line. All four are open: see Get Involved.

Examples

59 runnable examples, each in its own directory with a README covering what it teaches, the wiring, and its expected output. Full index: examples/README.md.

If you are new, run these three in order — they cover the whole model between them:

cargo run -p wingfoil --example hello_graph   # wire → build → run
cargo run -p wingfoil --example ema_crossover # fold/join/map/filter at backtest scale
cargo run -p wingfoil --features csv --example order_book

Core concepts

No services, no feature flags — these run with a plain cargo run.

Example	Description
hello_graph	The smallest complete program: wire, build, run.
ema_crossover	A backtest-shaped graph — fold, join, map and filter over a price series.
order_book	Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, write both back out.
run_mode	Swap RunMode::RealTime and RunMode::HistoricalFrom over the same wiring, for backtesting.
dual_mode	One wiring, three execution tiers — interpreted, compiled, and a compiled island — with the generated code committed beside it.
topological_sort	Why topologically sorted execution avoids the O(2^N) node explosion of naive per-path propagation.
dynamism	Add and remove nodes on a running graph — one price book, four wirings.
feedback	Close a loop between two nodes with feedback — a proportional control loop a plain DAG cannot express.

Adapters

One directory per adapter, each behind its cargo feature. See each README for the service to start and the command to run.

Example	Description
kdb	KDB+ in three parts: time-sliced reads, LRU-cached reads, and a round-trip write/read/validate.
postgres	PostgreSQL — time-sliced historical reads and streaming writes, round-tripped and asserted to tie out.
kafka	Consume a Kafka topic, transform each record, produce to another.
fluvio	Fluvio — seed a topic, consume it, transform, write to a second topic, from one GraphBuilder.
redis	Redis Pub/Sub end to end: publish, subscribe, transform, republish.
etcd	Watch an etcd key prefix, transform the values, write them back under another.
zmq	ZeroMQ pub/sub, with direct addressing or etcd service discovery.
fix	FIX 4.4 — an acceptor and an initiator in one process, over a loopback session.

Trading

The wingfoil::trading layer. The execution_* examples are offline and instant — they need the execution feature, and those that run against testing::SimVenue need execution-testing. market_adapter is the one that connects: a kdb+ replay or the LMAX demo over FIX, behind market,fix,kdb.

Example	Description
market_adapter	One strategy over the venue-neutral book vocabulary, fed by LMAX FIX (realtime) or a kdb+ replay (historical).
execution_strategy_loop	Start here. The whole layer in one graph: quoter, OMS, ceiling, venue, position fold, reconciliation and kill switch.
execution_order_edge	The order vocabulary: Order constructors and validate, epoch-carrying ids, Request / Report.
execution_oms_quote	Quoting through the OMS: one request in flight per slot, amends, withdrawals, and the order-rate budget.
execution_position_book	A position Book valued by your own Measure — equity, index and inverse futures, a variance swap.
execution_reconcile	A position fold read against the venue's holdings, with a grace period and a re-base.
execution_latch_and_cap	The kill switch over limits of your own, plus a per-order Cap on the request wire.

Showcase

Example	Description
latency	A two-process pipeline over iceoryx2 with per-hop stamping and an end-of-run report.
trading_e2e	Browser to live venue and back: WebSocket in, shared memory across processes, FIX/TLS out, with Grafana dashboards over the whole path.

Links

• Explore the examples
• Read the release notes
• Compare the field: stream processing, dataflow and trading frameworks
• Map the execution layer: orders, the OMS, positions and risk
• Browse the crates
• Read the benchmarks
• Use it from Python: wingfoil-python
• Use it from the browser: @wingfoil/client
• See CONTRIBUTING to build, test and contribute

Get Involved!

Four projects are open, and none of them has landed. Each is separable enough to be carried end to end by one person, and each links to a design rather than starting from a blank page:

Project	What it moves	Where it stands
Core pin — pin the graph thread to an isolated core, with the NUMA and warm-up knobs beside it	Deployment discipline — the dominant end-to-end win in the showcase deployment	#392. A working Linux implementation already sits in examples/showcase/trading_e2e/shared.rs; the job is promoting it into runtime/
Project Bypass — Onload validation, then a raw ef_vi/DPDK source behind an in-tree RxSource seam	Ingress, and the wire-to-trade number the benchmarks currently decline to claim	#957, designed in kernel-bypass-io.md. The first rung needs a Solarflare NIC and a measurement run, not a diff
Project Lightning — compiled graphs generated from procedurally wired ones	Config-driven topologies onto Nitro's compiled tier, where nitro! structurally cannot follow	#726, implemented on #769 — open and unmerged, so none of it is on main yet
Project Metal — FPGA/Verilog emission (RHDL) behind that same front-end	The sub-microsecond class: the graph as gateware, with the backtest as its testbench	#727 — exploratory, gated behind Lightning on a hand-written de-risk spike

The fuller picture, with what each one is worth against measured numbers, is in what moves the line.

Looking for something smaller to start with? The good first issues are the other end of the scale — self-contained, one area of the tree each, and every one names the file to change and an existing piece of code to copy. CONTRIBUTING.md has the build and test loop, and docs/adding-an-op.md is the recipe if the issue you pick is a new op.

Nothing to install: open the repo in a codespace and you get current stable Rust, protoc, Python and Docker already set up — cargo test -p wingfoil works as soon as it finishes. The same container runs locally under Docker; see .devcontainer/README.md.

We want to hear from you! Especially if you:

• are interested in contributing
• know of a project that Wingfoil would be well-suited for
• would like to request a feature or report a bug
• have any feedback

Please do get in touch:

• ping us on discord
• email us at [email protected]
• submit an issue
• get involved in the discussion