status · under active development; interfaces may change.
FlowLog compiles Datalog into efficient and scalable Differential Dataflow rust executables. As such, FlowLog has first-class incremental maintenance — outputs update without recomputation as facts change. On DOOP points-to it runs ~3.7× faster than Soufflé across 20 DaCapo programs (32 threads).
Quick Start
1 — Install the toolchain. One-time setup — Rust (1.80+) and required OS packages, then a cargo check smoke test.
$ bash env/env.sh # Linux / macOS
PS> .\env\env.ps1 # Windows (elevated PowerShell)
2 — Build. The compiler lands at target/release/flowlog-compiler.
$ cargo build --release
3 — Run an example. example/graph_analysis/reach.dl — nodes reachable from a seed set:
.decl Source(id: int32)
.input Source(IO="file", filename="Source.csv", delimiter=",")
.decl Arc(x: int32, y: int32)
.input Arc(IO="file", filename="Arc.csv", delimiter=",")
.decl Reach(id: int32)
Reach(y) :- Source(y).
Reach(y) :- Reach(x), Arc(x,y).
.printsize Reach
Make a tiny dataset, then compile and run:
$ mkdir -p reach
$ printf '1\n' > reach/Source.csv
$ printf '1,2\n2,3\n' > reach/Arc.csv
# Compile to a binary, then run it on 4 worker threads
$ target/release/flowlog-compiler example/graph_analysis/reach.dl -F reach -o reach_bin -D -
$ ./reach_bin -w 4
Flag reference: Compiler CLI. For incremental mode and the profiler, see https://www.flowlog-rs.com/.
Static, append and mutable inputs. An input relation is static unless its .decl says otherwise: it is loaded once and never changes. Declaring an input append or mutable compiles an incremental program, which accepts updates after the first load: insertions only for an append relation, insertions and deletions for a mutable one. An append relation keeps the cheaper presence weights of batch evaluation, so a program that only ever grows should prefer it. The three can mix in one program; the work that reads only static inputs runs once:
.decl Visited(x: int32) append
.decl Arc(x: int32, y: int32) mutable
System requirements
FlowLog's performance and stability depend on a number of host and OS-level
settings. For example, on Linux a large analysis can abort with memory allocation of <N> bytes failed even when memory is plentiful, because the
allocator maps many memory regions and exhausts a low
vm.max_map_count. Raising it resolves this:
$ sudo sysctl -w vm.max_map_count=1048576
Before running, review the recommended host configuration in the setup guide:
https://www.flowlog-rs.com/tutorial/getting-started/system-config.
Architecture
A .dl program compiles through five stages; three side modules assist the planner and codegen:
profiler
┊
↓
.dl → parser → typechecker → stratifier → planner → codegen → executable
↑
┊
catalog · optimizer
Pipeline
- parser —
.dl → typed AST, each node source-located.
- typechecker — resolves literal types (
1 → int32).
- stratifier — groups rules into dependency-ordered strata; a stratum with a cycle recurses to fixpoint.
- planner — lowers rules to a Differential Dataflow plan, sharing sub-plans to reuse arrangements.
- codegen — emits the plan as Timely + Differential Dataflow Rust.
Side modules
- catalog — per-rule metadata for the planner (signatures, pushdown filters, range checks).
- optimizer — cardinality-based join ordering and worst-case optimal joins (WIP).
- profiler — runtime metrics from Timely / Differential Dataflow operators.
Crates
flowlog-build — library; compile .dl to Rust from build.rs.flowlog-codegen — library; the code generator both frontends share.flowlog-compiler — CLI; compile .dl to a standalone executable.flowlog-runtime — linked into output (interning, IO, sort/merge, incremental-txn state); not a direct dep.
Compiler CLI
$ flowlog-compiler <PROGRAM> [OPTIONS]
<PROGRAM> is a path to a .dl file, or all / --all to compile every program in example/. Common options:
-F, --fact-dir <DIR> — default directory for relative .input filenames; the executable can override it at runtime.-o <PATH> — output executable path; defaults to the program stem (reach.dl → ./reach).-D, --output-dir <DIR> — default directory for .output files; - prints tuples to stdout. The executable can override it at runtime.-B, --build-dir <DIR> — keep the generated Rust project in this directory for subsequent builds.-T, --target-dir <DIR> — share Cargo artifacts across build directories; overrides CARGO_TARGET_DIR. Relative paths start at the compiler's working directory.--str-intern — intern string columns at load for faster joins and lower memory (off by default).-P, --profile — collect execution statistics.-h, --help — full help text.
Testing
A green oracle run is the definition of correct — see tests/README.md for per-suite contracts and recipes.
vs Soufflé — DOOP
On DOOP default points-to analysis (doop/default.dl) across all 20 DaCapo programs at 32 threads (FlowLog -w 32, Soufflé compiled and run with -j 32). All 20 match on 26 shared relation row counts, including VarPointsTo.
Run time (run only; one-off compile excluded) — FlowLog is faster on 20/20, geomean 3.68× (range 1.45–5.50×).
Peak memory — Soufflé is leaner: Soufflé/FlowLog geomean 0.45× (FlowLog trades memory for speed).
Benchmark suite: flowlog-bench.
Publication
FlowLog: Efficient and Extensible Datalog via Incrementality
Hangdong Zhao, Zhenghong Yu, Srinag Rao, Simon Frisk, Zhiwei Fan, Paraschos Koutris
VLDB 2026, Boston
Contributing
Issues and pull requests are welcome. Target main and sign off your commits with git commit -s. PRs must pass CI before merge. See the contributor guide and release process.
License
Licensed under either of Apache License, Version 2.0 or MIT license, at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in FlowLog by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Let's make Datalog fast — and incremental.
