Rewind VM runs a Nix build, a test suite or any Linux command inside a KVM
virtual machine whose every run is a function of its inputs. The same inputs
give the same run, at the same steps, every time. A failure you saw once is a
failure you keep: replay it, scrub through it step by step, read any file as it
was at any step, and fork it with a different thread interleaving from any
point.
$ rewind check github:fzakaria/rewindvm#mylib
schedule 0: exited:0 6169 steps a9d703ba8977 run 6380bb57fcb451da
schedule 1: exited:0 7202 steps a9d703ba8977 run e5003a7041880029
schedule 2: exited:0 7233 steps a9d703ba8977 run 70416cb698a5d521
...
schedule 6: exited:2 5468 steps run 1506c6347419c2d2
...
schedule 6 ends differently from schedule 0; finding the step that decides it
schedule 6 limited to steps 3629..5140 still ends differently
step 5139 decides it: a request to switch threads there makes the run fail
passing: run 1989f6d02c080616, schedule 6 over steps 3629..5139
failing: run 5c910df9774b38f2, schedule 6 over steps 3629..5140
the two are the same run until step 5139
where ./tests/test_pool_shutdown first behaves differently:
...
both 5150 166/174 write(1, "job 15 done: 42559\n")
failing 5153 166/174 SIGSEGV code=1 addr=0x108
failing 5155 166/166 SIGSEGV code=0 addr=0x0
passing 5151 166/174 write(1, "worker picked job 18\n")
passing 5162 166/173 write(1, "job 17 done: 43360\n")
$ rewind events 5c910df9 | grep SIGSEGV
5153 166/174 SIGSEGV code=1 addr=0x108
...
$ rewind replay 5c910df9
identical to the recording: 1713 events over 5192 steps
Install
On x86_64 Linux with KVM:
$ curl -fsSL https://rewindvm.dev/install | sh
With Nix, run it straight from the flake, or install it into your profile:
$ nix run github:fzakaria/rewindvm -- pmu status
$ nix run github:fzakaria/rewindvm#app
$ nix profile install github:fzakaria/rewindvm github:fzakaria/rewindvm#app
On NixOS, add the flake as an input and turn on its module:
inputs.rewind.url = "github:fzakaria/rewindvm";
# in your configuration, with inputs.rewind.nixosModules.default imported;
# it also adds rewindvm.cachix.org to Nix's substituters
programs.rewind.enable = true;
programs.rewind.app.enable = true;
# AMD only: make the branch counter exact at every boot
programs.rewind.amdBranchCounterWorkaround = true;
Or take the tarballs from the
latest release
yourself:
# the rewind command, with the VM's kernel
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-x86_64-linux.tar.gz | tar xz
# the desktop app
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-app-x86_64-linux.tar.gz | tar xz
# the kernel's debug symbols for rewind gdb, next to the command (150 MB)
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-debug-x86_64-linux.tar.gz | tar xz
/dev/kvm must be readable and writable by you. On AMD Ryzen and EPYC, run
sudo rewind pmu enable once after each boot so the VM's clock can follow the
work done inside it; Time inside the VM explains why.
Use it
# build a derivation in the VM; the output is checked against the host's
$ rewind nix nixpkgs#hello
# run the derivation under many thread schedules, and name the step that decides a failure
$ rewind check github:fzakaria/rewindvm#mylib
# the same, with the line of code each thread involved was on
$ rewind check --where github:fzakaria/rewindvm#mylib
# how many of 16 schedules from a step of a recorded run end differently,
# each a fork of the run
$ rewind check --run <run> --schedule-from <step> --schedules 16 --all --no-narrow
# judge every schedule by what a correct run is: here, a config rewritten in place
$ rewind check --property 'atomic-replace=*.conf' github:fzakaria/rewindvm#config-reload
# vary the program's randomness as well as its schedule
$ rewind check --seeds 8 github:fzakaria/rewindvm#hash-order
# any command in a root filesystem: a directory, an erofs image, or a docker export
$ rewind run --root mylib.tar --cwd /src -- make check
# whether this machine can record runs, and what to fix where it cannot
$ rewind doctor
# list runs and look inside one
$ rewind ls
$ rewind show <run>
$ rewind open <run> <step> --compare <other run>
$ rewind log <run> --steps
$ rewind ps <run> <step>
$ rewind cat <run> <step> /build/env-vars
# a shell inside the VM at a step, in a throwaway fork; --with brings more
# Nix packages into it
$ rewind shell <run> <step> --pid <pid>
$ rewind shell <run> <step> --pid <pid> --with nixpkgs#strace
# gdb on a fork at a step, with the symbols and sources of the kernel and of
# the process running there; arguments after -- go to gdb
$ rewind gdb <run> <step>
$ rewind gdb <run> <step> -- -batch -ex 'break pool.c:77' -ex continue -ex bt
# every thread of a process, even with the CPU idle, as at a deadlock
$ rewind gdb <run> <step> --pid <pid> -- -batch -ex 'thread apply all bt'
# the line of the program's own code a thread was on, with its callers: by
# default the thread of the step's event; --json for programs
$ rewind where <run> <step>
$ rewind where <run> <step> --tid <tid> --json
# which thread held the CPU at each step of a window, replayed step by step
$ rewind threads <run> --from <step> --to <step>
# branch a run at a step under another schedule or seed, or replay it exactly
$ rewind fork <run> <step> --schedule 2
$ rewind fork <run> <step> --seed 7
$ rewind replay <run>
# remove a run's forks that ran exactly as an older one did
$ rewind prune <run> --identical --dry-run
$ rewind prune <run> --identical
# remove a run and every fork of it, and forks of those
$ rewind remove <run> --dry-run
$ rewind remove <run>
# remove many runs in one call, here every fork, keeping the runs forked from
$ rewind ls | awk '/fork of/ {print $1}' | xargs rewind remove
# remove the cached images and stored pages no run uses any more
$ rewind gc --dry-run
$ rewind gc
# share a run as one file
$ rewind export <run> --replayable
$ rewind import <file>.rwd
$ rewind import https://github.com/fzakaria/rewindvm/releases/download/case-studies/nix-gc-closure-sigpipe-replayable.rwd
A program can say what a correct run is from the inside with the
Rewind SDK, a file or two for C, Rust, Go, Python, Java or
shell: rewind check judges every run by its assertions, and the app marks
where one broke.
The desktop app
The app scrubs a recorded run. Drag the playhead along the timeline and the
build log, process tree and files follow it, or jump to the failure or to
where the run parts from a passing one.
- Click a file to read it as it was at the playhead, syntax colored, or as a
colored hex dump.
- Show source (
s) opens the line of the program's own code the thread was
on, with the stack frames that led there; click a frame to see its line.
Debug info and sources come from the program itself, or from
nixseparatedebuginfod2 for Nix packages.
- Search (
Ctrl+F or /) finds text in the build log, the console, file
paths and events, and takes the playhead to each match.
- Bookmarks (
b) mark a step with a note, kept with the run and in its
.rwd exports.
- Two runs that differ only in their schedules show the step where the
schedules part, as a dashed blue mark, and the card says what only one of
them got there, such as a reschedule.
- The Compare tab sets the two runs' events side by side from just before
they part, in full, with what differs marked;
x swaps to the compared
run at the matching step.
- Show threads (
t) draws a lane per thread over a window of steps, a bar
where the thread held the CPU, this run above the compared run.
- Previous and Next can stop at one thread, process, file or kind of event.
- Open shell and Attach gdb work inside the VM at the playhead.
- Fork from here branches the run under another schedule. Its menu turns it
into Check from here, which tries 8 to 64 schedules from the playhead and
counts how they ended, a cell per schedule.
- The Runs panel draws a build's runs as a tree of schedules and forks.
$ nix run github:fzakaria/rewindvm#app -- ~/.local/share/rewind/runs/<run>
It opens .rwd exports too, from a file or an https URL, and comes with an
example run and a short tour. See pricing for
licenses.
Case studies
- A SIGPIPE in Nix's gc-closure test:
Rewind's first run of Nix's functional tests failed in
gc-closure.sh, a
flake nobody had reported. A two-line printf piped into head -n1 under
pipefail is two writes, and when head exits between them the writer
dies of SIGPIPE. rewind threads shows head taking the CPU between those
writes; on the host the test never failed in 200 runs.
- A hang in Nix's store schema migration:
a known, fixed bug, NixOS/nix#15693,
reproduced on the version it was reported against.
rewind shell and gdb
inside the VM pin it to SQLITE_BUSY_SNAPSHOT retried inside an open
transaction, which explains why the first fix did not stop it and the
second did.
- Lost task output in devenv:
a known, fixed bug, cachix/devenv#2281,
where a task's last lines went missing. A test that passed 2000 times on the
host failed under 223 of 257 schedules, and
rewind gdb shows the last line
still in the reader's buffer when tokio::select! took the child's exit.
The fix passes every schedule.
The derivations they run are in
examples/case-studies/flake.nix.
How it works
The VM has one vCPU on stock KVM, so code in it runs on the real CPU. Its
Linux kernel carries a small Rewind platform: interrupts arrive only when the
VM hands control to Rewind, the clock moves only then, and the timestamp
counter and hardware RNG are hidden. Each of those handoffs is a step, and the
sequence of steps is the same on every run. Inputs, a Nix closure or a root
filesystem, become a read-only erofs image mapped into the VM's memory.
Keyframes of the VM's memory go into a content-addressed page store, so seeking
to any step restores the nearest keyframe and runs forward.
A GNU hello build from nixpkgs runs at close to native speed in the VM, and its
output is bit for bit the one Nix builds on the host.
Design has the details, the limits, and what comes next.
Develop
$ nix develop # the toolchain, with REWIND_KERNEL and REWIND_INITRD set
$ cargo test --workspace
$ cargo run --release -p rewind -- nix nixpkgs#hello
$ nix flake check # prose, the NixOS module, and a determinism check that needs /dev/kvm
$ nix fmt # Nix, Rust, Python, HTML and Markdown
$ nix run .#serve # the website on a local port
guest/linux/: the kernel patch and config fragment for the VM.crates/rewind-vmm/: the virtual machine monitor: KVM, boot, steps, time,
keyframes.crates/rewind-init/: PID 1 inside the VM, a workspace of its own.crates/rewind-trace/: reading and querying run traces.crates/rewind-store/: the content-addressed page store.crates/rewind-core/: input images, Nix derivations, runs, seeking.crates/rewind/: the rewind command.crates/rewind-app/: the desktop app, a workspace of its own.examples/mylib/: the flaky thread pool the tutorials use.nix/: one file per derivation; flake.nix only wires them together.site/: rewindvm.dev.docs/: tutorials, the time reference and the design.tools/: the prose check and the local site server.
Pushing a v* tag that matches the version in Cargo.toml publishes a
release with both tarballs.
License
| Path | License |
|---|
crates/rewind-app/ | Proprietary to Lunch Time Surf LLC, source available, see its LICENSE |
crates/rewind-app/vendor/gpui-pre-linux/ | Apache-2.0, a patched copy of Zed's GPUI, see its LICENSE-APACHE |
crates/rewind-app/vendor/text-input/ | Apache-2.0, a text field adapted from GPUI's input example, see its LICENSE-APACHE |
crates/rewind-app/vendor/syntaxes/ | Grammars for syntax highlighting, under MIT, the Unlicense and Sublime HQ's terms, see its README |
guest/linux/ | GPL-2.0-only, as Linux is, see guest/linux/LICENSE |
| everything else | MIT, see LICENSE |
The engine and the command are open source. The desktop app's source is here
to read, but copying, modifying or redistributing it needs permission.
