PBP - PVE Backup Proofing

"Trust, but verify" --Ronald Reagan

Proof - the act of proving, verifying, or substantiating that something is true, correct, or officially acceptable. It confirms that a claim, process, or product meets specified requirements, functions as intended, and operates within expected parameters.

In the context of backups, we all understand the fact that an untested backup is not a backup at all. As we gain experience, we put into place practices which speak to this reality.

Proxmox Backup Server has made backups incredibly accessible, reliable, efficient, and stable. The verification of the backup snapshot written to disk is, however, not proof that we will have a working product after a restore.

That's where PBP steps in. We are going to trust the backup existing verification, but we are going to prove that our backups will support our disaster recovery plans.

PBP is built and maintained by Breadth Security.

PBP (PVE Backup Proofing) is a small daemon that runs in a container or directly on a Debian host. On a schedule it takes the newest backup of every guest from your Proxmox Backup Server storage, restores it into a temporary guest on an isolated network, boots it, checks that it is healthy, destroys the temporary guest, and writes the verdict into the backup's notes and into a notification. Deep options are provided for guest inclusion/exclusion, systemd/service ignore and health checks, per-guest checks, and many more options to help you ensure that your backup is valid.

• If you are migrating from the VMWare ecosystem, this is comparable to the functionality provided by Live Recovery.
• This is also comparable to the backup validation functionality provided by XenServer / XCP-ng / XO

How it works

For each run, PBP:

1. Preflight. Checks the API token and SSH, the node helper (its version, and that its VMID range and bridge match your config), the node's PVE version, that the proof bridge is really isolated, the restore storage and the notification templates. It cleans up after any crashed run (the janitor), and checks that the VMID range holds no guest PBP does not own.
2. Discovery. Picks the newest backup of each guest across all configured PBS storages, leaving out guests you exclude. Guests whose last PROVEN verdict is oldest (or who were never proven) go first. A backup that already carries a PBP PROVEN line is skipped unless restore.reproof: true.
3. Templates. A backup of a VM or container template (template: 1 in the config stored in the backup) is skipped: a template cannot be started, so it cannot be proven. Nothing is restored and the backup is not touched.
4. Capacity gate. Skips a guest if the backup does not fit in the restore storage (free space, minus space reserved by restores in progress, minus space_margin_percent of the restore storage's total size) or the node lacks free RAM.
5. Protect, restore, sanitize. Marks the backup protected so pruning cannot remove it mid-proof, and restores it to the next free VMID in the reserved range, tagged pbp (a VM is also named pbp-<original VMID>-<backup time, UTC, YYYYMMDDHHMM>; a container keeps its hostname). It then puts every NIC on the isolated bridge (a container's DHCP and IPv6 autoconfiguration become manual, since nothing answers them there), sets onboot=0, clears protection (a backup of a protected guest restores protected, and could then never be destroyed) and removes host passthrough. A drive the backup does not hold, whose line would still name the production volume, is removed when PBP's token may (VM.Config.Disk), and is an ERROR otherwise. Anything that cannot be made safe is an ERROR; the guest is never started.
6. Boot and check. Starts the guest and waits up to restore.boot_timeout (default 5m). Inside, within restore.probe_timeout (default 5m; each command at most checks.command_timeout, default 60s), it checks the system state, failed units (minus the ignore list), a read-write root filesystem, service profiles and your own per-guest checks. Linux VMs need the QEMU guest agent; containers are checked with . Alpine and Gentoo containers are checked with OpenRC instead of systemd (a service of the runlevel that is stopped or crashed fails the guest), as is a VM without ; they get the read-only-root and per-guest checks too, but not the built-in service profiles, which are written for systemd.

Verdicts

Verdict	Meaning	Says something about the backup?
PROVEN	Restored, booted, all checks passed	Yes: good
FAILED	Restore failed, boot failed, or any check failed	Yes: bad
BOOTED	The guest reached running but PBP could not look inside (a VM without the guest agent, or an agent without exec rights), or a container without nesting=1 whose systemd cannot start its sandboxed units (details)	Partially
SKIPPED	Not attempted (excluded by config, a template, not enough space or RAM, already proven, run deadline reached)	No
ERROR	A PBP or node problem (SSH or API failure, node out of RAM, a guest that cannot be made safe)	No

Only FAILED indicts the backup. SKIPPED and ERROR mean "not proven yet", and BOOTED means "starts, but not looked inside". Every BOOTED guest is listed with its reason in each run summary, under "BOOTED: started, not looked inside".

Requirements

• Proxmox VE 9.0 or newer and Proxmox Backup Server 4.0 or newer.
• On the node that runs the proofs:
  • a PBS storage holding the backups (storage config is cluster-wide, so a PBS storage of any cluster node works);
  • a restore storage with room for your largest guest, for VM disks and container volumes;
  • an isolated bridge for the proof guests, which you create once (Setup step 2);
  • a free range of VMIDs, which you only choose (Setup step 1);
  • sudo (Proxmox VE does not include it; the pbp package installs it as a recommended package, or run apt install sudo if apt is set not to install recommends);
  • a root shell, once, to run pbp node setup.
• A host for PBP that reaches the node's API (port 8006) and SSH (port 22): any host with Docker and Compose, or a Debian-based host for the .deb package (amd64 or arm64). For production, run PBP on a separate host, outside the Proxmox node or cluster it proofs (why). Running it on the node itself works and is supported for a single-node lab or a trial.

PBP never logs in to the node as root. It uses an API token with a narrow role, and SSH only as an unprivileged pbp user that is locked to one restricted helper command. What that means, and what a stolen key could and could not do: Security model.

Getting started

1. Get PBP on the PBP host: clone this repository (Docker), or install the .deb from the releases page (Install from the .deb). Releases are signed: check SHA256SUMS.sig with allowed_signers, key fingerprint SHA256:HzEvo7r8x719x/XzBa5JDQFsFdsw79kNXIdaUt2a7a8 (Verifying a release).
2. Choose a free VMID range and create the isolated bridge on the node (Setup steps 1 and 2). PBP checks both, but you choose the range and create the bridge.
3. Run pbp setup on the PBP host: the Docker one-shot command in Setup step 3, or sudo pbp setup --deb. It asks a few questions, writes config.yaml and the SSH key, and prints two lines to run on the node.
4. On the node, as root, run those two lines. The first installs the matching pbp package; the second, pbp node setup, sets up the restricted helper and the API token, and prints a reply block.
5. Paste the reply back into pbp setup. It writes the node's host key, certificate fingerprint and token secret, and checks everything.
6. Start PBP: docker compose up -d, or sudo systemctl start pbp.

Running PBP on the Proxmox node itself? sudo pbp setup --deb there does steps 3 to 5 in one shell, with nothing to paste: see PBP on the Proxmox node itself.

Documentation

Document	What it covers
Setup (Docker or .deb)	The guided setup step by step, PBP on the node itself, removing PBP from a node
Install from the .deb	Downloading and installing the Debian package, upgrading and removing it
Manual setup	What pbp setup and pbp node setup do, as commands you can run by hand
Operating	Everyday commands, a run on demand, exit codes, the health check, notifications, backup notes
Checks inside the guest	The ignore list, service profiles and per-guest checks, "not covered by backup"
Proofing on a dedicated node	Restoring and booting the proof copies on a separate Proxmox host instead of a production node, and the PBS credentials it needs
Security model	What PBP may touch, the isolated bridge, the restricted node helper, the API token's privileges, where to run PBP
Troubleshooting	What setup and pbp check errors mean and how to fix them
Upgrading

Configuration

pbp setup writes config.yaml for you; config.example.yaml documents every key, for anything you change afterwards. Secrets (the API token and SSH key) are always referenced by file path, never inline. TLS to the API is verified with either tls_fingerprint or ca_file; skipping verification is not supported.

Development and use of AI

This GitHub repository is a mirror of a private Gitea repository, where development happens; every push there is copied here. The workflows in .github/workflows/ run here on GitHub (SLSA build provenance for each release); those in .gitea/workflows/ run only on Gitea. See CONTRIBUTING.md.

PBP is written by traditionally trained software developers using modern AI-assisted practice: specification-driven generation with SpecKit and similar tooling.

License

PBP is licensed under the Apache License, Version 2.0. The Go modules it depends on (golang.org/x/crypto, gopkg.in/yaml.v3, github.com/robfig/cron) and the container base image it ships on (distroless) are distributed under their own licenses.