apibackuper

A command-line tool to archive and backup REST APIs

Version: 3.1.0

This README describes version 3.1.0. See the changelog for release details, the 3.1 upgrade notes before upgrading from 3.0, and the 3.0 migration notes before upgrading from 2.x.

apibackuper downloads paginated REST APIs into local storage and exports the archive as JSON Lines, gzip, zstd, Parquet or WARC. It is designed for repeatable, long-running backups rather than one-off HTTP requests.

Features

• GET and POST iterative APIs: page, skip/offset, range, cursor, next-URL and Link header pagination, with checks that fail incomplete collections
• Full, incremental, and update work modes, with resume from checkpoints
• ZIP, filesystem, or SQLite storage; each update in ZIP storage gets its own archive
• Provenance for every page (exchange_N.json: request, status, headers, fetch time, SHA-256) and the original bytes of XML/HTML responses
• Fixity manifests for every run and downloaded file, checked by apibackuper verify
• Export to JSON Lines, gzip, Zstandard, Parquet or WARC
• YAML configuration (INI still loaded, but deprecated)
• Authentication: Basic, Bearer, API Key, OAuth2 (refresh token or client credentials)
• Rate limiting, retries with Retry-After/X-RateLimit-Reset waits, response rules for error bodies, concurrency, and SSL controls
• Follow-up requests for related objects and streamed downloads of associated files
• Optional Python hooks for custom pipeline steps
• Built for AI coding agents: init probes an endpoint and writes a complete project, run full --max-pages N checks real pages without touching the archive, --json gives one machine-readable result per command, and a bundled Agent Skill tells agents to configure apibackuper instead of writing a scraper

Documentation

The full documentation site (Docusaurus) lives in docs/ and is published at ruarxive.org/apibackuper.

Section	What it covers
Getting started	Install, quick start, positioning
Cookbook	Task index by goal
CLI reference	Every command
Configuration	YAML sections and field reference
Examples	Working projects and templates
Troubleshooting	SSL, pagination, rate limits

Source pages: docs/docs/. Changelog: CHANGELOG.md.

Installation

Using pip

pip install --upgrade pip setuptools
pip install apibackuper

Using pipx or uv

pipx install apibackuper
# or
uv tool install apibackuper

Parquet export, WARC export and aria2 downloads:

pip install "apibackuper[recommended]"

Requirements

• Python 3.10 or greater

Install from source

python -m pip install --upgrade pip setuptools wheel
python -m pip install .

Quick start

For a first run with a known result, follow the local API walkthrough: it serves five records on your machine, backs up three pages and checks the exported IDs. The server and configuration are in examples/local-demo/.

For an external POST API:

apibackuper create etrust
cd etrust
# save apibackuper.yaml with your API settings (and params.json for POST APIs)
apibackuper validate-config
apibackuper run full --dry-run
apibackuper estimate full
apibackuper run full
apibackuper export etrust.jsonl

A worked POST API example is in the E-Trust walkthrough. Starter YAML files are in examples/templates/. create currently writes a minimal apibackuper.cfg; save your complete configuration as apibackuper.yaml, which takes precedence over that INI file. The API-related create flags are accepted for compatibility but do not populate the configuration. Follow the create reference.

Using with AI coding agents

The skills/apibackuper Agent Skill teaches Claude Code, Codex, Cursor and Kimi Code to configure an apibackuper project instead of writing a custom scraper for each API. In Claude Code:

/plugin marketplace add ruarxive/apibackuper
/plugin install apibackuper@apibackuper-marketplace

For other agents, and for an AGENTS.md rule without skill support, see the AI coding agents guide.

Commands

apibackuper init <dir> --url URL [--method POST --body JSON] [-H "Name: value"] [--auth bearer]
apibackuper create <name>
apibackuper detect [--url URL] [--write-config]
apibackuper estimate full
apibackuper run full [--resume]
apibackuper run continue
apibackuper update [--resume]
apibackuper follow full|continue
apibackuper getfiles
apibackuper export output.jsonl
apibackuper info [--json]
apibackuper verify [--strict] [--json]
apibackuper validate-config [--strict] [--json]
apibackuper schema
apibackuper --json <command>

init probes an endpoint and writes a complete project; run full --max-pages 2 checks two real pages in a temporary archive without touching the project. --json before any command prints one JSON object with the result or an error code; progress bars are hidden when output is not a terminal.

run full --dry-run (alias --profile), update --dry-run, and follow full --dry-run print a JSON plan without making HTTP requests. estimate full contacts the API to measure a sample response.

This is the beginning of the README. Read the rest at the original. Read the full README ↗