apibackuper
A command-line tool to archive and backup REST APIs
Version: 3.1.0
This README describes version 3.1.0. See the
changelog for release details, the
3.1 upgrade notes before upgrading from 3.0, and the
3.0 migration notes before upgrading from 2.x.
apibackuper downloads paginated REST APIs into local storage and exports
the archive as JSON Lines, gzip, zstd, Parquet or WARC. It is designed for
repeatable, long-running backups rather than one-off HTTP requests.
Features
- GET and POST iterative APIs: page, skip/offset, range, cursor, next-URL and
Link header pagination, with checks that fail incomplete collections
- Full, incremental, and update work modes, with resume from checkpoints
- ZIP, filesystem, or SQLite storage; each update in ZIP storage gets its own archive
- Provenance for every page (
exchange_N.json: request, status, headers, fetch
time, SHA-256) and the original bytes of XML/HTML responses
- Fixity manifests for every run and downloaded file, checked by
apibackuper verify
- Export to JSON Lines, gzip, Zstandard, Parquet or WARC
- YAML configuration (INI still loaded, but deprecated)
- Authentication: Basic, Bearer, API Key, OAuth2 (refresh token or client credentials)
- Rate limiting, retries with
Retry-After/X-RateLimit-Reset waits, response
rules for error bodies, concurrency, and SSL controls
- Follow-up requests for related objects and streamed downloads of associated files
- Optional Python hooks for custom pipeline steps
- Built for AI coding agents:
init probes an endpoint and writes a complete
project, run full --max-pages N checks real pages without touching the
archive, --json gives one machine-readable result per command, and a
bundled Agent Skill tells agents to configure
apibackuper instead of writing a scraper
Documentation
The full documentation site (Docusaurus) lives in docs/ and is
published at ruarxive.org/apibackuper.
Source pages: docs/docs/. Changelog: CHANGELOG.md.
Installation
Using pip
pip install --upgrade pip setuptools
pip install apibackuper
Using pipx or uv
pipx install apibackuper
# or
uv tool install apibackuper
Parquet export, WARC export and aria2 downloads:
pip install "apibackuper[recommended]"
Requirements
Install from source
python -m pip install --upgrade pip setuptools wheel
python -m pip install .
Quick start
For a first run with a known result, follow the
local API walkthrough: it serves
five records on your machine, backs up three pages and checks the exported IDs.
The server and configuration are in examples/local-demo/.
For an external POST API:
apibackuper create etrust
cd etrust
# save apibackuper.yaml with your API settings (and params.json for POST APIs)
apibackuper validate-config
apibackuper run full --dry-run
apibackuper estimate full
apibackuper run full
apibackuper export etrust.jsonl
A worked POST API example is in the
E-Trust walkthrough.
Starter YAML files are in examples/templates/.
create currently writes a minimal apibackuper.cfg; save your complete
configuration as apibackuper.yaml, which takes precedence over that INI file.
The API-related create flags are accepted for compatibility but do not populate
the configuration. Follow the create reference.
Using with AI coding agents
The skills/apibackuper Agent Skill teaches
Claude Code, Codex, Cursor and Kimi Code to configure an apibackuper project
instead of writing a custom scraper for each API. In Claude Code:
/plugin marketplace add ruarxive/apibackuper
/plugin install apibackuper@apibackuper-marketplace
For other agents, and for an AGENTS.md rule without skill support, see the
AI coding agents guide.
Commands
apibackuper init <dir> --url URL [--method POST --body JSON] [-H "Name: value"] [--auth bearer]
apibackuper create <name>
apibackuper detect [--url URL] [--write-config]
apibackuper estimate full
apibackuper run full [--resume]
apibackuper run continue
apibackuper update [--resume]
apibackuper follow full|continue
apibackuper getfiles
apibackuper export output.jsonl
apibackuper info [--json]
apibackuper verify [--strict] [--json]
apibackuper validate-config [--strict] [--json]
apibackuper schema
apibackuper --json <command>
init probes an endpoint and writes a complete project; run full --max-pages 2
checks two real pages in a temporary archive without touching the project.
--json before any command prints one JSON object with the result or an error
code; progress bars are hidden when output is not a terminal.
run full --dry-run (alias --profile), update --dry-run, and
follow full --dry-run print a JSON plan without making HTTP requests.
estimate full contacts the API to measure a sample response.
여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗
