btclib-benchmarks
Timings of btclib,
btclib-secp256k1 and
bip39-slip39 against the
packages they are usefully compared with.
What a benchmark here can compare is bounded by what its comparands
publish: coincurve builds no cp314 wheel yet, secp256k1 has none
for macOS arm64 past cp313, and on macOS its sdist does not build on an
Intel runner. "The interpreter is 3.13, not the newest" below is the
reason for the first two, and os-macos.yml's own header the third's.
The benchmarks, each answering a different question:
02-btclib-vs-btclib.py — btclib against btclib: its pure-Python
arithmetic against the libsecp256k1 that btclib-secp256k1 bundles and
compiles into a cffi extension03-libraries.py — btclib as installed, against other Python
bitcoin libraries, over curve operations and the address encodings04-pure-python.py — every pure-Python implementation of the same
operation, against each other01-libsecp256k1.py — btclib-secp256k1 against the other
wrappers of the same C library: coincurve and secp256k1-py through
cffi, electrum-ecc through ctypes05-key-reuse.py — what the second signature under the same key
costs: the rest time one verification with a fresh key, and a
verifier never does06-silentpayments.py — BIP352, which only btclib-secp256k1
implements of every comparand here: a payment made and found, and what
a recipient's setup costs before either07-mnemonics.py — BIP39 and SLIP-0039: bip39-slip39 against
Trezor's mnemonic and shamir-mnemonic, and embit
ARCHITECTURE is the design behind them: the shared
modules under src/btclib_benchmarks/, and why measuring and publishing
stay two separate commands.
Why this is its own repository
Most comparands are third-party packages, and pyproject.toml's
dependencies name them all. Measured from inside btclib,
btclib-secp256k1 or bip39-slip39 they would be resolved into the lock of
a library that never imports them — so a vulnerability reported against
a comparand would be reported against that library, and a reader of that
alert would have to work out that the package was a benchmark row rather
than a dependency.
Here the relationship is the right way round. The comparands are what
this project is for, so an advisory against one names the package it is
actually about, and btclib's own lock carries nothing it does not use.
Running them
Nothing is installed, and nothing is released: the scripts are run from
a checkout, and CONTRIBUTING.md's No release says what
the version in pyproject.toml is for.
uv sync --locked
uv run python scripts/03-libraries.py
Each prints what it resolved — package versions, and which arithmetic
backend each comparand actually ran — before any number, because a timing
means nothing without them.
A run also writes itself to results/<name>.json, and
scripts/render.py is what turns that into the published page. Two
commands rather than one, for the reason under "One run of each,
published" below.
The interpreter is 3.13, not the newest
.python-version pins 3.13 because coincurve installs on nothing newer,
and on macOS secp256k1 does not reliably (ISS 381). That file's
comment gives the reasons and says when to raise the pin.
The requires-python floor is 3.11, and that end is set by a comparand
too: secp256k1lab declares it, and scripts/04-pure-python.py imports it
unguarded.
A comparand without a wheel is built at install time
electrum-ecc publishes no wheel anywhere, so uv sync builds it from
its sdist, which carries libsecp256k1 as a submodule and runs its
autogen.sh — that is what autoconf, automake and libtool have to
be there for. The other wrappers resolve to a wheel where the index
serves one for the pinned interpreter and the platform in hand, and
are built from source where it does not, which is when a build of
secp256k1 runs, with the pkg-config that pyproject.toml's
[tool.uv] supplies. CONTRIBUTING.md points at the
workflows that name where each of those happens.
Either way the wrapper rows are honest: each times the libsecp256k1 that
pip install delivered, not whatever system library happened to be
findable.
Measuring a working tree instead of a release
The default is deliberate: every comparand is timed at its published
release, btclib, ecc, btclib-wallet and btclib-secp256k1 included,
because pip install is what an end user gets and that is the only
performance an end user has.
To time a checkout instead — an unreleased optimization, say — install
the package whose code the rows time over the top
(CONTRIBUTING.md's Running a benchmark says which).
uv run --with-editable /path/to/ecc python scripts/03-libraries.py
The packages block names the version, and where an install is not the
declared one it names that too — editable: /path/to/ecc for the
command above, sys.path: for a checkout shadowing an install. So which
of the two you measured is on the screen rather than assumed, and that is
not decoration: an editable install and a released wheel of the same
package resolve silently, and the wrong one produces a plausible table.
Reading the output
The numbers are an order of magnitude, never a figure to quote. Nothing
here repeats a measurement, discards an outlier, or controls the machine
it runs on: a shared CI runner and a laptop with a browser open disagree
by more than most of the differences being reported.
No workflow runs these. CI lints and type-checks; measuring is something
a person does, on a machine whose state they know.
The same C library is not the same binary
01-libsecp256k1.py compares the packages that all wrap
bitcoin-core/secp256k1, which is true of the API and not of what is
linked: each vendors a revision of its own, and they are not the same
revision. So that script prints, per row, which one is underneath it and
how the row reaches it. Read that beside the timings — a build against a
stale one is not the comparison the table looks like, and where
btclib-secp256k1 is the one lagging, the output is where it says so.
One run of each, published
results/ carries what the scripts printed in one sitting on one
machine, each file keeping the header its script printed above the
numbers — the versions, where each package came from, and the backend
every comparand resolved to — because that header is what makes a table
checkable at all. Every table is sorted fastest row first and carries a
ratio against its fastest row, never against btclib's:
The machine is named in each file. They are a record of one run, not a
claim about anyone else's hardware. Where the run recorded them, its run
block names this repository's commit and the SHA-256 of the uv.lock the
run used. The commit is the one CONTRIBUTING.md's Running a benchmark
asks to be on main, and the page names the command that took the run.
Checked out at that commit, synced with uv sync --locked and checked with
shasum -a 256 uv.lock, that command runs the same scripts on the same
package versions where the packages block names no other origin: a package
it marks editable:, local: or sys.path: was not the one the lock
installs. It does not reproduce the numbers, which belong to the machine.
not recorded in place of either is a run saved before runs recorded
them, or taken where git or uv.lock could not answer, and a reader
cannot check what it ran. , with uncommitted changes beside a commit says
the tree held changes outside results/ when the run was saved, at its
end, so the commit alone does not name the code that ran.
Beside each page is the run it publishes, results/<name>.json: the
numbers as they were measured, and everything the page states about how.
A benchmark writes it, scripts/render.py writes the page from it, and
the two are separate commands on purpose — the prose around a table gets
rewritten far more often than a machine gets measured, and it must not
cost a fresh run or a hand-edited number to do it. How the organization
decides, and who holds which role, is its
GOVERNANCE.md;
what it intends to do, and what it deliberately does not, is its
ROADMAP.md.
What a benchmark can get wrong, and where a defect is reported
The failure this repository is built to avoid is measuring something
other than what the table says. Every script prints the version and
the provenance of each package before any number — released, git ref,
editable, or shadowed on sys.path — and the suite checks the
comparands' answers (CONTRIBUTING.md's Writing a row). A number
produced without that header is not a result from this project; treat it
as unverified.
ASSURANCE_CASE.md argues why that check holds,
what this project defends and what it does not.
A vulnerability in a package this project measures belongs upstream,
with the package that has it. Dependabot alerts raised here are alerts
against a comparand, which is the reason under Why this is its own
repository: an advisory should name the project the package actually
belongs to. For the two btclib-org packages, report through their own
policies, btclib's and
btclib-secp256k1's. For anything about this
repository — a script that runs something it should not, a workflow
with a permission it does not need — open an issue, or, if it should not
be public first, take the private route in the security policy GitHub
shows for this repository, which is the organization's, inherited from
btclib-org/.github: this tree
carries no SECURITY.md of its own, publishing nothing an archive
would carry one in.
Contributing
Development is LLM-based: humans are welcome, as long as they are
comfortable with agentic development ;-). A contribution must have been
read and understood by the person who submits it.
CONTRIBUTING.md has the rules.
It also has the commands that measure a run and render it.
REVIEWING.md says what a pull request is answered
against.
Licence
MIT, as every btclib-org repository is.
