pyspi is a comprehensive python library for computing statistics of pairwise interactions (SPIs) from multivariate time-series (MTS) data.
The toolbox provides easy access to hundreds of methods for evaluating the relationship between pairs of time series, from simple statistics (like correlation) to advanced multi-step algorithms (like Granger causality).
The code is licensed under the GNU GPL v3 license (or later).
Feel free to reach out for help with real-world applications.
Feedback is much appreciated through issues, or pull requests.
Installation 📥
pyspi requires Python 3.11 or newer.
Upgrading from 2.x? Version 3.0 removes the Java/JIDT dependency and
changes the Calculator API. See CHANGELOG.md for the
migration table.
pip install pyspi
pyspi depends on a large scientific stack, so installing into a dedicated
environment is strongly recommended:
python -m venv .venv && source .venv/bin/activate
pip install pyspi
Developing on pyspi itself, with uv:
git clone https://github.com/DynamicsAndNeuralSystems/pyspi.git
cd pyspi
uv sync # runtime + dev dependencies from uv.lock
uv run pytest -q # fast test suite
Optional extras: .[tsfile] to load sktime/aeon .ts files, .[bench] for the
compute-cost benchmark suite in bench/.
For a more detailed guide, see the full documentation,
the troubleshooting guide,
and alternative installation options.
Getting Started 🚀
import numpy as np
import pyspi
from pyspi.calculator import Calculator
dataset = np.random.randn(5, 500) # 5 processes, 500 observations
calc = Calculator(dataset=dataset) # z-scores each process by default
calc.compute() # compute every SPI
calc.table # rows = processes, columns = (SPI, process)
calc.table["cov_EmpiricalCovariance"] # one SPI's 5x5 matrix
calc.save("results.npz") # canonical on-disk format
pyspi.load_table("results.npz") # round-trips exactly
Results are stored as an (n_spis, M, M) array plus the SPI and process names,
which is the data's natural shape. .csv is also accepted for eyeballing small
results, but it is one-way and impractical for the full SPI set.
Or from the command line, writing a results table next to your data:
python -m pyspi compute --data ts.npy --config fast --n-jobs 4 --output results.npz
Try it on a bundled example dataset:
from pyspi.data import load_dataset, available_datasets
available_datasets() # forex, cml, standard_normal
calc = Calculator(dataset=load_dataset("forex"), config="fabfour")
calc.compute()
forex is quantised: every one of its seven processes contains repeated
values (24–212 distinct values across 250 observations). Consequently, all
KSG/kraskov variants refuse it under pyspi's continuous, tie-free input
contract. pyspi does not provide a general discrete MI, TLMI or DI plug-in
estimator; its symbolic support is TE-specific. Use an external discrete
estimator when the missing quantities are needed.
Walkthrough tutorials in the full documentation:
simple demonstration ·
finance ·
neuroimaging
Choosing an SPI set
Computing all 296 SPIs is expensive, and cost grows steeply in both the number
of processes M and the series length T. config= takes either a bundled
name or a path to your own YAML:
config= | SPIs | Use when |
|---|
"full" (default) | 296 | You want everything and can afford it. |
"fast" | 194 | General use; drops the slowest SPIs. |
"benchmarked_p99" | 291 | Near-complete, with only the worst cost outliers removed. |
"benchmarked_p95" | 278 | Good coverage/cost trade-off. |
"benchmarked_p90" | 264 | Recommended default for large batches. |
"benchmarked_p80" | 237 | Cost-constrained sweeps. |
"sonnet" | 14 | One representative SPI per module (M01-M14). |
"fabfour" | 4 | Smoke tests and quick sanity checks. |
The benchmarked_p<N> sets keep the fastest N% of SPIs by measured
amortized compute cost, so the cut point is the Nth percentile of the cost
distribution. They were derived from a benchmark grid spanning M ∈ {4..64} and
T ∈ {200..3200}; see bench/README.md for the methodology
and bench/results/analysis/report.md for
the measurements.
To build your own subset by keyword:
from pyspi.utils import filter_spis
filter_spis(["nonlinear", "directed"], output_name="my_spis")
calc = Calculator(config="my_spis.yaml")
Running at scale
Prefer one dataset per process. Calculator.compute(n_jobs=...) parallelises
within a single dataset, but SPIs sharing a cache are grouped into one task
that runs serially in one worker, so the makespan is floored by the longest
single task. Measured on the benchmark grid at M=64, T=1600, that ceiling is
2.3-4.5x regardless of n_jobs, on every config. If you have more datasets
than cores — the usual case on a cluster — run each dataset in its own
single-core process instead. That scales close to linearly, isolates failures to
one dataset, and schedules far faster as an array job.
Use n_jobs > 1 when you have fewer datasets than cores, when one dataset must
finish under a walltime limit, or when memory forces you to share a single copy
of a large dataset across workers.
# PBS array job: one dataset per task, one core each
#PBS -J 0-499
#PBS -l ncpus=1,mem=8GB
python -m pyspi compute --data "datasets/${PBS_ARRAY_INDEX}.npy" \
--config benchmarked_p90 --checkpoint-dir "results/${PBS_ARRAY_INDEX}/"
--checkpoint-dir writes each SPI to <dir>/<identifier>.npy as it finishes and
resumes from those on a re-run, so a job killed at the walltime limit picks up
where it stopped. PYSPI_N_JOBS sets n_jobs from the environment.
When n_jobs > 1, workers pin their nested BLAS/OpenMP thread pools to one
thread each to avoid oversubscription. macOS is
an exception: its Accelerate BLAS cannot be pinned this way, so prefer n_jobs=1
there for a single dataset.
SPI Descriptions 📋
To access a table with a high-level overview of the pyspi library of SPIs, including their associated identifiers, see the table of SPIs in the full documentation.
For detailed descriptions of each SPI, as well as its associated estimators, we provide a full breakdown in the SPI descriptions page of our documentation.
Documentation
The full documentation is hosted on GitBooks.
Use the following links to quickly access some of the key sections:
Contributing to pyspi 👨👨👦👦
Contributions play a vital role in the continual development and enhancement of pyspi, a project built and enriched through community collaboration.
If you would like to contribute to pyspi, or explore the many ways in which you can participate in the project, please have a look at our
detailed contribution guidelines about how to proceed.
In contributing to pyspi, all participants are expected to adhere to our code of conduct.
SPI Wishlist
We strive to provide the most comprehensive toolkit of SPIs. If you have ideas for new SPIs or suggestions for improvements to existing ones, we are eager to hear from and collaborate with you!
Any pairwise dependence measure, provided it is accompanied by a published research paper, typically falls within the scope for consideration in the
pyspi library.
You can access our SPI wishlist via the projects tab in this repo to open a request.
Acknowledgement 👍
If you use this software, please read and cite this article:
Note that preprint and free-to-read versions of this article are available.
@article{Cliff2023:UnifyingPairwiseInteractions,
title = {Unifying pairwise interactions in complex dynamics},
volume = {3},
issn = {2662-8457},
url = {https://www.nature.com/articles/s43588-023-00519-x},
doi = {10.1038/s43588-023-00519-x},
number = {10},
journal = {Nature Computational Science},
author = {Cliff, Oliver M. and Bryant, Annie G. and Lizier, Joseph T. and Tsuchiya, Naotsugu and Fulcher, Ben D.},
month = oct,
year = {2023},
pages = {883--893},
}
Other highly comparative toolboxes 🧰
If you are interested in trying other highly comparative toolboxes like pyspi, see the below list:
- hctsa, the highly comparative time-series analysis toolkit, computes over 7000 time-series features from univariate time series.
- hcga, a highly comparative graph analysis toolkit, computes several thousands of graph features directly from any given network.
- pyhctsa, a python implementation of the highly comparative time-series analysis toolkit.
Our Contributors 🌟
We are thankful for the contributions of each and everyone who has helped make this project better.
Whether you've added a line of code, improved our documentation, or reported an issue, your contributions are greatly appreciated!
Below are some of the leading contributors to pyspi:
License 🧾
pyspi is released under the GNU General Public License.
