======== C-Blosc2

A fast, compressed and persistent data store library for C

:Author: Blosc Development Team :Contact: [email protected] :URL: https://www.blosc.org :Gitter: |gitter| :Actions: |actions| :NumFOCUS: |numfocus| :Code of Conduct: |Contributor Covenant|

.. |gitter| image:: https://badges.gitter.im/Blosc/c-blosc.svg :alt: Join the chat at https://gitter.im/Blosc/c-blosc :target: https://gitter.im/Blosc/c-blosc?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge

.. |actions| image:: https://github.com/Blosc/c-blosc2/workflows/CI%20CMake/badge.svg :target: https://github.com/Blosc/c-blosc2/actions?query=workflow%3A%22CI+CMake%22

.. |numfocus| image:: https://img.shields.io/badge/powered%20by-NumFOCUS-orange.svg?style=flat&colorA=E1523D&colorB=007D8A :target: https://numfocus.org

.. |Contributor Covenant| image:: https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg :target: https://github.com/Blosc/community/blob/master/code_of_conduct.md

What is it?

Blosc2 <https://www.blosc.org/pages/blosc-in-depth/>_ is a high-performance compressor and data format optimized for binary data, including numerical arrays, tensors, and other structured formats. It provides a flexible framework of codecs and filters, enabling developers to balance compression speed and ratio for specific use cases, from high-throughput data pipelines to persistent storage. As the successor to the original Blosc library (released in 2010), Blosc2 is built on a mature foundation and is integrated into many popular scientific computing libraries, such as PyTables, h5py, and Zarr.

C-Blosc2 is the latest major version of C-Blosc <https://github.com/Blosc/c-blosc>_, and is backward compatible with both the C-Blosc1 API and its in-memory format. However, the reverse thing is generally not true for the format; buffers generated with C-Blosc2 are not format-compatible with C-Blosc1 (i.e. forward compatibility is not supported by default, but see blosc2_ctx_set_blosc1_format() to write chunks that C-Blosc1 can read). In case you want to ensure full API compatibility with C-Blosc1 API, define the BLOSC1_COMPAT symbol.

See a 3-minute introductory video to Blosc2 <https://www.youtube.com/watch?v=ER12R7FXosk>_.

Blosc2 NDim: an N-Dimensional store

One of the more exciting additions in C-Blosc2 is the Blosc2 NDim layer <https://www.blosc.org/c-blosc2/reference/b2nd.html>_ (or B2ND for short), which allows creating and reading n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, that allows slicing and dicing arbitrary large and compressed data in a more fine-grained way:

.. image:: https://github.com/Blosc/c-blosc2/blob/main/images/b2nd-2level-parts.png?raw=true :width: 75%

To whet your appetite, here is how the NDArray object in the Python wrapper_ performs on getting slices orthogonal to the different axis of a 4-dim dataset:

.. image:: https://github.com/Blosc/c-blosc2/blob/main/images/Read-Partial-Slices-B2ND.png?raw=true :width: 75%

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful <https://www.youtube.com/watch?v=LvP9zxMGBng>_:

.. image:: https://github.com/Blosc/blogsite/blob/master/files/images/slicing-pineapple-style.png?raw=true :width: 50% :alt: Slicing a dataset in pineapple-style :target: https://www.youtube.com/watch?v=LvP9zxMGBng

.. include:: ../WHATS-NEW.rst

More info about the improved capabilities of C-Blosc2 can be found in this paper <https://www.blosc.org/docs/Exploring-MilkyWay-SciPy2023-paper.pdf>_. Please, cite it if you use C-Blosc2 in your research!

Open format

The Blosc2 format is open and fully documented <https://github.com/Blosc/c-blosc2/blob/main/README_FORMAT.rst>_.

The format specs are defined in less than 4000 words, so they should be easy to read and understand. In our opinion, this is critical for the long-term success of the library, as it allows for third-party implementations of the format, and also for the users to understand what is going on under the hood.

Python wrapper

We are officially supporting (thanks to the Python Software Foundation) a Python wrapper for Blosc2 <https://github.com/Blosc/python-blosc2>. It supports all the features of the predecessor python-blosc <https://github.com/Blosc/python-blosc> package plus most of the bells and whistles of C-Blosc2, like 64-bit and multidimensional containers. As a bonus, the python-blosc2 package comes with wheels and binary versions of the C-Blosc2 libraries, so anyone, even non-Python users can install C-Blosc2 binaries easily with:

.. code-block:: console

pip install blosc2

Compiling the C-Blosc2 library with CMake

Blosc can be built, tested and installed using CMake <https://cmake.org>_. The following procedure describes a typical CMake build.

Create the build directory inside the sources and move into it:

.. code-block:: console

git clone https://github.com/Blosc/c-blosc2 cd c-blosc2 mkdir build cd build

Now run CMake configuration and optionally specify the installation directory (e.g. '/usr' or '/usr/local'):

.. code-block:: console

cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory ..

CMake allows configuring Blosc in many different ways, including selecting external codec libraries, fetching pinned upstream codec sources during configuration, or disabling selected codecs altogether. Please note that configuration can also be performed using UI tools provided by CMake (ccmake or cmake-gui):

.. code-block:: console

ccmake .. # run a curses-based interface cmake-gui .. # run a graphical interface

Build, test and install Blosc:

.. code-block:: console

This is the beginning of the README. Read the rest at the original. Read the full README ↗