HDF5 C Library for R

hdf5lib is an R package that provides a self-contained, static build of the HDF5 C library (release 2.2.0). Its sole purpose is to allow other R packages to easily link against HDF5 without requiring users to install system-level dependencies, thereby ensuring a consistent and reliable build process across all major platforms.

This package provides no R functions and is intended for R package developers to use in the LinkingTo field of their DESCRIPTION file.

Features

• Zero-Dependency & CRAN-Ready: Builds the HDF5 library from source using only standard R build tools. This guarantees your package works "out of the box" on any operating system without requiring users to install system-level libraries or hold administrative privileges.

• Comprehensive Core API (v2.2.0): Provides complete access to the HDF5 C API, including both the Low-Level and simplified High-Level interfaces, giving you full control over file structure, metadata, and raw I/O.

• State-of-the-Art Compression Built-In: Bundles an extensive suite of compression filters directly within the package. Alongside native gzip/deflate and szip, your package gains immediate access to high-performance plugins like Zstandard (Zstd), LZ4, Blosc/Blosc2, Snappy, ZFP, Bzip2, LZF, and Bitshuffle - all without relying on external system libraries.

• API Version Locking: Protect your package from future breaking changes. Downstream packages can selectively compile against specific HDF5 API versions (e.g., 2.0, 1.14, 1.12), ensuring long-term stability and a predictable build cycle.

• Safe for Parallel Processing: Compiled with thread-safety enabled by default. This prevents data corruption when integrating with multi-threaded R frameworks like RcppParallel. (Note: External file locking is still required if you use the High-Level APIs, which are not thread-safe, or if you access files across multiple independent processes).

Installation

You can install the released version of hdf5lib from CRAN with:

install.packages("hdf5lib")

Alternatively, you can install the development version from GitHub:

# install.packages("pak")  
pak::pak("cmmr/hdf5lib")

Note: As this package builds the HDF5 library from source, the one-time installation may take several minutes. ⏳

Usage (For Developers)

To use this library in your own R package, you need to add hdf5lib to LinkingTo, create a src/Makevars file to link against its static library, and then include the HDF5 headers in your C/C++ code.

1. Update your DESCRIPTION file

Add hdf5lib to the LinkingTo field.

Package: myrpackage  
Version: 0.1.0  
...  
LinkingTo: hdf5lib

This step ensures the R build system can find the HDF5 header files in hdf5lib.

2. Create src/Makevars

Create a file named Makevars inside your package's src/ directory. This tells the build system how to find and link your package against the static HDF5 library. You can optionally use the api parameter to lock in a specific HDF5 API version (e.g., 2.0, 1.14, 1.12, 1.10, 1.8, 1.6) to prevent future updates to HDF5 from breaking your package.

Add the following lines to src/Makevars:

PKG_CPPFLAGS = `$(R_HOME)/bin/Rscript -e "cat(hdf5lib::c_flags(api = 2.0))"`
PKG_LIBS     = `$(R_HOME)/bin/Rscript -e "cat(hdf5lib::ld_flags(api = 2.0))"`

(Note: You only need this one src/Makevars file. The R build system on Windows will use src/Makevars.win if it exists, but will fall back to using src/Makevars if it's not found. Since these commands are platform-independent, this single file works for all operating systems.)

3. Manage Global Filter State (Important)

To utilize the bundled compression plugins (LZ4, Zstd, Blosc, etc.), they must be registered with the HDF5 library. Registering them changes process-wide state (HDF5's filter table and Blosc2's codec registry), so you should never register filters on a per-I/O basis. Tearing them down with hdf5lib_destroy_all_filters() also frees the Blosc worker threads and cached contexts that make multi-threaded Blosc fast, so calling it after each operation would throw them away.

Instead, call them exactly once from your package's DLL load/unload entry points.

Register in src/init.c:

#include <Rinternals.h>
#include <R_ext/Rdynload.h>
#include <hdf5lib.h>

static const R_CallMethodDef CallEntries[] = {
  // Add your package's own .Call entry points here
  {NULL, NULL, 0}
};

void R_init_myhdf5package(DllInfo *dll) {
  R_registerRoutines(dll, NULL, CallEntries, NULL, NULL);
  R_useDynamicSymbols(dll, FALSE);

  /* Register the compression plugins once, at load time */
  hdf5lib_register_all_filters();
}

void R_unload_myhdf5package(DllInfo *dll) {
  /* Free Blosc's worker threads and cached contexts, and unregister the filters */
  hdf5lib_destroy_all_filters();
}

R_init_<pkgname>() runs the moment your shared object is loaded, so the filters are in place before any of your code can run. R_unload_<pkgname>() only runs if the DLL is actually unloaded, which R does not do by default -- add this to R/zzz.R if you want the teardown to happen:

.onUnload <- function(libpath) {
    library.dynam.unload("myhdf5package", libpath)
}

4. Include Headers in Your C/C++ Code

With the filters safely registered at the package level, you can now include the standard HDF5 headers and perform native I/O in your C/C++ functions. Decompression of any supported filter will happen entirely transparently.

#include <R.h>  
#include <Rinternals.h>
#include <hdf5.h>

SEXP read_my_hdf5_data(SEXP filename) {  
    hid_t file_id;  
    const char *fname = CHAR(STRING_ELT(filename, 0));

    // Call HDF5 functions directly  
    file_id = H5Fopen(fname, H5F_ACC_RDONLY, H5P_DEFAULT);

    // ... your code using HDF5 APIs ...
    // Any required decompression happens automatically here.

    H5Fclose(file_id);  
    return R_NilValue;  
}

Included HDF5 APIs

This package provides access to the complete core HDF5 C API (v2.2.0). Developers have full access to all standard functions, macros, and types for local file I/O, metadata management, and data manipulation.

Note: To maintain a zero-dependency footprint, optional features requiring external system libraries - such as Parallel HDF5 (MPI), HDFS, and S3 support - are not included.

While the full core API is available, the following highlights represent the most commonly used modules:

High-Level (HL) APIs (Simplified wrappers)

The HL APIs provide "lite" versions of complex operations, making it significantly easier to perform common tasks without manual memory or hyperslab management.

• H5LT (Lite): Simplified dataset and attribute operations (e.g., H5LTmake_dataset_int, H5LTread_dataset_double, H5LTget_dataset_info).
• H5IM (Image): Standardized functions for working with image data (e.g., H5IMmake_image_24bit, H5IMread_image).
• H5TB (Table): Functions for creating and manipulating tabular data structures (e.g., H5TBmake_table, H5TBappend_records).

Low-Level APIs (Comprehensive core functionality)

The package exposes the full range of core HDF5 modules for fine-grained control over file structure, metadata, and raw I/O:

• H5F (File): Manage file lifecycle (H5Fcreate, H5Fopen, H5Fclose, etc.).
• H5G (Group): Organize objects within a file (H5Gcreate2, H5Gopen2, H5Gclose, etc.).
• H5D (Dataset): Manage raw data arrays and I/O (H5Dcreate2, H5Dread, H5Dwrite, etc.).
• H5S (Dataspace): Define data dimensions and selections (H5Screate_simple, H5Sselect_hyperslab, etc.).
• H5T (Datatype): Define and manage data types (e.g., H5T_NATIVE_INT, H5Tcopy, H5Tinsert).
• H5A (Attribute): Manage metadata attached to objects (H5Acreate2, H5Aread, H5Awrite).
• H5P (Property List): Configure library behavior, such as chunking or compression (H5Pcreate, H5Pset_chunk).

Note: For a complete list of all available functions, please refer to the official HDF5 Reference Manual. Any function documented there can be called from your package after including the headers as shown above.

Looking for an R Interface?

If you are looking for a high-level R interface rather than writing C/C++ code, check out the h5lite package. It uses hdf5lib under the hood to provide a fast, "no-nonsense" way to read and write HDF5 files directly from R with a single function call.

Relationship to Rhdf5lib

The Rhdf5lib package also provides the HDF5 C library. hdf5lib was created to provide a general-purpose, standalone HDF5 library provider that offers several key distinctions:

• Zero Configuration Installation: hdf5lib is designed for simplicity. Installation via install.packages() requires no user configuration and reliably provides a modern HDF5 build with important features enabled by default. Rhdf5lib, while flexible, requires users to manage compile-time configuration options for a customized build.

• Modern HDF5 Version: hdf5lib bundles HDF5 v2.2.0, providing access to the latest features and fixes, including native complex number support and improved UTF-8 handling on Windows. This is more recent than the version typically bundled in Rhdf5lib (v1.12.2 as of Bioconductor 3.19).

• Thread-Safety Enabled: hdf5lib builds HDF5 with thread-safety enabled, ensuring safe use with parallel R packages (like RcppParallel). Rhdf5lib does not support building with this feature.

• Predictable Versioning and Features: The version of hdf5lib directly corresponds to the bundled HDF5 version (e.g., hdf5lib v2.2.0.x bundles HDF5 v2.2.0). This allows developers to require a minimum hdf5lib version to guarantee a specific HDF5 version and a consistent set of features. In contrast, Rhdf5lib may link against a pre-existing system library or be configured at install-time, so its package version does not guarantee which version of HDF5 is actually in use or which features are enabled.

hdf5lib is intended to be a simple and reliable provider of the HDF5 C library for any R package.

License

The hdf5lib package itself is available under the MIT license. The bundled HDF5, zlib-ng, libaec, Bzip2, LZF, LZ4, Zstandard, Snappy, Bitshuffle, ZFP, Blosc, and Blosc2 libraries are available under their own permissive licenses, as detailed in inst/COPYRIGHTS.

(Note: Only the HDF5 library headers are exposed).