zfp is a BSD licensed open-source library for compressed floating-point and integer arrays that support high throughput read and write random access. One of zfp's unique features is its support for efficient in-memory representation of multidimensional numerical data for computations like differential equation solvers, data analysis, and visualization, with significant reductions in memory usage. zfp is primarily written in C and C++ but also has Python, Fortran, and other bindings. zfp is loosely based on the algorithm described in the following paper:
Peter Lindstrom, "Fixed-Rate Compressed Floating-Point Arrays," IEEE Transactions on Visualization and Computer Graphics, 20(12): 2674–2683, December 2014, doi:10.1109/TVCG.2014.2346458.
zfp was designed to achieve high compression ratios and therefore uses lossy but optionally error-bounded compression. Bit-for-bit lossless compression of integer and floating-point arrays is also supported. zfp provides high-quality compression and is fast, achieving throughputs of up to 2 GB/s per CPU core and 800 GB/s aggregate throughput on recent GPUs. zfp supports several different back-ends, including OpenMP, CUDA, and HIP. An FPGA implementation is also available.
zfp is hosted as open source on GitHub and can also be installed with package managers like conda, spack, RPM, and MacPorts. Separate conda and pip packages are available for zfPy, the Python interface to zfp. HDF5 users may be interested in the H5Z-ZFP compression plugin. zfp is supported by software tools and libraries like Intel® IPP, HDF5®, ADIOS, BLOSC, E4S, MVAPICH2, OpenInventor™, TTK, VTK-m, and Zarr.
zfp development is supported by the U.S. Department of Energy's Exascale Computing Project, by the Advanced Scientific Computing Research Program, and by the Advanced Simulation and Computing Program. Advanced features such as variable-rate random-access arrays were investigated on LLNL's Variable Precision Computing Project.