zfp: Compressed Floating-Point and Integer Arrays
zfp is a BSD licensed open-source library for compressed floating-point arrays that support high throughput read and write random access. zfp is primarily written in C and C++ but also has Python, Fortan, and other bindings. zfp is loosely based on the algorithm described in the following paper:
Peter Lindstrom, “Fixed-Rate Compressed Floating-Point Arrays,” IEEE Transactions on Visualization and Computer Graphics, 20(12): 2674–2683, December 2014, doi:10.1109/TVCG.2014.2346458.
zfp was designed to achieve high compression ratios and therefore uses lossy but optionally error-bounded compression. Bit-for-bit lossless compression of integer and floating-point arrays is also supported. zfp is often more accurate and faster than other lossy compressors, especially in its OpenMP and CUDA multithreaded modes. Our team has developed an FPGA implementation of zfp that further improves throughput by 1-2 orders of magnitude.
zfp is available for download below and is also hosted on GitHub. zfPy, the Python interface to zfp, can be installed as a conda package written by Kevin Paul or as a pip package written by David Wade. HDF5 users may be interested in the H5Z-ZFP compression plugin written by Mark Miller. zfp is supported by software tools and I/O libraries like Intel IPP, HDF5, ADIOS, VTK-m, TTK, and E4S.
zfp development is supported by the U.S. Department of Energy’s Exascale Computing Project and by the Advanced Simulation and Computing Program. Advanced features, such as variable-rate random-access arrays, were investigated on LLNL’s Variable Precision Computing Project.