A blocking, shuffling and lossless compression library
Blosc is a high performance compressor optimized for binary data.
It has been designed to transmit data to the processor cache faster
than the traditional, non-compressed, direct memory fetch approach via
a memcpy() OS call. Blosc is the first compressor (that I'm aware of)
that is meant not only to reduce the size of large datasets on-disk or
in-memory, but also to accelerate memory-bound computations.
It uses the blocking technique to reduce activity on the memory bus as
much as possible. In short, this technique works by dividing datasets
in blocks that are small enough to fit in caches of modern processors
and perform compression / decompression there. It also leverages, if
available, SIMD instructions (SSE2) and ulti-threading capabilities
of CPUs, in order to accelerate the compression / decompression
process to a maximum.
Blosc is actually a metacompressor, that meaning that it can use a
range of compression libraries for performing the actual
compression/decompression. Right now, it comes with integrated support
for BloscLZ (the original one), LZ4, LZ4HC, Snappy and Zlib. Blosc
comes with full sources for all compressors, so in case it does not
find the libraries installed in your system, it will compile from
the included sources and they will be integrated into the Blosc
library anyway. That means that you can trust in having all supported
compressors integrated in Blosc in all supported platforms.