What is the Compression Algorithm used by Facebook?

Zstandard (also known as zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by Facebook. It is a lossless compression algorithm written in C (there is a re-implementation in Java) – its thus a native Linux program.

1. Does Facebook use data compression?
2. Which algorithm is used for compression?
3. Is LZ4 better than zstd?
4. What is the best compression algorithm?

Does Facebook use data compression?

Similarly, Facebook's package distribution system, fbpkg, is responsible for distributing large files to the fleet. With such large files, fbpkg prioritizes compression efficiency and speed. However, it can't sacrifice any decompression speed, since it is write-once and read-many.

Which algorithm is used for compression?

In the mid-1980s, following work by Terry Welch, the Lempel–Ziv–Welch (LZW) algorithm rapidly became the method of choice for most general-purpose compression systems. LZW is used in GIF images, programs such as PKZIP, and hardware devices such as modems.

Is LZ4 better than zstd?

Larger is better! At the current compression ratios, reading with decompression for LZ4 and ZSTD is actually faster than reading decompressed: significantly less data is coming from the IO subsystem. We know LZ4 is significantly faster than ZSTD on standalone benchmarks: likely bottleneck is ROOT IO API.

What is the best compression algorithm?

The fastest algorithm, lz4, results in lower compression ratios; xz, which has the highest compression ratio, suffers from a slow compression speed. However, Zstandard, at the default setting, shows substantial improvements in both compression speed and decompression speed, while compressing at the same ratio as zlib.