Linux Kernels Can Now Be Compressed With LZ4

[phoronix]

Phoronix: Linux Kernels Can Now Be Compressed With LZ4

The Linux 3.11 kernel will support kernel images compressed using the LZ4 compression algorithm...

http://www.phoronix.com/vr.php?view=MTQwODY

[bnolsen]

lz4 is pretty amazing in its speed. It should be quite useful for zram use as well if it hasn't been deployed for that already. I use it for cache compression and redundant data removal for some custom file formats.

[uid313]

Maybe they can add it to some generic framework so its not only used for loading the kernel, but also accessible by file systems for compression such as on btrfs with transparent compression.

[mercutio]

Maybe they can add it to some generic framework so its not only used for loading the kernel, but also accessible by file systems for compression such as on btrfs with transparent compression.

i've been waiting for the lz4 stuff to be implemented in btrfs for ages. lz4 is definitely good, i was playing around with it a bit before, and you can tune it a bit more for more compression ratio or greater compression speed.

the lz4hc stuff does pretty good compression, and decompression speed remains good, so good for kernels where you compress infrequently and decompress frequently.

[uid313]

i've been waiting for the lz4 stuff to be implemented in btrfs for ages. lz4 is definitely good, i was playing around with it a bit before, and you can tune it a bit more for more compression ratio or greater compression speed.

the lz4hc stuff does pretty good compression, and decompression speed remains good, so good for kernels where you compress infrequently and decompress frequently.

Linux has the Cryptography API which is available to all subsystems.
Which makes the subsystem does not need to re-implement and re-invent it, they can just use the Crypto API.

Does Linux have anything similar for compression?
Would be great if all file systems that supported compression could use the same shared compression algorithms, and they were also available for initrd and other stuff.

[curaga]

Every time I hear about $NEW_FASTER_DECOMPRESSION, I think to myself why has nobody bothered to optimize zlib decompression speed. It compresses far better than the LZ*, and a mere 2x decompression speed improvement would put it on par with LZO.

Then I recall the one time I searched for that, and found Intel has a closed-source optimized zlib library, well faster than the default one. http://www.intel.com/content/dam/www...sion-paper.pdf

Intel's implementation claims 1.75x the decompression speed of the open-source zlib one.

[bnolsen]

Intel's implementation claims 1.75x the decompression speed of the open-source zlib one.

If you notice performance numbers printed are for ARM.

[mercutio]

Every time I hear about $NEW_FASTER_DECOMPRESSION, I think to myself why has nobody bothered to optimize zlib decompression speed. It compresses far better than the LZ*, and a mere 2x decompression speed improvement would put it on par with LZO.

but lz4 is six times faster than zlib. and lz4hc is seven times faster than zlib with similar compression ratio to zlib -1.

Then I recall the one time I searched for that, and found Intel has a closed-source optimized zlib library, well faster than the default one. http://www.intel.com/content/dam/www...sion-paper.pdf

Intel's implementation claims 1.75x the decompression speed of the open-source zlib one.

lz4 somehow is way faster than the others. the only advantage zlib has over lz4 is in compression ratios, and when people really care about that they tend to shift to lzma compression which gives even better ratios.

[snadrus]

Code duplication.

Linux has the Cryptography API which is available to all subsystems.
Which makes the subsystem does not need to re-implement and re-invent it, they can just use the Crypto API.

Does Linux have anything similar for compression?
Would be great if all file systems that supported compression could use the same shared compression algorithms, and they were also available for initrd and other stuff.

I was thinking exactly this. Cryptography may even be a good place to put it, but something similar should be made to reduce duplication when BTRFS, ZRam, ZSwap, init, network?, etc can all use these.

[bnolsen]

I was thinking exactly this. Cryptography may even be a good place to put it, but something similar should be made to reduce duplication when BTRFS, ZRam, ZSwap, init, network?, etc can all use these.

lz4 or any compressor could be considered as well known encryption algorithms.