Con Kolivas Releases LRZIP 0.650 With Optimizations, Fixes

[phoronix]

Phoronix: Con Kolivas Releases LRZIP 0.650 With Optimizations, Fixes

While free software developer Con Kolivas is known for his work on the Linux kernel to improve desktop responsiveness and efforts like BFS and MuQSS, there is also user-space software he has developed. One of those user-space programs under is belt is LRZIP, the Long Range ZIP format, that is focused on providing speedy compression of large files and to do so with lower amounts of memory...

https://www.phoronix.com/scan.php?page=news_item&px=LRZIP-0.650

[ihatemichael]

One of those user-space programs under is belt is LRZIP

I think you meant "his belt"?

[Paradigm Shifter]

I'm quite interested in this. Got some really big files it would be nice to compress down efficiently (read: at all)...

Need to give it a try.

[uid313]

Con Kolivas is a superhero, on day he is a doctor who works at a hospital as a anesthesiologist, then also a the MuQSS process scheduler for the Linux kernel, oh and using CAD he also designed COVID-19 treatment equipment.

[AdrianBc]

I'm quite interested in this. Got some really big files it would be nice to compress down efficiently (read: at all)...

Need to give it a try.

I use lrzip a lot and it is especially useful for compressing large movie files or, even better, large archive files containing many movie files.

With normal compression algorithms, the compression ratio for movie files is usually negligible, because they already use video-specific compression methods.

With lrzip, it is still possible to reduce the sizes by several percent, some times by up to 10%. While that might not seem much, it can still save a few gigabytes for each BluRay movie and for a large number of movies it leads to significant storage savings.

[halo9en]

With lrzip, it is still possible to reduce the sizes by several percent, some times by up to 10%.

Interesting, looks like I'll have to try it!

[geearf]

Con Kolivas is a superhero, on day he is a doctor who works at a hospital as a anesthesiologist, then also a the MuQSS process scheduler for the Linux kernel, oh and using CAD he also designed COVID-19 treatment equipment.

He also took over development of a BTC miner for a while, he's very versatile!

I use lrzip a lot and it is especially useful for compressing large movie files or, even better, large archive files containing many movie files.

With normal compression algorithms, the compression ratio for movie files is usually negligible, because they already use video-specific compression methods.

With lrzip, it is still possible to reduce the sizes by several percent, some times by up to 10%. While that might not seem much, it can still save a few gigabytes for each BluRay movie and for a large number of movies it leads to significant storage savings.

Out of curiosity I just tried with long range zstd, but it wasn't very interesting.
Both long and ultra yielded about 1.5% saving.
Out of same curiosity I tried lrzip and damn was it faster! But the result is actually bigger than with zstd so maybe I did not use the right option.

[AdrianBc]

He also took over development of a BTC miner for a while, he's very versatile!




Out of curiosity I just tried with long range zstd, but it wasn't very interesting.
Both long and ultra yielded about 1.5% saving.
Out of same curiosity I tried lrzip and damn was it faster! But the result is actually bigger than with zstd so maybe I did not use the right option.

With lrzip, depending on the options, one can achieve a very wide tradeoff between speed of compression/decompression and compression ratio.

The lrzip algorithm is just a preprocessing algorithm, which reorders the content of the large files, then it applies a standard compression algorithm.

For the final compression you may choose between lzma (xz), gzip, bzip2, lzo, zpaq.
For all those algorithms you can also apply the same options as those that are available when you use compression utilities dedicated for them.

So there are a lot of possible combinations for the options.

Using lrzip seldom makes sense for small files, up to a few megabytes.

For larger files, I almost always see a great improvement in compression, in comparison with the same algorithm used stand-alone, i.e. lrzip with --gzip compresses much better than gzip, lrzip with --lzo much better than lzo, lrzip with the default lzma much better than xz or 7Zip.

Like I have said, the best improvements are usually obtained for files that otherwise do not compress well, e.g. movies, pictures, ZIP files. If the files are easily compressible, a standard compression algorithm might already achieve all that can be done, and the long-distance reordering done by lrzip would not improve the result.

[geearf]

With lrzip, depending on the options, one can achieve a very wide tradeoff between speed of compression/decompression and compression ratio.

The lrzip algorithm is just a preprocessing algorithm, which reorders the content of the large files, then it applies a standard compression algorithm.

For the final compression you may choose between lzma (xz), gzip, bzip2, lzo, zpaq.
For all those algorithms you can also apply the same options as those that are available when you use compression utilities dedicated for them.

So there are a lot of possible combinations for the options.

Using lrzip seldom makes sense for small files, up to a few megabytes.

For larger files, I almost always see a great improvement in compression, in comparison with the same algorithm used stand-alone, i.e. lrzip with --gzip compresses much better than gzip, lrzip with --lzo much better than lzo, lrzip with the default lzma much better than xz or 7Zip.

Like I have said, the best improvements are usually obtained for files that otherwise do not compress well, e.g. movies, pictures, ZIP files. If the files are easily compressible, a standard compression algorithm might already achieve all that can be done, and the long-distance reordering done by lrzip would not improve the result.

Yeah I tried with a movie as well, but with just lrzip <MOVIE>.
Looking at the options I don't see anything interesting to add, but if you give me your standard line I'll try it.

Thanks!

[AdrianBc]

Yeah I tried with a movie as well, but with just lrzip <MOVIE>.
Looking at the options I don't see anything interesting to add, but if you give me your standard line I'll try it.

Thanks!

It is possible that you have stumbled upon a movie that does not compress well with the standard algorithm. Trying all alternative algorithms may find one that fares better, but this can be very time-consuming and it is seldom worthwhile.

In general, the chances for lrzip to achieve good compression increase with the size of the file, and, at equal sizes, are higher when compressing archive files that contain many files.

I use lrzip mainly to compress archive files for backup. I group the archived files so that before compression the archive files have a size between 50 gigabytes and 60 gigabytes.

This size corresponds to either a single BluRay movie that has not suffered any additional compression or to many smaller movies, pictures, music or documents.

At this large sizes, the best compression algorithms may need quite a long time, so, after experiments, I have settled on using lrzip with the old gzip algorithm.

Gzip compresses much better than LZO, while still having speeds well over 100 megabyte per second, which is much faster than the algorithms with better compression. (With lrzip+gzip, the compression speed may be limited by the SSD write speed for very large files, while for better algorithms the speed is limited to lower values by the CPU used)

On files which compress well, lrzip with gzip usually achieves compression ratios similar to LZMA alone (i.e. xz or 7-Zip), while on files that do not compress at all otherwise it may still achieve a few percent compression.

Nevertheless, the achieved compression is very variable. There are many movies where the compression ratio is less than 1%, while on others it may reach up to 10%.

Because I use it for backup, any individual compression ratio does not matter, but only the average, which is good enough, typically at least 5%.

I store the backups on 6 terabyte magnetic tapes. With lrzip I store about 111 or 112 archives per tape, while without it I store only between 100 and 105.

All archives also include 5% redundancy added after compression, to guard against bit rotting, and lrzip with gzip is good enough to cancel the file size increase due to the added redundancy.

The command line that I use is:

lrzip --verbose --gzip --level=9 "${ARCHIVE}".pax || exit


Like I have said, this is a choice for high compression speed, which targets only a moderate 5% average compression ratio for movies, which is enough for my purpose.

Also like I have said, with these options you will actually achieve the average only over a large number of movies, e.g. 100 movies, while the individual results will vary a lot.


In any case, in my experiments, lrzip is the only tool which can achieve this combination of parameters, compression speed above 100 MB/s and average compression ratio around 5%, for movies using the current video compression standards.