Canonical Working On Zstd-Compressed Debian Packages For Ubuntu

[phoronix]

Phoronix: Canonical Working On Zstd-Compressed Debian Packages For Ubuntu

Support for Zstd-compressed Debian packages was worked on last week by some Canonical/Ubuntu developers and already by the end of the year they are looking at potentially using it by default...

http://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-Zstd-Deb-Packages

[discordian]

Downloading & decompressing in the background would likely dwarf these gains. And whoever is using eatmydata for package managment of their running system?

guess after their previous attempts failed, they aim a bit lower for fragmenting the debian ecosystem

[sdack]

Downloading & decompressing in the background would likely dwarf these gains. And whoever is using eatmydata for package managment of their running system?

guess after their previous attempts failed, they aim a bit lower for fragmenting the debian ecosystem

True, but replacing a compression library with another isn't much of a change really. It still requires full testing, but will likely succeed quickly and without hiccups, which is why they aim to add quickly.

Larger or more fundamental changes require more effort and more testing. The change you're suggesting really has to come from Debian themselves, or as you've pointed out will it increase the differences between Debian and Ubuntu.

[discordian]

True, but replacing a compression library with another isn't much of a change really. It still requires full testing, but will likely succeed quickly and without hiccups, which is why they aim to add quickly.

And then you will have archives from third-parties available that can't be used in debian. The gains are a joke, and leave out alot of variables, ie. with a mechanical HDD or SDCards I could speculate that the gains would be even smaller or even reversed (read/write at the same time, bigger files with Zstd). And to some people 6% bandwidth matters aswell.
I personally still have a repository running on debian wheezy (serving multiple distributions), the repo servers have to get the tools to deal with this changed format as well.

Larger or more fundamental changes require more effort and more testing. The change you're suggesting really has to come from Debian themselves, or as you've pointed out will it increase the differences between Debian and Ubuntu.

Mine would be transparent to the other distributions, changing the format of archives is not.
I got some little shell scripts that can setup a system in a couple minutes instead half an hour (setup the filesystem in an tmpfs, then copy over the result). If a fast installation would be the aim, you could outdo the current installer by a magnitude - particularly on embedded platforms or other usecases where you don't have low-latency harddrives.

[shmerl]

XZ decompression can be fast... as long as you don't use just one thread for it, and when you compress, you also split things in chunks. Take a look at pixz: https://github.com/vasi/pixz

Anyway, the fact that Debian still doesn't use parallelized decompression for packages installation is a pain point.

[sdack]

And then you will have archives from third-parties available that can't be used in debian. The gains are a joke ...

You don't know much about zstd. It can produce files compatible to gzip, which is the compression Debian still uses. Only zstd is 2-6 times faster than gzip. Ubuntu doesn't have to recompress any packages. The fact that they end up with slightly larger files suggests they're still just using the gzip format for their recompressed packages. zstd can certainly produce ratios beyond gzip and close to xz/lzma2 when using it's own format. Even if you were to run into the odd Ubuntu package that was compressed in the zst format can you recompress it back yourself in a single command line.

Mine would be transparent to the other distributions ...

Actually for it to be yours you'd have to be the one who implements it, and you're only talking. Debian's use of gzip sure can use an upgrade and zstd offers the perfect upgrade path. There's nothing wrong with that. If I was you I'd shut up and switch to Ubuntu instead of complaining about what other distros do for their customers. Complain to the makers of your distro for not being as modern as Ubuntu if you're not happy with what you've got.

[discordian]

You don't know much about zstd. It can produce files compatible to gzip, which is the compression Debian still uses. Only zstd is 2-6 times faster than gzip. Ubuntu doesn't have to recompress any packages. The fact that they end up with slightly larger files suggests they're still just using the gzip format for their recompressed packages. zstd can certainly produce ratios beyond gzip and close to xz/lzma2 when using it's own format. Even if you were to run into the odd Ubuntu package that was compressed in the zst format can you recompress it back yourself in a single command line.

So much wrong...
zstd`s format has nothing to do with gzip' s
debian (and ubuntu) uses xz by default and through the bank, not gzip.
ubuntu needs to compress the packages with zstd, making those incompatible with upstream for little gains.

Actually for it to be yours you'd have to be the one who implements it, and you're only talking. Debian's use of gzip sure can use an upgrade and zstd offers the perfect upgrade path. If I was you I'd shut up and switch to Ubuntu instead of complaining about what other distros do for their customers. Complain to the makers of your distro for not being as modern as Ubuntu.

I wouldn't complain if Ubuntu improved anything, what they do is create isolate solutions and try to shove em upstream. ie.. actually discuss this with upstream before making another mess like MIR.
I would only complain to debian if they were to crap around in a similar fashion, but they aint such a bottomfeeder.

[sdack]

zstd`s format has nothing to do with gzip' s
debian (and ubuntu) uses xz by default and through the bank, not gzip.

zstd can handle gzip, lzma and xz files, mate. It doesn't just handle it's own format. That, and because it's fast, is why it's the perfect replacement.

$ zstd --help
*** zstd command line interface 64-bits v1.3.4, by Yann Collet ***
Usage :
      zstd [args] [FILE(s)] [-o file]
...
--format=gzip : compress files to the .gz format
--format=xz : compress files to the .xz format
--format=lzma : compress files to the .lzma format
...

[discordian]

zstd can handle gzip, lzma and xz files, mate. It doesn't just handle it's own format.

You misunderstand what Ubuntu wants to do then. They want use the binary format of zstd, and not just some commandline tool. Means anyone that packages a tool on the new ubuntu will by default create a .deb package that wont install on other debian-based distros or the previous ubuntu version.
There are tons off different compression algos that have different (dis-)advantages to xz and gzip, zstd is no notable "knight in shining armor" here. Such modest gains are more that debatable as you throw out compatibility at the same time.
Particularly if you could keep the 6% lower filesize and just decompress the archives ahead in background threads (ie. as they are transferred over the wire), making those speed gains a non issue.

[sdack]

You misunderstand what Ubuntu wants to do then. ...

No, you misunderstand. Use Ubuntu or live with your choice, but stop complaining over what other distros do. You're playing the violin, because you want to unite the world with one format, mate. Not even Windows has one single format, and yet here you are complaining like a communist.