Debian APT Performance Is Becoming Much Better For Incremental Updates

[phoronix]

Phoronix: Debian APT Performance Is Becoming Much Better For Incremental Updates

If using APT with PDiff enabled for package diffs to do incremental updates, the latest code should now be much faster...

http://www.phoronix.com/scan.php?page=news_item&px=Debian-APT-PDiff-Speed

[mercutio]

it's curious that people seem to still think that using lzo makes sense when lz4 is always better to use instead of lzo.

lz4 gives slightly faster compression (around 10%), very slightly smaller archives (around 1%), and much faster decompression (around 3 to 4x as fast!) .

[schmidtbag]

it's curious that people seem to still think that using lzo makes sense when lz4 is always better to use instead of lzo.

lz4 gives slightly faster compression (around 10%), very slightly smaller archives (around 1%), and much faster decompression (around 3 to 4x as fast!) .

True, but the performance increases are hardly worth the downsides of switching to it. LZ4 was released in 2011, APT was released in 1998. I'm guessing such a transition would break compatibility. To my knowledge, APT is the most widely used package manager and not all maintained distributions use the same version. Breaking compatibility with .deb packages for only a performance improvement is not worth doing. There is no easy way to transition to a different compression method. Older versions of APT (which would be common in distros like the Debian stable release) don't have the ability to read LZ4. To my knowledge, there isn't a way for APT programs to detect the type of compression method of the package, so it would just see the .deb file as being corrupt. In other words, people using an older APT installer wouldn't know to upgrade.

In a distro like Arch or Gentoo, a change like this is possible for their package managers because stability isn't guaranteed and you're expected to keep up with the latest software.

[mercutio]

True, but the performance increases are hardly worth the downsides of switching to it. LZ4 was released in 2011, APT was released in 1998. I'm guessing such a transition would break compatibility. To my knowledge, APT is the most widely used package manager and not all maintained distributions use the same version. Breaking compatibility with .deb packages for only a performance improvement is not worth doing. There is no easy way to transition to a different compression method. Older versions of APT (which would be common in distros like the Debian stable release) don't have the ability to read LZ4. To my knowledge, there isn't a way for APT programs to detect the type of compression method of the package, so it would just see the .deb file as being corrupt. In other words, people using an older APT installer wouldn't know to upgrade.

In a distro like Arch or Gentoo, a change like this is possible for their package managers because stability isn't guaranteed and you're expected to keep up with the latest software.

There wouldn't be any compatibility breaking at this is just based on the locally stored contents list as per the article, where there is a proposition to use lzo to recompress it. (it would still be using gzip for the downloaded file)

I wouldn't advocate to use lz4 to compress debian packages themselves as that would make mirrors use more space, and hurt people with slower download speeds.

Changing existing software to use lz4 over lzo requires some effort. I assume that's why btrfs hasn't done it yet. But when you're implementing compression where none existed earlier and you want to have minimum overhead lz4 seems to always make sense
over lzo. There may be some arguement for lzo over lz4 for smaller block sizes or such, but the contents list is 10s of megabytes.

[doom_Oo7]

There is no easy way to transition to a different compression method. Older versions of APT (which would be common in distros like the Debian stable release) don't have the ability to read LZ4.

Why couldn't they provide an updated APT ? afaik wheez is still getting fixes. They could just pin the apt package so that it would always use the old compression scheme so that people who update once a decade would still be able to update.

[Luke]

Unfortunately I have not been able to update APT past version 1.0.10.2, with 1.1.5 as current in Debian Unstable. My installation was converted back in May from Ubuntu over the Snappy mess. If I update APT, a huge number of installed packages won't be detected, many more are considered "broken" and apt asks for several GB of downloads to fix that, for which I do not have the bandwidth. Reverting APT causes the packages to be detected properly and not declared broken.

I won't consider a new install from scratch, I have far too much built from source for that sort of thing, 67 packages at the moment including gtk3, all of mate compiled with it, ffmpeg, and more. Unfortunately those are not the packages the new APT has issues with. My setup has diverged a long way from what Debian, Ubuntu, or anyone else installs by default . Wondering it I will be permanently stuck with the older APT version at this point. That's OK if packages don't get build with compression 1.0.10.2 can't read, otherwise I might have to build a custom version of APT adding back the newer compression to the older package management,

[bug77]

If the default setting is to use non-incremental updates, how does one enable the incremental ones?

[bobwya]

Unfortunately I have not been able to update APT past version 1.0.10.2, with 1.1.5 as current in Debian Unstable. My installation was converted back in May from Ubuntu over the Snappy mess. If I update APT, a huge number of installed packages won't be detected, many more are considered "broken" and apt asks for several GB of downloads to fix that, for which I do not have the bandwidth. Reverting APT causes the packages to be detected properly and not declared broken.

I won't consider a new install from scratch, I have far too much built from source for that sort of thing, 67 packages at the moment including gtk3, all of mate compiled with it, ffmpeg, and more. Unfortunately those are not the packages the new APT has issues with. My setup has diverged a long way from what Debian, Ubuntu, or anyone else installs by default . Wondering it I will be permanently stuck with the older APT version at this point. That's OK if packages don't get build with compression 1.0.10.2 can't read, otherwise I might have to build a custom version of APT adding back the newer compression to the older package management,

Sounds like it would be easier to run Gentoo in your case :-) I'm biased as it's my distro of choice...

[carewolf]

If the default setting is to use non-incremental updates, how does one enable the incremental ones?

The default is to use incremental updates, but some people have disabled it because it felt slower.

[Ericg]

Sounds like it would be easier to run Gentoo in your case :-) I'm biased as it's my distro of choice...

I've never used Gentoo, and even I'm reading that going "Dude.. run gentoo."