Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Ubuntu Still Unsure On Using XZ Packages

  1. #1
    Join Date
    Jan 2007
    Posts
    14,912

    Default Ubuntu Still Unsure On Using XZ Packages

    Phoronix: Ubuntu Still Unsure On Using XZ Packages

    While Fedora has been using XZ-compressed packages for their RPMs for a while now with having a greater compression ratio than Gzip, Ubuntu developers remain unsure of switching to using XZ compression for the Ubuntu 13.04 release...

    http://www.phoronix.com/vr.php?view=MTIxOTY

  2. #2
    Join Date
    Jun 2011
    Posts
    316

    Default

    Right now, just from watching the Debian mailing lists.
    It looks like Debian is going to be using XZ compression on a very small number of packages for the sole purpose of cramming more content onto installation media that is targeted for Desktops.

    It was discussed at DebConf that unpacking XZ files eats a lot of system memory (RAM) and if you don't have the RAM, the unpacking could take several times longer than expected. This may make using XZ compression a bad idea for tablets and cell phones which is important to Ubuntu right now (See the videos of Ubuntu running on cell phones / tablets at Ubuntu's UDS Youtube feed).

    If you have the RAM (2GBs or more) then you're golden for XZ compression as it really isn't noticeably slower than the alternative.

    Debian recognizes that a lot of hardware out there still has less than 2GBs of RAM which means switching entirely to XZ compression could be a *very* bad idea. So they're not going to do it for the Wheezy release for the overwhelming majority of packages and likely won't do it for Wheezy+1 either, but it's still up in the air.

    Using XZ everywhere would be talking about bumping the RAM requirements of Debian from 64MB to 2GBs... That's one heck of a leap.
    Last edited by Sidicas; 11-01-2012 at 12:06 PM.

  3. #3
    Join Date
    Jun 2010
    Location
    ฿ 16LDJ6Hrd1oN3nCoFL7BypHSEYL84ca1JR
    Posts
    1,052

    Default

    You're not going to run a bloaty distribution like ubuntu on something with low ram... I hope.

    Meanwhile archlinux has used xz packages for a while and is still unsure about using lrzip.

  4. #4
    Join Date
    Jan 2010
    Posts
    364

    Default

    The memory usage and performance problems may be tackled simply by reducing the compression setting. xz level 0 or 1 use very little memory and should decompress much faster than e.g. bzip2, with much better compression ration than old gzip.

  5. #5
    Join Date
    Jan 2012
    Location
    Moscow
    Posts
    61

    Default

    I'm not sure about RAM usage, but as far as decompression speed on x86 and x86-64 goes, LZMA/XZ are second in speed only to LZOP/LZ4. Well, for files filled with zeroes at least.
    Here's some research I did back in the day, improving on some wildly inaccurate benchmarks I found in Wikipedia: http://www.webupd8.org/2011/07/free-...utilities.html

  6. #6
    Join Date
    Jan 2010
    Posts
    87

    Default

    If Ubuntu/Canonical actually care about bandwidth then they'd fix their servers. It took me several attempts (and several regressions) to get them to reliably set etags. They were using the Apache defaults which include using the inode number so that meant each server gave a different etag which then prevented any kind of caching since each server was essentially claiming the same file was different. (DNS round robin meant you would hit different servers when downloading packages.) They eventually fixed this.

    I then tried to get them to set correct HTTP caching headers. Package files can't change so they could be set to be forever cacheable. However the Canonical people refused to set such headers which means that even if there are http caches, they have to keep contacting the servers to double check the package files are still valid.

  7. #7
    Join Date
    Nov 2008
    Posts
    118

    Default

    How does XZ compare to 7zip or bzip2?

    I've never liked tar/Gzip archives. Too slow to open large archives because they have to be completely decompressed first. I can get a directory listing of the contents of a an equivalent 7z archive much faster. The deb files are created with Gnu ar then Gzip, also requiring a two-stage extraction process FWICT.

  8. #8
    Join Date
    Feb 2008
    Location
    Linuxland
    Posts
    5,130

    Default

    Quote Originally Posted by ChrisXY View Post
    You're not going to run a bloaty distribution like ubuntu on something with low ram... I hope.

    Meanwhile archlinux has used xz packages for a while and is still unsure about using lrzip.
    lrzip for packages? Dear God, unpacking one of those can take 8gb of ram. What the heck is their target audience?

  9. #9
    Join Date
    Nov 2012
    Posts
    1

    Default

    Quote Originally Posted by jhansonxi View Post
    How does XZ compare to 7zip or bzip2?

    I've never liked tar/Gzip archives. Too slow to open large archives because they have to be completely decompressed first. I can get a directory listing of the contents of a an equivalent 7z archive much faster. The deb files are created with Gnu ar then Gzip, also requiring a two-stage extraction process FWICT.
    xz is just compression, in the same way as gzip - you generally still use tar as the archiver (or whatever archiver you want).

    Where exactly are the memory concerns coming from? The xz manpage says:

    Code:
    The following table summarises the features of the presets:
    
                         Preset   DictSize   CompCPU   CompMem   DecMem
                           -0     256 KiB       0        3 MiB    1 MiB
                           -1       1 MiB       1        9 MiB    2 MiB
                           -2       2 MiB       2       17 MiB    3 MiB
                           -3       4 MiB       3       32 MiB    5 MiB
                           -4       4 MiB       4       48 MiB    5 MiB
                           -5       8 MiB       5       94 MiB    9 MiB
                           -6       8 MiB       6       94 MiB    9 MiB
                           -7      16 MiB       6      186 MiB   17 MiB
                           -8      32 MiB       6      370 MiB   33 MiB
                           -9      64 MiB       6      674 MiB   65 MiB
    So using the default, 6, you only need 9MiB RAM to decompress.

    As for speed, I'm sure they're right to look further into it, but pacman uses xz and is arguably the fastest package manager (and certainly the fastest I've ever used).

  10. #10
    Join Date
    Mar 2011
    Location
    Canada
    Posts
    96

    Default

    Quote Originally Posted by curaga View Post
    lrzip for packages? Dear God, unpacking one of those can take 8gb of ram. What the heck is their target audience?
    https://bbs.archlinux.org/viewtopic.php?id=137918&p=1

    It doesn't seem like it's really being considered at all for building packages yet (see the responses there from the devs), just talk of possibly adding it as another supported format in pacman through libarchive, once the work in the library is done.
    Last edited by strcat; 11-01-2012 at 03:33 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •