Mozilla Firefox Switches To .tar.xz For Linux Packaging

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts
  • Anux
    Senior Member
    • Nov 2021
    • 1941

    #11
    Originally posted by filbo View Post
    `xz` definitely has the best balance of size, compression time, and decompression time in this list. `zstd` had a poor showing and would not appear in further rounds of testing if I were working on this issue.
    It's a compress once decompress often scenario. So we shouldn't really care about resource usage for compression.
    If their goal was to make it as small as possible, then xz with extreme settings might be the best (supported by most distros) method.
    zstd would be much less resource intensive on decompression (RAM, CPU) which might lead to less frustration on older/limited systems and I don't see that it lacks behind in distro support.

    Comment

    • egorfine
      Junior Member
      • Aug 2014
      • 31

      #12
      xz!? What do you mean I won't be able to uncompressed the new releases on my Ultrix 3.1 box?!

      Comment

      • Vistaus
        Senior Member
        • Jul 2013
        • 5108

        #13
        Originally posted by filbo View Post
        Let's look at some actual data.
        I took the data.tar.xz out of my system's recent Firefox update package and recompressed it with a few compressors I'm familiar with.

        Running `xz` with no flags produced a file identical to the one out of the .deb I started with, so I know that's what was used.

        `zstd` with no flags was fast -- but the output was huge. I ran it with `-19` and then even `--ultra -22`, and the outputs were still significantly larger than `xz`, while also much slower. `lrzip` with no flags was comparable to `xz`. `xz -9e` was much slower, for a ~3% gain in file size. `lrzip -z` gained another 4% file size and was only twice as slow (still vastly faster than `zstd`) -- but decompression took just as long. And `lrzip` was using all cores; installs based on it would leave the system rather hot!

        Table sorted by output size. Sorry, don't know how to format it better.

        Compressor T-z T-u size
        lrzip -z ......... 34 36 64228419
        xz -9e ......... 106 2.0 67272804
        xz ................ 17 0.5 69250616
        lrzip ............ 17 1.6 69879595
        zstd -22 .... 147 0.4 72142410
        zstd -19 .... 125 0.5 74016120
        bzip2 ......... 15 7.0 84933534
        zstd ............. 1 0.4 93623216

        `xz` definitely has the best balance of size, compression time, and decompression time in this list. `zstd` had a poor showing and would not appear in further rounds of testing if I were working on this issue.
        Thanks for the tests! That explains a lot. However, it still doens't explain Mozilla's stance, as they were not talking about these things, but about distro support.

        Comment

        • piotrj3
          Senior Member
          • Mar 2019
          • 839

          #14
          Originally posted by filbo View Post
          Let's look at some actual data.
          I took the data.tar.xz out of my system's recent Firefox update package and recompressed it with a few compressors I'm familiar with.

          Running `xz` with no flags produced a file identical to the one out of the .deb I started with, so I know that's what was used.

          `zstd` with no flags was fast -- but the output was huge. I ran it with `-19` and then even `--ultra -22`, and the outputs were still significantly larger than `xz`, while also much slower. `lrzip` with no flags was comparable to `xz`. `xz -9e` was much slower, for a ~3% gain in file size. `lrzip -z` gained another 4% file size and was only twice as slow (still vastly faster than `zstd`) -- but decompression took just as long. And `lrzip` was using all cores; installs based on it would leave the system rather hot!

          Table sorted by output size. Sorry, don't know how to format it better.

          Compressor T-z T-u size
          lrzip -z ......... 34 36 64228419
          xz -9e ......... 106 2.0 67272804
          xz ................ 17 0.5 69250616
          lrzip ............ 17 1.6 69879595
          zstd -22 .... 147 0.4 72142410
          zstd -19 .... 125 0.5 74016120
          bzip2 ......... 15 7.0 84933534
          zstd ............. 1 0.4 93623216

          `xz` definitely has the best balance of size, compression time, and decompression time in this list. `zstd` had a poor showing and would not appear in further rounds of testing if I were working on this issue.
          Also i would add that 7za is even better. It can also make .xz in shorter time and similar/smaller size.

          Another limitation is that .tar is really not efficient format. Some example results over one projects directory (mostly python stuff):
          62723254 archive.7z (7za) real 1m16.410s user 2m3.207s
          76637181 archive.tar.7z (7za) real 0m37.988s user 1m33.928s​
          76636196 archive7za.tar.xz (made by tar+7z) Time taken(without creating tar): real 0m34.536s user 1m31.649s​
          77034312 archive.tar.xz​ (made by tar+xz) Time taken(without creating tar): real 0m49.215s user 1m42.787s​
          439429120 archive.tar

          Now I know that tar preserves a little more data comparing to 7z (like permissions of files), but there is absolutly no way that data costs 14-15MB (like 22% size of entire 7z archive). Also 7za/7z are monsters being a lot faster and compressing more when they are allowed to compress directly itself. Also there are valid cases when this data like permissions of files do not matter for you (mozilla firefox archive i believe to be such case).

          (decompression speeds for 7z and xz on same archive is the same, 7z compresses (archive.7z) more, and decompresses a tiny bit longer but close to testing margins).

          I just wish we had new instead of tar format that is more efficient or perhaps compression format on linux similar to 7z or rar that can pack all files optimally together. 7z isn't modern format by any means, but it allows 7za to individually look at each file and apply different filters to get best results compression wise, i assume some .so files weren't compressed at all by .xz in tarballs but they were by .7z .
          Last edited by piotrj3; 28 November 2024, 01:12 PM.

          Comment

          • fitzie
            Senior Member
            • May 2012
            • 672

            #15
            Originally posted by ahrs View Post
            Why not Zstd? It should have the same compression benefits but much faster decompression speeds.

            EDIT:

            I glossed over this.

            Is there a distro not on this list that matters?


            https://repology.org/project/zstd
            personally zstd is fine, but their cli interface is lacking, it should be like gzip/xz/bzip2 but it differs in annoying ways and of course there's pzstd which differs from zstd in annoying ways. those reasons and the fact it's not as widely deployed, means it shouldn't be considered a standard archive format just yet.

            Comment

            • kwiatek
              Junior Member
              • Apr 2016
              • 2

              #16
              I believe this is Mozilla's rationale behind ditching zstd https://bugzilla.mozilla.org/show_bu...id=1710599#c22

              Comment

              • avis
                Senior Member
                • Dec 2022
                • 2252

                #17
                Originally posted by filbo View Post
                Let's look at some actual data.
                I took the data.tar.xz out of my system's recent Firefox update package and recompressed it with a few compressors I'm familiar with.

                Running `xz` with no flags produced a file identical to the one out of the .deb I started with, so I know that's what was used.

                `zstd` with no flags was fast -- but the output was huge. I ran it with `-19` and then even `--ultra -22`, and the outputs were still significantly larger than `xz`, while also much slower. `lrzip` with no flags was comparable to `xz`. `xz -9e` was much slower, for a ~3% gain in file size. `lrzip -z` gained another 4% file size and was only twice as slow (still vastly faster than `zstd`) -- but decompression took just as long. And `lrzip` was using all cores; installs based on it would leave the system rather hot!

                Table sorted by output size. Sorry, don't know how to format it better.

                Compressor T-z T-u size
                lrzip -z ......... 34 36 64228419
                xz -9e ......... 106 2.0 67272804
                xz ................ 17 0.5 69250616
                lrzip ............ 17 1.6 69879595
                zstd -22 .... 147 0.4 72142410
                zstd -19 .... 125 0.5 74016120
                bzip2 ......... 15 7.0 84933534
                zstd ............. 1 0.4 93623216

                `xz` definitely has the best balance of size, compression time, and decompression time in this list. `zstd` had a poor showing and would not appear in further rounds of testing if I were working on this issue.
                ZSTD is almost an order of magnitude faster to decompress. Also, did you use ZSTD properly? And that's

                zstd --ultra -22 --long file ?

                Comment

                • Weasel
                  Senior Member
                  • Feb 2017
                  • 4500

                  #18
                  Originally posted by ahrs View Post
                  Why not Zstd? It should have the same compression benefits
                  It doesn't, even at ultra 22. It's worse than xz/LZMA.

                  I don't get why do people keep thinking zstd compresses as good as xz. It fucking doesn't. It is indeed way faster at decompression though.

                  But when it takes like 1 second to decompress Firefox do you even give a shit? Compared to someone on very slow wifi internet having to download more because of zstd, and strain on server etc.

                  xz is just the better choice.

                  Comment

                  • brian_lindholm
                    Junior Member
                    • Apr 2020
                    • 3

                    #19
                    Hmmm... The lzip utility actually yields the best results on the firefox-133 tarball. It uses the same LZMA algorithm that xz and 7zip use, but it's been tuned a little differently and does a little better for most things that I (re)compress:

                    firefox-133.0.tar.lz 71138190
                    firefox-133.0.tar.xz 71237996
                    firefox-133.0.tar.zst 74924328
                    firefox-133.0.tar.bz3 79715431
                    firefox-133.0.tar.bz2 89436257 (original from http://download.cdn.mozilla.net/pub/...-133.0.tar.bz2)

                    And bzip3 is an interesting "new kid on the block". The code base and resulting executable are astonishingly small, and it definitely compresses better than bzip2. I've occasionally seen it do better than even xz/lzip/7zip, but on average it's not quite as good.

                    Comment

                    • caligula
                      Senior Member
                      • Jan 2014
                      • 3332

                      #20
                      Originally posted by Weasel View Post
                      It doesn't, even at ultra 22. It's worse than xz/LZMA.

                      I don't get why do people keep thinking zstd compresses as good as xz. It fucking doesn't. It is indeed way faster at decompression though.

                      But when it takes like 1 second to decompress Firefox do you even give a shit? Compared to someone on very slow wifi internet having to download more because of zstd, and strain on server etc.

                      xz is just the better choice.
                      zstd is often almost as efficient in terms of compressed size. It does not compress better. However it's significantly faster. That's why e.g. Arch switched to zstd.

                      Comment

                      Working...
                      X