Announcement

Collapse
No announcement yet.

Fedora 31 Considers Compressing Their RPM Packages With Zstd Rather Than XZ

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    There's a lot of talking about `xz` compression time in this thread,
    but the main take away from the Fedora change proposal is that they are taking this direction to save _decompression_ time.
    That's the metric that actually matters.

    Now I get it that some users consider decompression time a non-issue,
    because what matters much more is the nb of bits transmitted through their long-distance modem,
    and sure, their use case is the only that should matter for everybody, right ?

    The world has changed so much in the past decade.
    Serving an isolated desktop on a remote island is no longer a "thing" in the Linux world, if it ever was.
    What truly matters is to serve fleets of workstation in large data centers, flooded by high-speed local connections, serving multiple VM and container per system.
    Think AWS, or equivalent, that's where investments are.

    And in this setup, the decompression speed improvements are just _huge_,
    it directly translates into much faster boot time and VM preparation.
    That means less resource "in transit", mobilized but not yet able to serve their target scenario.
    This directly translated into actual infrastructure savings, hence money.
    Not even counting the better experience due to lower latency, with pervasive benefits.

    It's just a no brainer.

    Comment


    • #12
      Originally posted by F.Ultra View Post

      My bad, I thought that you where talking about parallel deflating and not compression. Well perhaps they are using a shared build server and don't want the system all bobbed down by one single maintainer?
      If it's a shared build server then they're likely using numactl, VMs, or some other method that would limit core/thread usage in a shared environment. That still isn't a good reason for them not using multithreaded compression. If my stoner-ass can use numa to lock random stuff to my 2nd processor and the 16gb of ram on its riser board, Fedora can too...like I said, not a good reason for an enterprise grade company like Fedora when a random stoner can do it with no formal training.
      Last edited by skeevy420; 30 May 2019, 04:10 PM.

      Comment


      • #13
        Originally posted by DoMiNeLa10 View Post
        If it's supposed to decompress into RAM, it would bring memory usage up significantly, so it would have to be optional, since people might make assumptions about how many resources upgrading will take. Trying to install multiple packages at once would be I/O bound, and could result in nasty race conditions. In general, I support using state of art compression whenever possible, so I think they should go ahead and break compatibility with ancient releases.
        Trivial since the uncompressed size is known before decompression. Assuming the typical computer is 8 threads and 8GB ram, there's plenty of hardware there to decompress 6 rpms in parallel. Why should we cater to the lowest denominator of 1 thread and 512MB of ram. That's a slow computer.

        I personally have 16 thread and 32GB ram. I suspect the entire installation dvd iso could be decompressed into system memory ... my ssd root partition only has 17GB used. Switching from xz to zstd seems like the wrong solution. It will increase iso size and internet bandwidth usage compared to what xz -9 can offer.
        Last edited by xorbe; 30 May 2019, 03:59 PM.

        Comment


        • #14
          I love zstd. But where are they looking to get gains by increasing decompression speed? Distro installation? Probably useful. Package download & upgrades? Surely decompression speed for xz on a modern machine must exceed average internet download speed? Total operation time will probably get larger if the RPMS get bigger? Or? Are they going to improve compression with architecture set dependent dictionaries? Or flexibile dictionaries per commonly occuring file types?

          Either way. It's an interesting suggestion but could easily fall flat on it's face if it increases bundle sizes for slow connections.

          Comment


          • #15
            Originally posted by xorbe View Post
            Trivial since the uncompressed size is known before decompression. Assuming the typical computer is 8 threads and 8GB ram, there's plenty of hardware there to decompress 6 rpms in parallel. Why should we cater to the lowest denominator of 1 thread and 512MB of ram. That's a slow computer.

            I personally have 16 thread and 32GB ram. I suspect the entire installation dvd iso could be decompressed into system memory ... my ssd root partition only has 17GB used. Switching from xz to zstd seems like the wrong solution. It will increase iso size and internet bandwidth usage compared to what xz -9 can offer.
            Consider doing a live upgrade on a server that's running an application in production. You'd want to have some guarantee that upgrading won't take up resources needed to run your application well enough. Also, don't assume your software will only run on fast machines, as someone might want to make a fedora-derived distribution that runs on legacy hardware.

            Comment


            • #16
              Live upgrade on an active production server? I would have a stern talk with my admin, but maybe that's just me ...

              Comment


              • #17
                The statement "If going for the Zstd Level 19 compression level that's being considered, it would also offer a much better compression ratio." is simply wrong, even according to the linked Fedora proposal.

                I for one never had to wait any relevant/annoying time for the decompression of an RPM - but on many occasions, I had to wait significant time for the transfer of RPMs over slow networks. The better compression of xz (and the much better compression with "-9") to me certainly outweighs the faster decompression speed of zstd. And parallel decompression could certainly be improved with xz, if somebody felt the need to.

                Comment


                • #18
                  Originally posted by xorbe View Post
                  Trivial since the uncompressed size is known before decompression. Assuming the typical computer is 8 threads and 8GB ram, there's plenty of hardware there to decompress 6 rpms in parallel. Why should we cater to the lowest denominator of 1 thread and 512MB of ram. That's a slow computer.
                  Considering current generation Intel hardware has as low as 2 threads, and systems often still start out spec wise with 4GB ram or less, assuming 8 threads with 8GB ram is typical isn't a great assumption.

                  Comment


                  • #19
                    i remember some Fedora Dev saying how XZ was Crap, i just dont remember his arguments on it, an im pretty sure RPM5 had ZSTD in it, an im pretty sure it was Jeff Johnson that made the RFE for RPM4 to also have it once RPM5 was trashed

                    Comment


                    • #20
                      I can't surely say how much it matters but from my experience network speeds is seldom the bottleneck and even if it is I'd rather wait a bit longer than my ram runs full or my cpu get's blocked and something in my desktop usage gets slowed down.

                      I remember several problems with deltarpms that delayed the update longer than it helped it because a small overhead in file size does not matter that much than the decompression speed.

                      And you don't need extremely slow machines if you have a fast internet connection. You probably sit all on your 16 core machines behind a 3g modem or something. If we are talking about deltarpms 10 20% file size does not matter at all in most cases because deltarpms are tiny anyway. If that accelerates deltarpm speed it might get more useful for more system so you don't need a monster machine to have benefits from it.

                      That's just my experience as far as I remember switched ohh I forgot yes I long had a self build htpc with back then a amd E350 processor which was good enough as combined "nas" and kodi box which I think also run fedora at some point, there cpu speed really matters. I replaced that with a Intel Celeron J4105 again not very powerful cpu-wise. Not to mention my Netbook I bought this year 2nd hand on ebay

                      So yes there are tons of use cases with low cpu and sometimes ram environments so this zstd compression sounds good to me.

                      Comment

                      Working...
                      X