Announcement

Collapse
No announcement yet.

Zstd-Compressing The Linux Kernel Has Been Brought Up Again

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #21
    Originally posted by ms178 View Post
    I don't get it why there was backlash to add Zstd, the developers showed them the numbers and it makes sense to use it in several widely used scenarios. Other obsolete algorithms can be dropped after this lands. So why does it take so long to get something beneficial like this into the Kernel?
    There is a trend to always chase the "new shiny" to always say.. oh the Macintosh is out the terminal is obsolete GUI only. Or Plan 9 makes Unix obsolete! Sometimes the new thing really sucks and people don't see the value in the old methods. We don't advance as fast as we think.

    So be careful here.. however with this one I agree with you. Zstd or LZ4 sounds more optimal.

    Comment


    • #22
      Originally posted by Zan Lynx View Post
      Windows systems can boot from hibernate in about 3 seconds. Why should Linux take longer?
      What use case do you think of where hibernation, which to wake up from is way slower than S3-sleep, is preferable?
      (If I intend to not use my laptop for a time long enough to drain the battery in S3 sleep, I don't mind doing an actual shutdown.)

      Comment


      • #23
        Originally posted by phuclv View Post
        It is really relevant in a lot of cases. Modern laptops can boot up in 5-10s which means a 1s reduction is a massive improvement
        But why would I want to boot my laptop more than a few times per year? For laptops, S3 sleep is usually the much more convenient option, and that takes < 0.5s (and does not involve kernel decompression).

        Comment


        • #24
          Originally posted by andreano View Post
          Need motivation? Steve Jobs (an expert motivator) used the analogy of saving lives.
          Steven and his apostles could have "saved even more lives" had they removed the requirement to boot their machines so often. Because not booting takes 0 seconds.

          Comment


          • #25
            Originally posted by DoMiNeLa10 View Post
            A second is a lot, considering it doesn't take much to set up your system to boot up in <4s on a cheap TLC SSD. Some people don't like to wait for these things.
            Then by all means tell them how to wait even less, using S3 sleep instead of rebooting.

            Comment


            • #26
              Originally posted by stormcrow View Post
              What may take 1s on the latest Intel/AMD x64 hardware, could be much more of a factor (like tens of seconds to minutes) on low power embedded hardware, which is probably the biggest sector using the Linux kernel.
              And look how easily this problem is solved by e.g. Android devices, as they just do not boot that often. My phone certainly takes a lot of time - like 30 seconds - to reboot. Do I mind? No, because that does not happen frequently enough to be of relevance.

              As for Zstd from a security prospective, you do realize zstd is already in the Linux kernel in different spots as it is. Btrfs uses it as does squashfs. The algorithms (as opposed to the implementation, don't conflate the two) have been around for decades already, the zstd group just put them together in a unique way to produce better performance than previous incarnations.
              It is precisely optimization of decompression implementations that can easily introduce additional risks. Because every safety "if"-clause in the decompression code path costs time, only to protect against malicious inputs.

              Comment


              • #27
                Arguing about this is dumb. If it is faster, that is good for no other reason than it is faster. No purpose is ever served by going slower.

                Faster! Faster!

                Comment


                • #28
                  Originally posted by stormcrow View Post

                  Not really. What may take 1s on the latest Intel/AMD x64 hardware, could be much more of a factor (like tens of seconds to minutes) on low power embedded hardware, which is probably the biggest sector using the Linux kernel. It's not about how fast it takes your laptop to boot, um... unless it's like my vintage HP Centrino Duo laptop I use for a terminal, but I don't really care much there either... it's about how fast it takes a Blackfin controller, or a low power ARM32 board (like early R-Pi models) to unpack its kernel before it can even start to boot. Leaving the kernel uncompressed isn't an option either, as non-volatile storage on embedded devices aren't always large enough and/or the boot loader may have kernel size limits.
                  (...)
                  .
                  if you go by that..
                  Take a Look at ARM64 speeds with lz4 at bottom of page..
                  Last edited by tuxd3v; 10 June 2019, 05:36 PM. Reason: typos

                  Comment


                  • #29
                    Originally posted by dwagner View Post
                    And look how easily this problem is solved by e.g. Android devices, as they just do not boot that often. My phone certainly takes a lot of time - like 30 seconds - to reboot. Do I mind? No, because that does not happen frequently enough to be of relevance.

                    It is precisely optimization of decompression implementations that can easily introduce additional risks. Because every safety "if"-clause in the decompression code path costs time, only to protect against malicious inputs.
                    You're taking a single use case in a very broad market. Android is the most visible case of Linux in the computing device market, but I rather struggle to call it "embedded" without being very loose with the term. Tablets and smart phones are more like single slab laptops these days with almost as much computing power in the latest generation as a laptop from just a few years ago. Most embedded devices do indeed not reboot or cold boot very often, but they do actually have to do so every so often either after an update or to clear technical or mechanical glitches. In which case it's very desirable to go from minutes from power on to usable to a minute and a half, a minute, or less depending on which compression system is being used, yes even on a smart phone.

                    As for your security assertions, you have evidence of such tight timing integrity attacks carried out in practice rather than theoretical?

                    Comment


                    • #30
                      Originally posted by Raka555 View Post

                      I am curious as to what you have decided to use for your backups ?
                      I use in my backup scripts zstd or pigz (multi-threaded gzip) depending on my needs. I've also tried switching linux logrotate from gzip to zstd defaults too https://community.centminmod.com/thr...g-sizes.16371/ and disk space savings on very large logs is quite noticeable

                      Note though zstd by default can consume quite a bit more memory than lz4, pigz, pbzip2 (multi-threade bzip2) so might not be suited to all situations out of the box unless you tune zstd options to use 33-66% less memory at the expense of compression ratio somewhat at higher levels of compression.

                      Also when comparing zstd note the specific version of zstd used as zstd 1.3.4+ and higher releases have had very noticeable performance improvements https://github.com/facebook/zstd/releases

                      Comment

                      Working...
                      X