Announcement

Collapse
No announcement yet.

EXT4 In Linux 5.6 To See Big Write Performance Boost For Direct I/O

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by Azrael5 View Post

    I don't know if ext4 is envolved in the issue, however I have 8GB of RAM and once they are all occupied the swap partition doesn't help and the computer crashes. I assume that ext4 is envolved in file management. The computer begins to take access of hard drive continuously making the computer unusable once the memory is full.
    First, if you do have a swap partition, it's not using ext4, so ext4 is definitely not involved.
    Second, when I filled my computer's RAM (thanks, Chrome/Chromium), it didn't crash even if I had no swap at all. It just became really, really unresponsive. To the point it would take a minute or so register a mouse click and close a browser window and free up some RAM. That said, it's expected for a computer that's out of RAM to become unresponsive, regardless of the OS used. It's not expected to crash though.

    Fwiw, if you have the swap on a HDD, that may/will behave differently.

    Comment


    • #12
      Originally posted by bug77 View Post
      First, if you do have a swap partition, it's not using ext4, so ext4 is definitely not involved.
      You *can* add a swapfile on ext4 (a swapfile is not a swap partition of course, but they both provide swap space to the kernel and many people are ignorant about the differences since they operate the same).

      The real issue is that once you move to excessive ram pressure, you will encounter a number of well known artifacts, such as ejecting shared libraries, which results in re-reading them again and again from your system disk. It can get ugly fast. This was discussed back in an reference on this site back in August: http://www.phoronix.com/scan.php?pag...es-Bad-Low-RAM

      Comment


      • #13
        Originally posted by bug77 View Post

        First, if you do have a swap partition, it's not using ext4, so ext4 is definitely not involved.
        Second, when I filled my computer's RAM (thanks, Chrome/Chromium), it didn't crash even if I had no swap at all. It just became really, really unresponsive. To the point it would take a minute or so register a mouse click and close a browser window and free up some RAM. That said, it's expected for a computer that's out of RAM to become unresponsive, regardless of the OS used. It's not expected to crash though.

        Fwiw, if you have the swap on a HDD, that may/will behave differently.
        I cannot unlock the computer because of the crash. Both mouse and keyboard ore not responsive while the hard drive works continuously reading and or writing. this happen when I have opened since 100 or 200 link in chrome and or firefox and other programs are running. some times I have many tasks running because I have to work by all of them. However if ext4 is not responsible of the crash I don't understand which the cause is.

        Comment


        • #14
          I feel for the person struggling with OOM, but can you post a new topic and let things thread there?

          Comment


          • #15
            Well, there's not much to say about the corner case Direct I/O performance boost.

            Wrt Linux OOM, I do think it should lock the GUI and terminals and basic shell tools into memory, such that top / ps / kill / ssh / etc and the mouse/kb always work. I understand the arguments either way, but that would be my preference. ie, it's generally one program that balloons ... nothing should swap out for the offender. Let the offender swap out some of its bloat if it wants to keep going. I have 32GB my machine should never freeze up because one program got out of control requesting memory. It's a different story when a machine only has 1GB and it has to swap when switching from Firefox to Emacs.
            Last edited by xorbe; 03 January 2020, 04:14 PM.

            Comment


            • #16
              Originally posted by cjcox View Post
              I feel for the person struggling with OOM, but can you post a new topic and let things thread there?
              Azrael5 Well, Michael apparently read your mind and decided to make a thread for you: https://phoronix.com/forums/node/1149881

              Comment


              • #17
                EXT4 always gets performance improvements, there's no stop. It seems to be the preferred child in the Linux family. When it will not be possible?

                Will there be a filesystem as fast or faster than EXT4 yet providing all the features of more modern filesystems such as ZFS and Btrfs?

                Comment


                • #18
                  Originally posted by Azrael5 View Post
                  this happen when I have opened since 100 or 200 link in chrome and or firefox and other programs are running. some times I have many tasks running because I have to work by all of them.
                  This is probably your problem right there. I have even more browser tabs and other applications running for same reasons, but my machine has 32GB of RAM(often using 16-20GB, sometimes I get to 30GB and start using my swap on an SSD disk).

                  You need to better manage the memory for this use case. If the browser tab just has to be open, but you don't need anything highlighted or text field inputs saved, try a plugin like TheGreatSuspender, I'm not sure if that keeps scroll positions if that's important to you, if it is, I think that can be saved with some Session saving extensions/plugins. I use Session Buddy as a fallback to Chromes own session restore. TheGreatSuspender will unload the browser tab for you freeing up the memory it used, and when you click the tab again, it will reload/restore it. Session Buddy saves the browser session(all windows/tabs), sort of like an advanced bookmark, you can name it and reload at a later point, useful for grouping windows/tabs. There is another one that might be more useful for that, I think it's called OneTab, which gives you a bit more control/flexibility.

                  Other than that, keep an eye on memory and try not to get close to filling it I guess.. or if possible upgrade the system with more RAM, and for the swap issue, if you could use an SSD instead of HDD, it'll go a lot more smoothly. The HDD likely has poor random read/write performance, and once you're paging out memory from RAM To disk(swap), it gets too slow and everything slows down as a result to speed of a snail..

                  You can google some tweaks to apply to your system. One is the vm.swappiness value iirc, you might want to trigger swap usage earlier, so that memory that isn't being used frequently and just sitting there is moved to disk earlier. Another useful tweak for you would be to enable zram or zswap. zram may help "increase" your RAM, as it compresses memory, allowing you to fit more into RAM, the tradeoff though will be CPU time. zswap is similar, but for once your storing stuff in swap on disk, something that you generally want to avoid in your case I think.

                  One final tweak to assist with the HDD woes when swapping, change the disk I/O scheduler to BFQ if it's not already. This should give you a more responsive system. If the crashing you mention really is the entire OS, and not apps or a very slow to respond OS(usually the experience), this probably won't help and you might find that the HDD/partition is out of space(unrelated to swap if using swap partition vs swap file), you mention the 8GB RAM gets filled, but not your swap size, and how full that gets, if it fills up completely too, I imagine that's a bad time.

                  Comment


                  • #19
                    How does this Direct I/O block overwriting thing work at that speed? With HDDs you overwrite the blocks, but on SSDs, you're only overwriting a virtual block mapping, while the SSD itself is managing the blocks under the hood, and afaik never overwrites data inline, it just writes to it's own blocks, then erases the old references to free up for writing to again, it's CoW like SMR drives can be?

                    I know there are SSDs available that drop that abstraction/virtual layer and allow you to manage the blocks directly, but that's uncommon afaik, I don't think traditional filesystems work with those either? I am not familiar with the particular Optane DC Persistent Memory disk/device, I know of those small Optane disks(32GB) and the larger capacity 900p series, is it those or something else?

                    Majority of SATA/NVMe SSDs(excluding Optane, does it even use either of those protocols?) don't benefit from this change right? HDDs and those PMEM storage devices do however.

                    Comment


                    • #20
                      Originally posted by Azrael5 View Post

                      I cannot unlock the computer because of the crash. Both mouse and keyboard ore not responsive while the hard drive works continuously reading and or writing. this happen when I have opened since 100 or 200 link in chrome and or firefox and other programs are running. some times I have many tasks running because I have to work by all of them. However if ext4 is not responsible of the crash I don't understand which the cause is.
                      which task requires you to view 200 tabs at once? I bet there are many unnecessary tabs that you can close. Some tabs can be delayed load and some others can be removed from memory temporarily. AFAIK newer Chrome versions automatically unload many tabs from memory, and Firefox will delay load tabs from the previous session

                      But the first suggestion is to use zram (which is far faster than normal swap) and lower the priority of the swap partition

                      Comment

                      Working...
                      X