Announcement

Collapse
No announcement yet.

Fedora 32 Looking At Using EarlyOOM By Default To Better Deal With Low Memory Situations

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #61
    Mmmm... when you said there were at least four, I presumed that you talked about low memory monitors, but only the fourth is that; the other three are OOM Killers, which should be just a "last resource" option. Before doing that, all and every application in the system should have the opportunity of freeing some non-critical memory, and for that, only the fourth one is useful...

    Comment


    • #62
      The whole idea of OOM is wrong. The user doesn't want his process killed. When building a distribution with Yocto I want it built, even if it tries to build gcc and nodejs each with 8 threads simultaneously on a quad core + HT cpu.

      What actually needs to be done is restrict the number of swapin. This can in theory be implemented in the scheduler.

      That this will work can easily be verified by stopping (SIGSTOP) the offending process. It will get swapped out by the remaining processes, but not swapped in again and everything will return to normal. SIGCONT will continue it again.

      Of course the process needs to be allowed to run longer then the swapin time needed.

      Comment


      • #63
        Originally posted by Raka555 View Post
        In general I hate that the amount of dirty buffers scales with the amount of memory that you have.
        I now always run with the follow settings in my /etc/sysctl.conf:
        vm.dirty_bytes = 16777216
        vm.dirty_background_bytes = 4194304


        I never have any issues while copying large files any more.
        Thanks! It really made a huge difference while moving a 10 GB directory (before: frozen desktop, after: can use all sorts of programs while files are being moved).

        Btw beware of tlp's defaults: https://www.mail-archive.com/kernel-...msg359847.html

        Comment


        • #64
          Originally posted by halo9en View Post

          Thanks! It really made a huge difference while moving a 10 GB directory (before: frozen desktop, after: can use all sorts of programs while files are being moved).

          Btw beware of tlp's defaults: https://www.mail-archive.com/kernel-...msg359847.html
          tlp should really be phased out in favour of tuned. tuned is a modern replacement, so has a dbus service, isn't built with shell scripts and can be adapt to suggestions from powertop.

          Comment


          • #65
            Originally posted by Britoid View Post
            tlp should really be phased out in favour of tuned. tuned is a modern replacement, so has a dbus service, isn't built with shell scripts and can be adapt to suggestions from powertop.
            Never heard of it before. I'm going to try it instead of tlp, thanks.

            Comment


            • #66
              Originally posted by polarathene View Post

              I wrote a naive program in rust for reading binary data into memory to process, and while I wrote this at work on Windows 10 in 2017 it ended up filling up 128GB RAM in short time frame. I think at one point Windows properly terminated it, but another it somehow caused Windows to end the session closing all my open programs and losing any unsaved state..

              Rewrote the program to use a buffer and process the data in a stream(writing back to disk the processed version and appending to it), didn't exceed 500MB.

              Windows(at least 10, pretty sure I had bad experiences on 7/8 and earlier) probably still does the best job compared to my experiences with macOS and Linux by default for desktop users.
              Since AFAIK Windows does still not have any OoM-killer what happened was that your application terminated when malloc() returned NULL, could be some library function or rust function that have an assert around malloc that simply made it call exit(). But that is pure speculation from my part, as I've already said my Windows experience is from a long time ago.

              Comment


              • #67
                Originally posted by Raka555 View Post

                Have you tried running a kernel with overcommit off ?
                I tried that already back in 2012: https://ewontfix.com/3/
                You start getting memory allocation failures long before you reach the actual RAM limit in your machine. There is a reason why overcomit exits.
                Overcomit is not bad. It is our ability to deal with it in our apps that is bad.
                Well Linux is built for overcommit so I would think that the no-overcommit scenario is not that well tested in practice. What matters here though is that Windows does not have overcommit, there you can only allocate as much memory as there are available swap-space. Yes that swap-space can and do increase but that is different from the overcommit that happens in say Linux. So therefore the comparison that people make with Windows vs Linux is not really equal, when they tested a OoM scenario on Linux they put the VM-system through one hell of a lot more pressure than when they did the same on Windows due to Linux doing overcommitting.

                Originally posted by Raka555 View Post

                1) Languages with garbage collection could run the GC and give some memory back to the OS.
                2) Free some memory and see if it helps.
                3) Programs can try to shutdown as clean as is in their power with the memory they have.
                4) Those on an allocation spree might be made aware that they are the culprit.
                #1 Yeah I can give you that!
                #2 Why did your application allocate memory that it didn't need? Now this will only happen for special applications that use internal file buffering and those should be in the tiny minority if applications and totally changing the memory allocation routines for those few applications sounds like a waste of resources.
                #3 Then the problem is "faulty" to begin with, I'll explain a little bit more below.
                #4 And what should they do about it? Still as I've written over and over, the application allocated memory because it needed it and not just for the fun of it so what should it do with this information? It's not like every single userspace application runs a full AI that can rewrite the entire application logic when this happens. So in reality we are left with a single solution which is to kill the application outright and that is what the OoM-killer will do anyway so we have not solved anything.

                What I mean by "faulty" in #3 above is that our entire software industry is sadly still operating in the floppy+tape paradigm that we hardware wise left several decades ago. My claim here is that if your application have a "save" button then you're doing it wrong.

                In 2019 there does not exist a single reason why every single change to a document/spreadsheet/whatever is not immediately committed to disk with an infinite (where infinite is defined as the size of your total storage) undo disk buffer. There should be no reason for an orderly shutdown, nor should there be any "do you want to save?" option to the user when (s)he quits the application. You should be able to yank the power cord and boot your machine and be back exactly where you left of.

                That we in 2019 is not there is the major problem with software development and the real question to ask.
                Last edited by F.Ultra; 01-04-2020, 02:25 PM.

                Comment


                • #68
                  halo9en Yes. Problem solved.

                  Comment


                  • #69
                    Britoid The new MR makes the RT priority default. So code and capability will be tested more extensively. I think most people would prefer a kernel fix though..

                    Comment


                    • #70
                      Originally posted by tildearrow View Post
                      Why not disable paging executable code out at all and making it resident on memory at almost all times?
                      because executable code is not stored in memory. it is stored in filesystem, so it has to be paged in to run. so your clever idea should be reworded to something like "read all executable code to memory on boot and fail immediately due to memory exhaustion"

                      Comment

                      Working...
                      X