Fedora 32 Looking At Using EarlyOOM By Default To Better Deal With Low Memory Situations

[polarathene]

Why not just fix the kernel's OOM killer? Other OS's don't hang up like Linux does in that situation..

Ehh.. macOS isn't great when it has this problem in my experience, Windows did it better, but in some cases can be no different to the result of just pressing reset button, I wouldn't say that's a massive improvement.

I remember Windows would try notify the user about deciding on something, or if the memory pressure was too much, it'd automatically kill the process, or in some cases, everything died and I was pushed to login again(Windows 10 in 2017 iirc, with a program running that I wrote which accidentally allocated the full 128GB of RAM and still wanted more for reading a 40GB binary file into memory/structs).

macOS in 2016, I had an experience where the kernel process(kext?) was either eating up a tonne of CPU or memory, and I think showed heavy swap occurring. When that happened mouse/keyboard input was delayed 10-30 secs, not unlike what you'd get with Linux. I couldn't really interact with the system to kill any process, I think it took around 20-30 minutes to receive enough input from me to kill off some programs to free up memory. One notable thing that macOS did handle differently, was the busy beachball spinning cursor was buttery smooth animation, unaffected by what was going on. The docks app icons also responded as if nothing was wrong, with an app bouncing up and down to alert me for attention. So macOS prioritizes aesthetics over usability, make it look pretty rather than be able to use your mouse/keyboard to fix the problem in a timely manner..

[k1e0x]

Ehh.. macOS isn't great when it has this problem in my experience, Windows did it better, but in some cases can be no different to the result of just pressing reset button, I wouldn't say that's a massive improvement.

I remember Windows would try notify the user about deciding on something, or if the memory pressure was too much, it'd automatically kill the process, or in some cases, everything died and I was pushed to login again(Windows 10 in 2017 iirc, with a program running that I wrote which accidentally allocated the full 128GB of RAM and still wanted more for reading a 40GB binary file into memory/structs).

macOS in 2016, I had an experience where the kernel process(kext?) was either eating up a tonne of CPU or memory, and I think showed heavy swap occurring. When that happened mouse/keyboard input was delayed 10-30 secs, not unlike what you'd get with Linux. I couldn't really interact with the system to kill any process, I think it took around 20-30 minutes to receive enough input from me to kill off some programs to free up memory. One notable thing that macOS did handle differently, was the busy beachball spinning cursor was buttery smooth animation, unaffected by what was going on. The docks app icons also responded as if nothing was wrong, with an app bouncing up and down to alert me for attention. So macOS prioritizes aesthetics over usability, make it look pretty rather than be able to use your mouse/keyboard to fix the problem in a timely manner..

The oom killer situation has nothing to do with swap. Thrashing swap is a .. I guess related.. but different problem. Linux should cleanly call oom, terminate a process (however it decides, even at random) and recover.. It doesn't do that and stalls for large amounts of time even with no swap.

[polarathene]

1. When the system is starving for RAM there might be no resources or/and time to ask the user to terminate something.
2. The user might be away and by the time he decides what's the best process to kill the system will be completely down.
3. The user might not be running GUI at all or his session might be down itself.

1. The system could reserve a portion of RAM just for itself that other processes can't contend for? How much is it going to need to make a notable difference? Probably not a tonne that it'd bother anyone, especially if tunable. I think this actually exists and just the default might not be ideal, I know I've got a link about understanding swap/OOM properly from an experienced dev that deals with some common myths/misunderstandings, and explains why something happens and how to actually deal with it.

2. The system can via it's own monitoring, ideally know when that point of no return is approaching and the user input window is lost, so it takes over. AFAIK, Windows does this, it'll prompt the user, but if the situation is getting too bad it takes the situation into it's own hands and either kills a process or the entire session requiring login again(I've experienced this).

3. A generic way that isn't DE/GUI specific, might be to just switch to a new TTY with an interactive prompt, it won't be pretty or well integrated(into the users DE or whatever they've got setup), but should work for most use cases? With #1 and #2 addressed, that should make that rather possible. A better option would be to send a signal/event that some other process can listen to and respond to in a similar way, allowing for DEs and the like to handle it in their own environment, assuming they can do so effectively in time(but #2 would kick in if not).

[polarathene]

The oom killer situation has nothing to do with swap. Thrashing swap is a .. I guess related.. but different problem. Linux should cleanly call oom, terminate a process (however it decides, even at random) and recover.. It doesn't do that and stalls for large amounts of time even with no swap.

OK, so given the two experiences shared with Windows and macOS, how are they handling out of memory situations better?

macOS just becomes as unresponsive as Linux tends to, and if it didn't perform swapping I imagine would be no different either? Windows does it alright, except when the entire session is killed.

[k1e0x]

1. The system could reserve a portion of RAM just for itself that other processes can't contend for? How much is it going to need to make a notable difference? Probably not a tonne that it'd bother anyone, especially if tunable. I think this actually exists and just the default might not be ideal, I know I've got a link about understanding swap/OOM properly from an experienced dev that deals with some common myths/misunderstandings, and explains why something happens and how to actually deal with it.

2. The system can via it's own monitoring, ideally know when that point of no return is approaching and the user input window is lost, so it takes over. AFAIK, Windows does this, it'll prompt the user, but if the situation is getting too bad it takes the situation into it's own hands and either kills a process or the entire session requiring login again(I've experienced this).

3. A generic way that isn't DE/GUI specific, might be to just switch to a new TTY with an interactive prompt, it won't be pretty or well integrated(into the users DE or whatever they've got setup), but should work for most use cases? With #1 and #2 addressed, that should make that rather possible. A better option would be to send a signal/event that some other process can listen to and respond to in a similar way, allowing for DEs and the like to handle it in their own environment, assuming they can do so effectively in time(but #2 would kick in if not).

I'm stunned by this. All of this is bloat and craziness. All of these are large frameworks that take compute and resources from systems starved for them.

I really don't care how it makes the decision on what to kill.. you've already blown a tire. Let's not roll the car over and crash.

Windows GUI elements and OOM elements are probably different. OOM probably just kills the user session. The GUI element I'd imagine is dumb. Windows seems to handle this situation pretty well actually. I've seen this happen before and it tends to kill explorer.. It does slow down and kind of get jerky but it's not too bad.

I haven't tested macOS but I can get back to you. I'd expect it to act like FreeBSD and kill a process at random (this is the classic behaviour for oom killers) .. no it's not good.. but you've already blown a tire. I think there is a method to whitelist a process so it isn't targeted also. Looking at how Solaris deals with this situation might be useful too.

Linux does all kinds of pathological behaviour like writing to the disk (yes even without swap) and becomes unresponsive. It seems to not respond well to this..

[CommunityMember]

OK, so given the two experiences shared with Windows and macOS, how are they handling out of memory situations better?

The grass is always greener over the septic tank? More to the point, the answer is that each OS (Windows, MacOS, FreeBSD, Linux, ....(*)) addresses the issues slightly differently, and will have different experiences for some use cases, but still fails under other uses cases. Which returns to the point that there is no known best solution which works well for everyone all of the time, as the needed ML model for determining best has not yet been refined (sounds like an interesting research topic). Facebook, which created oomd, is apparently moving on to a oomd2 (or oom2d?) as while oomd worked much better, it still had edge case issues, and they have a fairly well controlled workload. Many people here have their own idea of a heuristic that will work for them. Please develop that idea into actual code, and contribute.

(*) 40+ years ago on IBM mainframes the same issues were experienced, too. This is not a new problem.

[Raka555]

The whole way that memory is managed at the moment is actually wrong.
We are trying to map bare metal memory semantics from 30+ years ago onto modern kernels with virtual memory.

void *mem=malloc(some_big_amount) will almost never fail as modern kernels always over subscribe the memory.(EDIT: To turn off overcommit is not a workable solution,. Try it an you will see)
So user space programs have no mechanism to cope with what is going to happen next.
When they actually try to use that memory, all kinds of havoc is going to break loose when the OS can't actually supply the demand.

The problem should be rethink at the application level as well as the OS level.
At the moment the two are disconnected and creating these problems.

Something like:
void *mem=malloc(some_big_amount, some_function_to_cope_with_oom_for_this_allocation );

Would already be a big improvement. Or languages with Exceptions can catch and handle the situation.
When they get the signal from the OS and they are in a position to release memory or exit cleanly, they can do so.

At the moment there is no mechanism for user space to handle it.

POSIX, please step into the 21st century wrt memory management !

Why aren't there at least an OOM signal here ? http://man7.org/linux/man-pages/man7/signal.7.html

[polarathene]

The *real* problem and the one that needs to be addressed is not how Linux deals with that problem, it's why Linux hangs and gets into a unresponsive state when it is in that situation.

I think oomd by facebook seems to tackle the situation well?

https://facebookmicrosites.github.io/oomd/

From the github README:

This is sometimes a slow and painful process because the kernel can spend an unbounded amount of time swapping in and out pages and evicting the page cache.

oomd aims to solve this problem in userspace. oomd leverages PSI and cgroupv2 to monitor a system holistically. oomd then takes corrective action in userspace before an OOM occurs in kernel space.

In practice at Facebook, we've regularly seen 30 minute host lockups go away entirely.

And then from EarlyOOM which this article is about:

https://github.com/rfjakob/earlyoom

This may be part of the reason Linux invokes it only when it has absolutely no other choice.
It will swap out the desktop environment, drop the whole page cache and empty every buffer before it will ultimately kill a process.

earlyoom checks the amount of available memory and free swap up to 10 times a second (less often if there is a lot of free memory). By default if both are below 10%, it will kill the largest process (highest oom_score).

They each seem to have pro/cons depending on context, as this issue considering including Facebooks PSI into earlyoom discusses:

https://github.com/rfjakob/earlyoom/issues/100

Unfortunately the PSI support relying on cgroups support being present, doesn't seem to play well with MuQSS which only has partial support for cgroups afaik?

---

Is there information about Windows/macOS handling it? Is it all handled via the kernel, or do they use userspace as well like the oomd/earlyoom solutions?

[F.Ultra]

The whole way that memory is managed at the moment is actually wrong.
We are trying to map bare metal memory semantics from 30+ years ago onto modern kernels with virtual memory.

void *mem=malloc(some_big_amount) will almost never fail as modern kernels always over subscribe the memory.
So user space programs have no mechanism to cope with what is going to happen next.
When they actually try to use that memory, all kinds of havoc is going to break loose when the OS can't actually supply the demand.

The problem should be rethink at the application level as well as the OS level.
At the moment the two are disconnected and creating these problems.

Something like:
void *mem=malloc(some_big_amount, some_function_to_cope_with_oom_for_this_allocation );

Would already be a big improvement. Or languages with Exceptions can catch and handle the situation.
When they get the signal from the OS and they are in a position to release memory or exit cleanly, they can do so.

At the moment there is no mechanism for user space to handle it.

POSIX, please step into the 21st century wrt memory management !

Well you can always disable memory overcommit completely to make malloc() return NULL on OoM. I would however say that there would not be a whole lot that most userspace applications could do anyway in that case to relieve the situation. I mean it's not like the application did malloc/new just for the fun of calling a function so the appropriate action in most cases should be to simply kill the application and then we are back with the OoM-killer again.

People keeps on claiming that WIndows "does this right", is that a recent change? I haven't really used Windows since W2K and back then Windows had no OoM-killer at all and neither did it handle the situation gracefully. All it did was increasing the size of the swap-file which completely froze the whole machine for for long periods while every application was being swapped in and out or ram/disk.

edit: So I looked it up and Windows 10 does not have a OoM-killer, nor does it overcommit (it hands out more RAM if there is free swap available but that is it) so that probably explains why people see differences here since they simply didn't run as much applications when they tested the situation on Windows.

[polarathene]

The problem should be rethink at the application level as well as the OS level.
At the moment the two are disconnected and creating these problems.

Why aren't there at least a OOM signal here ? http://man7.org/linux/man-pages/man7/signal.7.html

Your suggested language fix would need to be supported by many languages, and wouldn't really help with software prior to that which doesn't use it. The OS could probably handle it accordingly without applications having to be updated/"fixed" to handle it. Also how would that work with languages that aren't as explicit with memory allocation? Like script languages with runtimes? I'd imagine the proposed solution wouldn't be too different to that experience, in which case it'd be no different to having it implemented at OS level and leave the application layer alone as-is?

nohang I think sends SIGTERM, and if the process doesn't behave under some conditions it will follow up with SIGKILL. Is an OOM specific signal useful there to behave differently? (free up some memory to avoid being killed/terminated if possible I guess?)

There is this project low-memory-monitor that provides some support. It will send out signals that your app can react to, or use the GMemoryMonitor API being worked on for glib if using that, or a D-BUS API apparently?(I haven't looked into it much). At least that seems to be going in the right direction, with the glib support.

Many people here have their own idea of a heuristic that will work for them. Please develop that idea into actual code, and contribute.

There's already quite a few oom solutions out there now. Many come with plenty of config/tunables to adjust to each users specific needs if the defaults provided aren't suitable.