Ubuntu 22.04 LTS Integrating systemd-oomd For Improving Low Memory Handling

[timrichardson]

It is strange that Ubuntu has not previously used a solution like this. I've been using earlyoomd on all my low memory servers and all my Ubuntu desktops for years. I thought among even moderately experienced users installing earlyoomd was one of the first post-install Ubuntu tweaks that people did, but reading the comments here, it seems not.
earlyoomd is very mature, very light and has been in the ubuntu repositories for years.

I've been using Fedora on my laptop for a couple of years. The first one I used had earlyoomd, then Fedora changed to the systemd approach. Both work, and kudos to Fedora for sensible defaults.

[ssokolow]

The article reads "50% memory pressure limit for user sessions". Is memory pressure the same as memory usage? If so that seems low. Why not allow the user to use something like 90% ram?

I often run memory intensive computational tasks, though I tune them to leave a couple of GB (out of my 32 GB) to the system, X etc.

Agreed. I use earlyoom (a precursor/competitor to systemd-oomd) and set it to trigger at 90%, prefer browser content processes, and avoid stuff that'd be a pain to bring back or would kill my X session.

I'd paste my /etc/default/earlyoom options, but this forum is so determined to "403 Forbidden" it that, after I copy-pasted it piecemeal to avoid "403 Forbidden" on Ctrl+V, selecting it and attempting to fix the formatting caused a 403 Forbidden that nuked my previous copy of this message with no working Ctrl+Z.

(I also have 32 gigs but chose 90% back with my previous 16 gigs, where I found that thrashing tended to start a little after the system had less than 10% of that for disk cache.)

It also helps to pair it with zram-backed swap, since the kernel does need some swap for memory defragmentation to work properly, but backing it with zram is a good compromise since you don't get the slow-down (and, if you have an SSD, the wear) of disk-backed swap and stuff in long-running processes that's been loaded but never/almost never gets touched will get compressed eventually.

[arQon]

Replacing one shitty OOM manager with another equally-shitty but newly-trendy one doesn't do anything at all to help with the underlying problem, which is that Linux's memory-exhaustion handling is just utter garbage and always has been.

Short of getting a better *core* implementation for that piece of things (Michael has covered at least one of them very recently) the only approach that works acceptably well is to avoid exhaustion scenarios in the first place. Whether you do that by adding RAM, using zswap, or any other means is almost irrelevant: the only thing that matters is that if your OOM killer is triggering for any reason other than a thoroughly-broken piece of code with a rampant memory leak, the system is probably going to be completely nonfunctional for far longer than it would take to just reboot the machine and restart everything, and almost equally likely to lose data either way.

On a desktop, you can probably cobble together a "working" approach by using the absolute bare minimum of swap (ideally "none") and/or having something aggressively kill the hungriest process right before the OOM hits, since that will generally be a browser dealing with some clown-tier webpage - but for Firefox at least that risks it also taking out the complex bug report / long forum post / etc in an unrelated tab. Even so, it's still a better outcome than watching the machine go into swap death, waiting 10 minutes for it to fail to recover, and having to just power-cycle it anyway.

[Danny3]

I'm sure it will come with an exception for Snap crap or other shitty things from Canonical so they can consume as much memory as they want or never bekilled.

[ssokolow]

a thoroughly-broken piece of code with a rampant memory leak

In other words, a browser tab where you've allowed JavaScript and its fifty friends?

but for Firefox at least that risks it also taking out the complex bug report / long forum post / etc in an unrelated tab

If they haven't fixed that already, they will be soon as part of the Spectre and Meltdown protections. I believe they're calling it Project Fission.

[sb56637]

From what I understand systemd-oomd works on the cgroup, so for most desktops outside of Gnome that would mean it takes down the entire desktop session if it has to intervene. Maybe someone who understands better could comment on this.

[hakavlad]

Pity this won't get backported to CentOS 7 for work. Hate it when OOM goes on a rampage and the server has to be restarted.

You can use nohang on CentOS 7.

https://github.com/hakavlad/nohang

[hakavlad]

Nice, but why not le9 with prelockd? Been using it since last month and never had the system hang during memory pressure ever again.

No one is interested in these amateur creations. system-oomd is a proven solution from respected developers.

[tildearrow]

No one is interested in these amateur creations. system-oomd is a proven solution from respected developers.

It's very sad because you truly did an amazing job fixing the hang problem.
I was trying to do a speedrun a week ago, and the timer had a memory leak problem which caused it to eat the system's memory.
Had it not been for le9 + prelockd, my system would have hung and been unresponsive without explanation. With your patches, the timer promptly closed, and the game kept running without a single lag spike.

Have you tried sending the patch to Linus directly? I know it's mostly tuning options, but these really come in handy.

[hakavlad]

I had major issues in OOM situations on super powerful servers, small servers, desktops, laptops, hdds and ssds for many years. One of the things I never understood, why Linux seems to handle memory-pressure situations so poorly.

Michal Hocko answers:

https://lore.kernel.org/lkml/YbcNUEZ...hcp22.suse.cz/

Adding new knobs to improve performance under memory pressure is a pain for mm maintainers. In words they may be in favor of solving the problem, but when it comes to business, they will find thousands of objections (some of them, of course, are correct).