MGLRU v14 Released For Improving Linux Low-Memory Performance

[phoronix]

Phoronix: MGLRU v14 Released For Improving Linux Low-Memory Performance

While Linux 6.0 will bring a lot of shiny new features, Multi-Gen LRU (MGLRU) is one of the anticipated changes that isn't going to land now until Linux 6.1. But in the interim, MGLRU v14 was posted today that re-bases the code against 6.0-rc1 to help facilitate more testing of this kernel change that primarily helps Linux systems under memory pressure...

https://www.phoronix.com/news/MGLRU-v14-Released

[direc85]

85% reduction in OOM kills is the best part in the details, and the report from an Arch Linux user whose system kept responsive instead of swapping to hell and back!

[uid313]

How does other operating systems such as Windows solve this?

[Mitch]

How does other operating systems such as Windows solve this?

I'm unaware of what Windows does exactly, but I have a history working with Windows servers and Windows client machines that are hugely underspec'ed. It's functional, but it's not pretty. There's thrashing, crashing, and spinning everywhere. Even being less than 1GB over-allocated can result in stalls, crashes, and bluescreens in any scenario. The Windows ecosystem is very ram-hungry, and Windows can't necessarily decide what memory needs to stay and what can be swapped. It will not let you tune how proactively it swaps, nor whether it should favor some apps' memory over others, nor whether you'd like more file-based swapping vs applications. It has optional memory compression (like ZRAM) but that's also not configurable.

Two of the big MS apps that understandably use a ton of RAM but are NOT as RAM-friendly as advertised are SQL Server (Microsoft's version of SQL, aka T-SQL) and Tabular SSAS. They usually need to be tuned, tamed, and spec'ed based on how much data your organization is using and how many people are using that data. If you don't tune them, they'll often battle the OS itself, or your management applications for RAM. The people I work with often think that 95% disk or RAM utilization means we're running efficiently, rather than running off a cliff. Fortunately, I'm not usually the one chasing crash logs every week, but when I backfill, I tell them we need more resources.

[stormcrow]

How does other operating systems such as Windows solve this?

Windows automatically assigns swap space in a file - unless you're an idiot and limit it in some way.

MacOS does the same. The gist is you don't run out of RAM so long as you have storage space available. It might make things slow(er), but the system will eventually respond an you can manage the situation. Linux is the only OS that willy-nilly kills off processes regardless of importance. It's a stupid, flawed from the start design.

[blackshard]

Windows automatically assigns swap space in a file - unless you're an idiot and limit it in some way.

MacOS does the same. The gist is you don't run out of RAM so long as you have storage space available. It might make things slow(er), but the system will eventually respond an you can manage the situation. Linux is the only OS that willy-nilly kills off processes regardless of importance. It's a stupid, flawed from the start design.

And then you end up with an unresponsive system because it goes into trashing if a userspace process leaks heavily. The matter is like choosing what kind of death you wish.

[Mitch]

And then you end up with an unresponsive system because it goes into trashing if a userspace process leaks heavily. The matter is like choosing what kind of death you wish.

I appreciate some ability to swap, as it's a superior canary-in-a-coal-mine scenario. Instead of programs dying and crashing or the system going down, we see our swap get hammered and some pausing, stuttering, and load. In practice, this doesn't always save us from the worst, but it will often give us a head start to begin manual intervention before the kernel or other programs step in.

I'm really excited for MGLRU. I'd love to have an elegant over allocation solution. If only they'd let me use Linux at work (they actually would if I weren't IT).

[timrichardson]

How does other operating systems such as Windows solve this?

Windows has a somewhat different model for managing memory. It does not do overcommit. The memory you have is ram + swap. Compressed ram in windows optimises swapping but it does not increase the commit limit, I think, so I conclude it can not be as useful as compressed memory in macos or linux.

macos and linux do overcommit: the OS pretends there is a lot more memory than there actually is. But the OS can not actually allocate more memory than ram + swap.
macos's major escape from out of memory is to dynamically increase the swapfile. There is a package for linux which does that too. It is called swapspace in Ubuntu. You just install it. I tested it just now, and it seems to work well.

So out of the box, macos is perhaps the "best" of both worlds: over-commit and dynamic swap.

You can turn off overcommit in linux, and tune it in other ways, but turning it off is regarded as a specialised tweak for very specific server loads, when you have confidence that the applications are looking after memory, such as a database server.

On windows, application developers are supposed to expect memory allocation failure and they are supposed to code for it. How well they do that is up to the application developer. THis model has some obvious weaknesses: how does an application know whether it should wait and hope that some other application deals with low memory? For instance, if I stress a windows 10 vm by loading lots of Firefox tabs, it just crashes. Firefox gives an apology that it crashed, but no explanation.
If the app ignores it, Windows will supposedly notify the user that the user must close something ... ignore that, windows will crash. But it can certainly just crash. On another run, I was loading up Chrome with 100 tabs (of course, they did not all load), then with memory full, I tried to start Firefox ... it immediately crashed. Then I started Excel ... Windows interactivity is much better than Linux under this circumstance. Excel opened ... and then a kernel crash, the whole windows session dies. There was no warning. Even under this situation with 4GB of ram and 2GB of swap used, compressed ram was reporting as <200MB. Zram or zswap in linux does much, much better, because of over commit. In my testing, the Windows swap file was limited to 2GB.

Linux applications are allowed to assume every request for memory will succeed. The OS takes responsibility for out of memory. This is the kernel oom killer (which works much better with MGLRU in my testing of a 2 core 4 GB gnome VM). THere are background watchdogs which kill applications before the kernel, such as systemd-oomd or earlyoom. Whatever ... in Linux, some central administrator must do it, and such an administrator process could be taught to make decisions that suit the user ... this is what systemd-oomd is hoping to achieve, although it doesn't work very well yet. But Fedora, Ubuntu and the systemd project are working on improvements. While it is not very good at the moment, it has a lot of promise. Right now, though, MGLRU makes by far the biggest improvement. You can try it in various custom kernels, such as xanmod.

A 4gb Linux will VM will take a bigger load before trouble hits than a Windows Vm with the same fixed swap size. The effect of over commit seems to be that it buys time for the OS to recover, under the hope that the high memory load is transient. If this is true, it works, and we get recovery in a situation where Windows would have at best killed things and at worst just crashed. I don't know if this is actually a design goal of over-commit.

If it is not true, the system can appear to stall (but it almost never actually crashes, leaving the door open to some recovery techniques, even if you just force a disk sync). Overall I think it is better than Windows. But it is not ideal.


I don't know what macos does apart from growing the swap file. That is obviously a complete hack but I suppose it attempts to buy time.
EDIT: By default, Windows will do the same thing. So this is a type of overcommit: slow, but in principle it guarantees each memory request is honoured, even if the system if not practical to use. Which means we now arrive at same horrible performance that Linux offers, except that compressed memory is not effective in Windows. I guess if that's your cup of tea, you could look into 'swapspace' on linux.

To get a good low memory linux experience at the moment, use a MGLRU kernel, use swap of 2GB and use zswap. And if you still have problems, use swapspace. Also, remember that all these solutions for low memory are designed to get you through short term demands without the system crashing or becoming unusable until the short term demand passes. If you frequently need more RAM than you have, even the best optimised OS can't fix swap being much slower than RAM.

[Lbibass]

Windows has a completely different model for managing memory. It does not do overcommit. The memory you have is ram + swap. Compressed ram in windows optimises swapping but it does not increase the commit limit, I think, so I conclude it can not be as useful as compressed memory in macos or linux.

macos and linux do overcommit: the OS pretends there is a lot more memory than there actually is. But the OS can not actually allocate more memory than ram + swap.
macos's major escape from out of memory is to dynamically increase the swapfile. There is a package for linux which does that too, I saw. It is called swapspace in Ubuntu.

You can turn off overcommit in linux, and tune it in other ways, but turning it off is regarded as a specialised tweak for very specific server loads, when you have confidence that the applications are looking after memory, such as a database server.

On windows, application developers are supposed to expect memory allocation failure and they are supposed to code for it. How well they do that is up to the application developer. THis model has some obvious weaknesses: how does an application know whether it should wait and hope that some other application deals with low memory? For instance, if I stress a windows 10 vm by loading lots of Firefox tabs, it just crashes. Firefox gives an apology that it crashed, but no explanation.
If the app ignores it, Windows will supposedly notify the user that the user must close something ... ignore that, windows will crash. But it can certainly just crash. On another run, I was loading up Chrome with 100 tabs (of course, they did not all load), then with memory full, I tried to start Firefox ... it immediately crashed. Then I started Excel ... Windows interactivity is much better than Linux under this circumstance. Excel opened ... and then a kernel crash, the whole windows session dies. There was no warning. Even under this situation with 4GB of ram and 2GB of swap used, compressed ram was reporting as <200MB. Zram or zswap in linux does much, much better, because of over commit. In my testing, the Windows swap file was limited to 2GB.

Linux applications are allowed to assume every request for memory will succeed. The OS takes responsibility for out of memory. This is the kernel oom killer (which works much better with MGLRU in my testing of a 2 core 4 GB gnome VM). THere are background watchdogs which kill applications before the kernel, such as systemd-oomd or earlyoom. Whatever ... in Linux, some central administrator must do it, and such an administrator process could be taught to make decisions that suit the user ... this is what systemd-oomd is hoping to achieve, although it doesn't work very well yet. But Fedora, Ubuntu and the systemd project are working on improvements. While it is not very good at the moment, it has a lot of promise. Right now, though, MGLRU makes by far the biggest improvement. You can try it in various custom kernels, such as xanmod.

A 4gb Linux will VM will take a bigger load before trouble hits. The effect behind over commit seens ti be that it buys time for the OS to recover, under the hope that the high memory load is transient. If this is true, it works, and we get recovery in a situation where Windows would have at best killed things and at worst just crashed. I don't know if this is actually a design goal of over-commit.

If it is not true, the system can appear to stall (but it almost never actually crashes, leaving the door open to some recovery techniques, even if you just force a disk sync). Overall I think it is better than Windows. But it is not ideal.


I don't know what macos does apart from growing the swap file. That is obviously a complete hack but I suppose it attempts to buy time.
EDIT: By default, Windows will do the same thing. So this is a type of overcommit: slow, but in principle it guarantees each memory request is honoured, even if the system if not practical to use. Which means we now arrive at same horrible performance that Linux offers, except that compressed memory is not effective in Windows. I guess if that's your cup of tea, you could look into 'swapspace' on linux.

Thank you very much for this explanation! It's super helpful.

[binarybanana]

It's huge improvement on my 256MB SBC (my router/firewall/wireless AP/etc). It stays responsive like nothing is going on even as it swaps like crazy to a Samsung SD card. Before it became quite laggy, although still somewhat responsive. And back when I still used some no name card swapping meant almost certain death... I wonder how long the Samsung SD card will hold up, but so far it has worked without issues for over half a year. FWIW, I only use 8GB of the 128GB capacity, which probably helps.