Amazon Working On New Proactive Memory Reclamation For The Linux Kernel

[phoronix]

Phoronix: Amazon Working On New Proactive Memory Reclamation For The Linux Kernel

For over a year Amazon engineers have been working on DAMON as a new means of monitoring data accesses under Linux. That patch series has yet to be mainlined but continues being worked on with the intention of getting it upstreamed when ready. More recently the engineers involved have been working on a DAMON-based page reclamation implementation for the Linux kernel in dealing proactively dealing with systems having high memory load...

https://www.phoronix.com/scan.php?page=news_item&px=DAMON-Page-Reclamation-RFC

[schmidtbag]

With numbers as precise as 1.448%, part of me thinks this has only been tested a couple times. In other words, there is nowhere near enough testing to know how safe and reliable this is.

[Jabberwocky]

Mainline with a config option. Please don't make it default.

This is default behaviour in FreeBSD and it was the last straw for me.

[skeevy420]

It finds memory regions that didn't accessed for specific time duration and page out...Using these, sysadmins can enable/disable it...

I installed it at my house and it's working as intended. It was able to find memory that paged out pretty quickly and then it disabled it for a while.

[Linuxxx]

Mainline with a config option. Please don't make it default.

This is default behaviour in FreeBSD and it was the last straw for me.

Could You elaborate on this, please?

What kinds of problems did it cause?

Thanks!

[Jabberwocky]

Could You elaborate on this, please?

What kinds of problems did it cause?

Thanks!

Server with RAM > 1TB kept writing to swap with hundereds of gigabytes still free, not even cached. I tried changing pagedaemon config but it didn't help. I could have spent more time on it or worked with people to fix the problem, but like I said there were other issues/missing-features too so I just gave up on it.

I still miss the simplicity and uniformaty of the OS. It makes automation very easy, you can just hack a few shell scripts. On GNU/Linux you need invest in software that abstracts configuraiton for you like puppet/chef/ansible or something like packer to build your own images. There's a lot of customization and things change frequently (in userland). Still I'd rather investment time in development of abstract mangement software, opposed to frantic debugging and calling datacentres to manually reboot my FreeBSD boxes (yes some places still don't have/allow IPMI).

[chuckula]

That's easy, just kill chrome every 15 minutes or so.

[RahulSundaram]

That's easy, just kill chrome every 15 minutes or so.

Didn't know of large swaths of people running Chrome on servers.

[Zan Lynx]

Server with RAM > 1TB kept writing to swap with hundereds of gigabytes still free, not even cached. I tried changing pagedaemon config but it didn't help. I could have spent more time on it or worked with people to fix the problem, but like I said there were other issues/missing-features too so I just gave up on it.

This doesn't sound like a problem, unless those writes interfered with useful work.

One of the pretty ideal memory management methods writes inactive pages out to swap ahead of time. As long as those pages don't change it then becomes very cheap to remove them, because they're already in swap. Note that in that system the pages are mirrored to swap, and not actually moved.

I haven't looked into the details of what these Linux patches do, but if the result is more swap use and faster memory paging then that sounds like a good plan. And I note that it was using zram so all the swapping would go to compressed RAM first, which is even better.

[Jabberwocky]

This doesn't sound like a problem, unless those writes interfered with useful work.

One of the pretty ideal memory management methods writes inactive pages out to swap ahead of time. As long as those pages don't change it then becomes very cheap to remove them, because they're already in swap. Note that in that system the pages are mirrored to swap, and not actually moved.

I haven't looked into the details of what these Linux patches do, but if the result is more swap use and faster memory paging then that sounds like a good plan. And I note that it was using zram so all the swapping would go to compressed RAM first, which is even better.

I could have been a bit more clear on paging vs swapping.

I totally agree. That feature would be very useful for many other scenarios (provided that there aren't any bugs).

The problem I had with paging would not have been that bad if the "coldness identification time threshold" was optimized for that server's workloads and if there wasn't such a massive difference between RAM capacity and I/O bandwidth. The SSD endurance cost would also have been wasted IMO but it's not such an immediate/obvious problem as the system grinding to a halt. Due to the intensity of paging combined with dynamic system activity it was a catastrophe.

I have had a related problem on Linux too, but that was purely because I did not have enough memory. Between Fedora 29 and 31 swap thrashing lockedup the system and recovery sometimes took hours. The solution was buy more RAM or not sue so much RAM/swap or the official solution https://fedoraproject.org/wiki/Changes/EnableEarlyoom it was annoying but still easier to solve than to fight a daemon that decides when it wants to page your data. Come to thing of it now... this method by Amzaon could have helped to avoid swap thrashing.

The frustration came in when the system was very capable of handling the workloads but the OS/configuration made a mess, especially since the previous OS release worked for that given workload and there was no mention of fundamental changes. The extent of the mess was also very frustrating, sometimes a hard rest was required, other times processes entered state d and could not be killed with "-9" even after it looked like the system recovered. This required a soft reboot. That specific system was a huge monolith and even soft reboot was a massive PITA.