Fedora 34 Approved To Enable Systemd-OOMD By Default For All Spins

[phoronix]

Phoronix: Fedora 34 Approved To Enable Systemd-OOMD By Default For All Spins

The release of Fedora 34 this spring is now cleared to enable systemd-oomd by default for all spins in an effort to enhance the out-of-memory / memory pressure experience on Linux...

http://www.phoronix.com/scan.php?page=news_item&px=Fedora-34-Approved-OOMD

[birdie]

Changes/EnableSystemdOomd

What an asinine hasty decision:

• Can we exclude certain units from being killed?
  The proposal has been updated to include implementation of a knob to exclude units from being killed by systemd-oomd. The actual interface is to be determined (i.e. it's not yet implemented), but will likely be an extension of the ManagedOOM*= systemd properties.
• How will this work if everything is in the same cgroup?
  It will not work as systemd-oomd acts on a per-cgroup level. Applications will need to spawn processes into separate cgroups (e.g. with systemd-run) or use a desktop environment (e.g. GNOME, KDE) that does this for them. (And if you use any obscure DE or WM you're basically fucked - your entire user session might be killed).

earlyoomd works beautifully for all use cases, now let's replace it with something catered for enterprise and let desktop users sort it out on their own.

[oiaohm]

Changes/EnableSystemdOomd

What an asinine hasty decision:

• Can we exclude certain units from being killed?
  The proposal has been updated to include implementation of a knob to exclude units from being killed by systemd-oomd. The actual interface is to be determined (i.e. it's not yet implemented), but will likely be an extension of the ManagedOOM*= systemd properties.
• How will this work if everything is in the same cgroup?
  It will not work as systemd-oomd acts on a per-cgroup level. Applications will need to spawn processes into separate cgroups (e.g. with systemd-run) or use a desktop environment (e.g. GNOME, KDE) that does this for them. (And if you use any obscure DE or WM you're basically fucked - your entire user session might be killed).

earlyoomd works beautifully for all use cases, now let's replace it with something catered for enterprise and let desktop users sort it out on their own.

https://github.com/rfjakob/earlyoom/issues/8

Birdie there are different faults that are not fixable with earlyoomd without cgroups either so the idea that it works beautifully is not right. Most of these faults result in under Linux need cgroups on processes so you in fact know what is what.

Its not a asinine decision. Linux kernel process management without cgroups is a guess me mess on what process IDs own to what process.

Those obscure DE/WM that cannot change to start processes in their own cgroups most likely have other issues on Linux as well due to using generic stuff that is not perfectly Linux compatible.

[Markopolo]

As a dev the OOM experience inside VMs has been very painful under Ubuntu 18.04.

does anyone have a good article explaining how it currently work there and and OOM events might be different on fedora?

[RahulSundaram]

As a dev the OOM experience inside VMs has been very painful under Ubuntu 18.04.

does anyone have a good article explaining how it currently work there and and OOM events might be different on fedora?

Try https://engineering.fb.com/2018/07/1...ineering/oomd/

[MadeUpName]

I sure hope it works better than systemd-resolved.service because there is nothing that can't turn into an 8 hour outage.

[polarathene]

I sure hope it works better than systemd-resolved.service because there is nothing that can't turn into an 8 hour outage.

Can you expand on that? Your DNS wasn't working for 8 hours?

[MadeUpName]

Can you expand on that? Your DNS wasn't working for 8 hours?

Most companies have internal DNS servers to cache external name requests and to hold their internal DNS entries. One of the key reasons for that is security, you don't want to advertise the make up of your internal network to the world. Well this POS the second it fails to reach you internal DNS servers disables them from it configuration and broadcasts all of you DNS lookup to external DNS servers like cloudflare, google etc. It doesn't mention it is doing it and it sure doesn't ask it just does it. So that leaves you with a system that mysteriously looks up external addresses but not internal.

Well lets do a dig +trace to see what is going on. Oh that doesn't help because it just traces back to your loop back address with out giving any useful information. It high jacks port 53 disabling what ever you had running there. Even on your DNS server it will hijack the port that bind is listening to. But again doesn't tell you. If your DHCP server points to your DNS server so you only have to update names in one place your systems will lose their network connections meaning you may have to go around restarting your machines manually.

There are no administration tools for it. They should be built into NetworkManager for the way this is supposed to work but it doesn't. Well lets do a search on the internet for some info to see if we can figure out WTF is going on. If you firewall blocks 53 out going from your machine then you have no way to get info from the internet.

If you restart a network interface it disables any changes you made and puts it's crap back in again. If you completely disable it and patch it takes out you changes and put it's self back in place again. If you find the config files on disk and modify them it keeps using it's cache until it times out even if you restart the service with a new config.

The behaviour of this thing is insane. This is a case of some kid fresh out of school with absolutely zero experience in enterprise system administration cooking up a solution designed solely for his laptop at home with absolutely no regard for the effects of others. I get what they are trying to do with it and there is a use case for it if they fix it's many short comings. But this your taking it no matter what and we don't care what you have currently or why you built it that way approach was pure BS and amateur hour.

[polarathene]

While I understand your frustrations MadeUpName, given the context it seems to be probably some missing knowledge and quite possibly misconfiguration.

I can relate to how stupid things seem to be in these situations as I had to go through with mDNS (which Android doesn't even properly support along with other platform specific gotchas and config woes). At the same time, once you know what you're meant to, it can make more sense and be easier.... such is the joys of underappreciated sysadmin tasks.

Most companies have internal DNS servers to cache external name requests and to hold their internal DNS entries.

I've set this up locally, as in caching and internal DNS entries, but obviously not for a larger enterprise network, so I might misunderstand something with my response here.

One of the key reasons for that is security, you don't want to advertise the make up of your internal network to the world.

The only info that should be getting out is the DNS queries? IPs would be cached locally otherwise the FQDN is sent as a DNS query no? Internally your TLDs shouldn't be public ones (eg you have `.test`, `.internal`, `.arpa`. Not `.com`, `.dev`, `.net`, and never `.local`, `.localhost`, etc).

Ok, but the query still goes out, and perhaps whatever your hostname is (eg `example.internal`, `service.example.internal`), you still find as sensitive information? (none of the path afterwards in the URI is revealed)

Those should all result in`NXDOMAIN` when reaching a public resolver. If your company is serious about security, then DNS-over-HTTPS (DoH) or DNS-over-TLS (DoT) would also help keep these queries private, additionally consider looking into oDoH (Oblivious DoH) if you want to go a step further.

Well this POS the second it fails to reach you internal DNS servers disables them from it configuration and broadcasts all of you DNS lookup to external DNS servers like cloudflare, google etc. It doesn't mention it is doing it and it sure doesn't ask it just does it. So that leaves you with a system that mysteriously looks up external addresses but not internal.

Sounds like it's doing what it's configured to do? Isn't this something a SysAdmin is expected to know in an enterprise environment? Defaults aren't always appropriate there.

Your local DNS should be an authoritative DNS server for your internal domains, and a recursive resolver server for any public internet you want to reach by forwarding queries. Ensure that the issue isn't related to whatever this is (wasn't clear if you said systemd-resolved was handling your entire network DNS or if that was client side only per user). There's some good open-source DNS servers out there, I use CoreDNS but it might not be appropriate for your enterprise needs. Just don't fallback to public queries for internal queries if there's no response.

IIRC, systemd-resolved will use `nss-resolve` which has configuration at `/etc/nsswitch.conf`, this defines how a DNS query should be processed, eg local system checks first, mDNS, and whatever else (I haven't touched it in ages myself). If you absolutely do not want queries that fail to respond being forwarded further into that chain, you need to explicitly be clear about that, or remove the entries you don't want processing DNS queries.

Here's an example from my notes `hosts: mymachines resolve [!UNAVAIL=return] myhostname files dns`, I'm not sure if it's that return condition there's another `[NOTFOUND=return]` which might be valid, docs should explain them, SysAdmin should know this when configuring DNS. I'm pretty sure the `resolve` is `systemd-resolved` handling it, and the `dns` at the end is to forward to your connections DNS gateway (eg whatever NetworkManager has setup iirc), thus actually unrelated to systemd-resolved really isn't it?

Note that Ubuntu and Fedora have different systemd-resolved configs by default. Ubuntu doesn't use `nss-resolve` like I described above, but `nss-dns` which has glibc read `/etc/resolv.conf` (traditional way of managing DNS IIRC?), upstream advice is what Fedora follows which is to use `nss-resolve`.

systemd-resolved can also setup split DNS, it's useful for a personal system that connects to a work VPN where you only want the internal names being forwarded to the work VPN (and it's DNS), while your other DNS queries to the public internet can be handled as you see fit going to a different DNS service, preventing the company knowing that you're browsing facebook or whatever (and if they were malicious or compromised, preventing you from going to facebook, gmail or any other website where you're sent to a fake server and served a phishing copy for your credentials).

Well lets do a dig +trace to see what is going on. Oh that doesn't help because it just traces back to your loop back address with out giving any useful information. It high jacks port 53 disabling what ever you had running there. Even on your DNS server it will hijack the port that bind is listening to. But again doesn't tell you.

I'm pretty sure it's documented that systemd-resolved requires port 53, how else is it going to be responsible for your systems DNS handling? Some apps expect DNS to be queried on port 53 and this isn't configurable iirc, I definitely recall it being an issue for my Android device when specifying a local network DNS server to connect to, I used CoreDNS for this, but I can't say if systemd-resolved behaves differently as I didn't try it for such.

Port 53 is on `127.0.0.1:53`, and systemd-resolved has a stub resolver listening on`127.0.0.53` for anything that still tries to read `/etc/resolv.conf` directly (which isn't valid for config when using systemd-resolved)? For serving to external services, you'd be binding port 53 to the local network IP or even `0.0.0.0:53` if you want to listen on all network interfaces and route DNS. Sounds like the loopback address was a local trace and as expected was being handled by the system for local DNS? (doesn't necessarily mean it was systemd-resolved at fault IIRC, I had similar problems with mDNS setup I think)

Since you know you're using systemd-resolved, did you try any of the `resolvectl` commands? There's a `status` and `query` command. Should be some way to add some logging for additional inspection if systemd-resolved is processing it and failing for some reason, along with any additional DNS service you may have on the network (I used CoreDNS logging to debug my mDNS issues).

My notes also mention `resolvectl dns` which should return a flattened list of DNS servers from `/run/systemd/resolve/resolv.conf`. When using systemd-resolved, `/etc/resolv.conf` is managed by it, instead of NetworkManager. Fedora has `/etc/nsswitch.conf` managed by `authselect`, which I think means you can't manually modify this config file reliably and need to do so via `authselect`.

If your DHCP server points to your DNS server so you only have to update names in one place your systems will lose their network connections meaning you may have to go around restarting your machines manually.

I've not done this myself, but I'm not sure why that would happen? Network connections should work fine, why do you need to restart machines? Is this about a caching issue? If so have the DNS server manage your cache and remove or reduce how long your allow a local system to cache it's DNS queries. Doesn't make much sense to me why you'd lose network connections though those should still be valid, just potential failure or public queries if the internal network DNS server went down. Your router may also be falling back to it's own DNS here if it's about the public queries, each local system is doing it's thing it's configured to do as fallback, use the connections DNS gateway as fallback (or the router internally when configured to use a custom DNS service).

There are no administration tools for it. They should be built into NetworkManager for the way this is supposed to work but it doesn't. Well lets do a search on the internet for some info to see if we can figure out WTF is going on. If you firewall blocks 53 out going from your machine then you have no way to get info from the internet.

Pretty sure there are administration tools, you just weren't aware of them and how to debug this situation. Sounds like something that could happen regardless of systemd-resolved and you're blaming it for not being able to reach the public internet when you explicitly disallow that? This issue would happen regardless of whatever you substitute it with in this scenario...?

If you restart a network interface it disables any changes you made and puts it's crap back in again. If you completely disable it and patch it takes out you changes and put it's self back in place again. If you find the config files on disk and modify them it keeps using it's cache until it times out even if you restart the service with a new config.

Again... sounds like you're not familiar with how to properly configure and manage this. As I've stated above, some files are being managed (or no longer have effect) due to how systemd-resolved works. Perhaps you made some assumptions of how it should work and be configured based on other experiences?

The behaviour of this thing is insane. This is a case of some kid fresh out of school with absolutely zero experience in enterprise system administration cooking up a solution designed solely for his laptop at home with absolutely no regard for the effects of others. I get what they are trying to do with it and there is a use case for it if they fix it's many short comings. But this your taking it no matter what and we don't care what you have currently or why you built it that way approach was pure BS and amateur hour.

Pretty sure they went with what works for majority of situations and deferred anyone needing something different to read the docs? Is that not normal at your enterprise environment?

If the configs you manipulate are manipulated/generated by a process (this isn't something that only happens with systemd-resolved btw, my DNS records are auto-generated from a template config (which has nothing to do with systemd), you modify that and not the config that the process uses), you probably shouldn't expect messing with them to work well (they're probably loaded into memory and upon restart re-generate config, NetworkManager does something similar).

[MadeUpName]

Well to start it is easy to say the things you do a year after it was released. In our case we patched and started losing machines with no idea why. Sure given enough time you can figure out what happened and why and then a way to work around as I have done. But lets silently wipe out their systems and let them guess what we have done is never appropriate. As is disabling a DNS server if their is one running. Are you out of your mind? Under what circumstances is it appropriate to replace a DNS server with a DNS request forwarder? Get serious.

As for you snide remarks about my skills fuck you. I have been a dedicated Linux administration at the highest levels for decades. As for the existing administration tools they even said when they first released it that they were coming later. The behaviour of RPM is supposed to be to not touch your configs if you have modified them It is supposed to put in a new copy with the extension I believe .NEW. but leave yours alone.

Ok, but the query still goes out, and perhaps whatever your hostname is (eg `example.internal`, `service.example.internal`), you still find as sensitive information? (none of the path afterwards in the URI is revealed)

Ya try telling that t the security oversite officer when you are working for a secure organization like a government entity or bank. It isn't yours or any one elses position to decide what is and isn't sensitive information. It is the decision of the people who own the network. I am sorry you had problems with mDNS on your cell phone but a multi hundred node network isn't a cell phone.