systemd Talks Up Automatic Boot Assessment In Light Of The Crowdstrike-Microsoft Outage

[phoronix]

Phoronix: systemd Talks Up Automatic Boot Assessment In Light Of The Crowdstrike-Microsoft Outage

In light of the CrowdStrike-Microsoft outage/disaster that has been wreaking havoc on corporate Windows systems around the world since Friday, systemd lead developer Lennart Poettering pointed out how such a situation on Linux systems could be averted by leveraging systemd's Automatic Boot Assessment functionality...

https://www.phoronix.com/news/systemd-Auto-Boot-Assessment

[Eonfge]

While I generally agree with Poettering, the bullshit that CrowdStrike pulls of on Linux only makes the situation worse.

In the last year, CrowdStrike crashed both RHEL 9 and Debian 12 with shitty updates. And worse... They have their own proprietary drivers for Linux that requires users to recompile their own kernel. So even with Auto Boot Assessment, they'll just force their own broken product to corrupt the previously stable kernel.

[Volta]

Microsoft, windows and enterprise. What could go wrong? Fail of the decade. Nice one m$! You've just ruined microsoft trolls.

[spicfoo]

While I generally agree with Poettering, the bullshit that CrowdStrike pulls of on Linux only makes the situation worse.

In the last year, CrowdStrike crashed both RHEL 9 and Debian 12 with shitty updates. And worse... They have their own proprietary drivers for Linux that requires users to recompile their own kernel. So even with Auto Boot Assessment, they'll just force their own broken product to corrupt the previously stable kernel.

CrowdStrike can corrupt the current kernel but if distributions have been setup to detect that and fallback to the previous kernel automatically, the recovery path is much easier than the shitty situation where you will have to jump into rescue mode and manually clean up a file on Windows. This shouldn't be happening.

[mdedetrich]

Microsoft, windows and enterprise. What could go wrong? Fail of the decade. Nice one m$! You've just ruined microsoft trolls.

You clearly didn't read and/or understand the issue.

What happened with Windows can easily happen in Linux, and the Linux customers that use Crowdstrike for Linux have gotten broken boot loops before. Both Windows and Linux have solutions for this, but the solution is pointless if people (in this case it admins/integrations) don't actually use it.

Windows has a mechanism that also prevents locked systems from booting, its just that the implementation of Crowdstrike deliberately avoids it (which wouldn't be any different with Crowdstrikes Linux's integration avoiding systemd's Automatic Boot Assessment). This is exactly what Pottering is saying, Linux/systemd has functionality to prevent such issues but companies/users are avoiding it!

If you want to know the more intricate details, watch https://www.youtube.com/watch?v=wAzEJxOo1ts . This is an actual person that worked on the Windows kernel/OS so he has an actual understanding of how it works.

[nvaert1986]

I think this would actually be a good idea for most end user and enterprise setups to automatically repair systems that refuse to boot, as long as a person can turn it off, I don't see any issues. This could save IT admins that manage Linux workstations a lot of headache in certain set-ups.

[oiaohm]

CrowdStrike can corrupt the current kernel but if distributions have been setup to detect that and fallback to the previous kernel automatically, the recovery path is much easier than the shitty situation where you will have to jump into rescue mode and manually clean up a file on Windows. This shouldn't be happening.

This is not going to work in the CrowdStrike class problems all the time.

https://www.youtube.com/watch?v=wAzEJxOo1ts

Yes this link from mdedetrich covers the CrowdStrike fairly well.

But lets go into the case with a little more look at this class of problem.

CrowdStrike has it own byte code like Linux kenrel BPF/eBPF but with what you call garbage validation. The update file you are told to delete to fix the Cloudstrike problem contains the bytecode and is not in fact a Microsoft signed driver. Yes crowdstrike came up with a plan to bypass Microsoft Windows driver validation. Also their Linux driver is not going to be submitting their code upstream to kernel.org for full validation against stupidity.

Yes you boot to a prior kernel and that prior kernel has cloudstrike driver loaded that cloudstrike driver can find the broken update that has been added to system and still totally fail. This is why a straight kernel rollback does not work.

Now this class of problem can get worse than the Crowdstrike one. Lets think that this is a driver that is connecting to cloud services and the cloud services is no longer there. So you could have had a driver in your system for years appearing to behave itself and now it totally not working.

This is argument of why closed source binary drivers in kernel space should not exist because you need your kernel drivers audited inside a inch of their life to be sure they are not up to no good.

Automatic boot assessment is a good idea but this does need a fail safe back to a distribution known good kernel with all third party added questionable drivers removed yes a safe mode/rescue mode.

There was a historic anti-virus that no body uses any more that did in fact have the kernel mode driver connecting to network to get updates and when that server was taken over by another group and send back garbage data the driver jumped off cliff this is quite a few years back. So this is not the first time something sold as protection against viruses/malware turn out to be the malware themselves due to being poorly coded.
.

[oleid]

Automatic boot assessment is a good idea but this does need a fail safe back to a distribution known good kernel with all third party added questionable drivers removed yes a safe mode/rescue mode.

Wouldn't it then make sense to use previous known-good filesystem snapshots instead of only booting an older kernel?

[mdedetrich]

Wouldn't it then make sense to use previous known-good filesystem snapshots instead of only booting an older kernel?

That's up to crowd strike to implement which they clearly did not. They didn't enough do basic sanity checking, as the broken payload that caused the boot loop was just filled with 0's.

The point of automatic boot assessment is to not put any trust in outside sources to implement things correctly.

[mb_q]

Boot roll-back is no go for corporations, because it opens a way to utilize a known bug in older version: by crashing a newer one (which is generally easier, and easy with a physical access) and triggering an update roll-back. They would rather deal with bricked machines than allow users to take control this way.
The problem with CrowdStrike is their broken architecture -- this is literally an auto-updating closed-sourced rootkit, it is quite obvious that it will bring more problems than solutions. Computers are not magick, one can gather the same intel about the system from monitoring boundary than from monitoring the inside.