How Cloudflare Updates The BIOS & Firmware Across Thousands Of Servers

[phoronix]

Phoronix: How Cloudflare Updates The BIOS & Firmware Across Thousands Of Servers

For those wondering how Cloudflare keeps their thousands of servers around the world up-to-date for the latest BIOS and firmware, Cloudflare's engineering blog has put out an interesting post that outlines their process of handling system BIOS updates as well as various other firmware updates...

https://www.phoronix.com/news/Cloudflare-BIOS-Firmware-Update

[luno]

maybe LVFS/fwupd was not mature when they started doing BIOS update, nowdays LVFS/fwupd is one of the best part of Linux stack, though Desktop PC motherboard companies doesn't support it

[Velocity]

Interestingly, the Raspberry Pi 4 can be updated using PXE netboot code. So i can automate it by having it reboot and go through a firmware update cycle, then it reboots back to normal operation.

[anarki2]

maybe LVFS/fwupd was not mature when they started doing BIOS update, nowdays LVFS/fwupd is one of the best part of Linux stack, though Desktop PC motherboard companies doesn't support it

Oh yes, it's great when fwupd decides to update the system BIOS on its own, without asking me.

It's particularly amusing since on Linux this clears the Clevis LUKS bind info, making the workstations not boot until an IT person enters the key manually, then does a rebind.

The end result? fwupd being not only disabled, but even masked on all workstations.

(btw yeah, many PC companies do support it, all Dell XPS laptops are supported for instance)

[Anux]

Oh yes, it's great when fwupd decides to update the system BIOS on its own, without asking me.

What? it has a BIOS auto update function? Did Microsoft help to develop this? Who would ever do BIOS updates without checking all surrounding variables? What if a user decides to not wait and pull the plug?

Is this just a bad decision of the distro you're using or standard fwupd behavior?

[Ironmask]

Cloudflare seems to have a culture of making their own solutions. They've already said in the past that forking or patching software is too cumbersome and they'd rather just start from scratch. I'm wondering if this is purely a bureaucratic decision.

[hughsie]

Oh yes, it's great when fwupd decides to update the system BIOS on its own, without asking me.

This is not true. I have had to fight both OEMs and other security teams in making all updates require manual approval and action before being deployed. The daemon is literally incapable of downloading metadata and firmware, and all actions are initiated by the user-client. Please don't spread FUD.

[cloudflare_chris]

Hi all,

As the author of the article, I wanted to provide a little more detail. Firstly, as a Linux user of over 20 years and minor open source contributor in the past, I think it's great that LVFS/fwupd exist.

As always, there is more than one way to implement something - that's why we have both emacs and vi, and a good choice of Linux distributions.

Just because something exists - such as fwupd - it doesn't mean that it is the easiest way to implement something in a particular environment. As I described in the blog post, we already used iPXE, an excellent open source project which I am a big fan of. And we already regularly rebooted our servers. And since we already used iPXE, it was literally 13 lines of iPXE script to use iPXE to automate BIOS updates using UEFI utilities.

If we used fwupd, we would need a maintenance coordinator. Some way to coordinate taking machines out of production, applying the update, and then putting them back into production. Could we build this? Absolutely - we already have a reboot coordinator. But it surely would have been more complicated than 13 lines of iPXE script. This is why I said in the blog post: "We only deploy new firmware when our systems are out of production, so we need a method to coordinate deployment only on out of production systems. The simplest way to do this is when they are rebooting, because by definition they are out of production then."

Sometimes the best way to solve a problem is by finding a simpler alternative. We chose the simpler alternative. I wish the fwupd authors all the best.

[hughsie]

I wish the fwupd authors all the best.

Thanks! Would it be interesting to use a statically linked fwupdtool install-blob nvme.bin for non UEFI firmware in the iPXE setup? Or alternatively, use the firmware packages or metadata locally (decompressed) from the LVFS as an alternate for downloading from the vendor homepages? Having Cloudflare saying to Samsung "I wish you'd upload critical NVMe updates to the LVFS as it makes our life much easier" is orders of magnitude more pressure than 100 end users. Anyway, if any of that sounds good I'm richard_at_hughsie_dot_com and I'd love your any feedback. Thanks.

[bavay]

Having Cloudflare saying to Samsung ...

Wow, early in the morning, that is so painful. My Samsung SSDs are still a very sore point for me (as for everybody else running Linux). I spent a good hour Yesterday trying to update the firmware on my 860Evo connected to my Arm home server (and I will have to spend more time to try again, the update does not go well through a usb/sata adapter). I still don't understand why they don't want to provide their firmware updates through LVFS (knowing that some OEMs provide such updates for their integrated Samsung SSDs).