DRM Panic Handling Is Back To Being Talked About

[DeepDayze]

ideally, the system would just recover from any errors and (or, at least) write something to the system log about what happened. However, whenever that is not possible, it is nice to be able to get some information about what happened before rebooting.

Another thing is to take a memory and stack dump at the moment of the crash (whether an application, X or the kernel crashes) that can be analyzed by gdb or other debugging tools. Error recovery is a good thing to have in the kernel or within X, so for example X could hiccup when the ddx encounters an error for example, and a means to recover from the error should be implemented.

[DrYak]

means broken efivars which is really inlikely

let me introduce you to this wonderful brand called Supermicro (popular on self-buoltbservers from of-the-shelf parts) whose EFI firmware is so buggy that on rabdom write operation efivars will commit suicide, and the boot table gets so much corrupted, the the firmware isn't even able to boot its Setup payload, and the mother board is essentially bricked (efivars aren't cleared even on cmos reset or emergency reflash).

been already bitten by such a problem. (at least the board was still under warranty and got RMAd)

using the nv storage on suc broken UEFI firmwares is just calling for massive troubles

[starshipeleven]

let me introduce you to this wonderful brand called Supermicro (popular on self-buoltbservers from of-the-shelf parts) whose EFI firmware is so buggy that on rabdom write operation efivars will commit suicide, and the boot table gets so much corrupted, the the firmware isn't even able to boot its Setup payload, and the mother board is essentially bricked (efivars aren't cleared even on cmos reset or emergency reflash).

been already bitten by such a problem. (at least the board was still under warranty and got RMAd)

using the nv storage on suc broken UEFI firmwares is just calling for massive troubles

Yeah, broken UEFI that fubars the board isn't unheard of here, and when it happens outside of warranty you're fucked.

To fix these situations I usually keep a flashrom-supported SPI flasher around so I can manually reflash the damn chips on my own. Lately (as of flashrom 9.8 or 9.9) they added support for one of the dirt cheap chinese one (the ones using the CH341A chip), and it works fine.

[DrYak]

To fix these situations I usually keep a flashrom-supported SPI flasher around so I can manually reflash the damn chips on my own.

Practical question: What is your standard or most successful procedure with an EFI-bricked motherboard ?
(Though in my case, the emergency reflasher did work. Only none of the official SuperMicro-supported EFI firmware was able to function with a corrupted Non-volatile EFI content)

Though we were lucky to be able to RMA the board under warranty, I was thinking that maybe, as a last resort, one could try a coreboot openBIOS, with linux (or something else) directly backed in, to be able to boot the brick even with a corrupted NV.

[starshipeleven]

Practical question: What is your standard or most successful procedure with an EFI-bricked motherboard ?

-locate the NOR flash chip (on server mobos and most workstation mobos and all Asrock mobos is socketed and usually has a sticker with FW version on it)
-remove the chip (if it is socketed it is easy, if it is not socketed I need to bring a hot-air desoldering station/device to bear, still easy to do if you have it. Please note that it's a professional soldering tool with temperature control, not a common hot-air gun)
-mount the chip on the chip flashing device, making sure that electrical connections are correct (most SPI flash chips follow an informal standard on pinout) datasheets are usually easy to find for these chips.
-get a firmware image from the site of the manufacturer, it's usually the same file you flash when updating the firmware, in SOME rare cases they did some bullfuckery like encrypting it, but it's a long while since I've seen this.
-use flashrom (software that controls the chip flashing device, it's opensource and multiplatform) to flash manually the chip.
-mount the chip again (soldering may be required, still easy if you have hot-air desoldering device)

For most boards with a socketed chip you can also find some german small companies that sell you a pre-flashed chip for your board for like 10 euros or so, probably there is someone covering the US market too.

This is also much cheaper than buying another processor in the situations where you get a board and it's firmware is too old to recognize the CPU you want to use with it.

[karolherbst]

let me introduce you to this wonderful brand called Supermicro (popular on self-buoltbservers from of-the-shelf parts) whose EFI firmware is so buggy that on rabdom write operation efivars will commit suicide, and the boot table gets so much corrupted, the the firmware isn't even able to boot its Setup payload, and the mother board is essentially bricked (efivars aren't cleared even on cmos reset or emergency reflash).

been already bitten by such a problem. (at least the board was still under warranty and got RMAd)

using the nv storage on suc broken UEFI firmwares is just calling for massive troubles

well, every efi implementation is kind of broken, some more and some less But efivars is used by windows to store the product key I think.... so this should really work.... (at least)

[DrYak]

But efivars is used by windows to store the product key I think.... so this should really work....

Even more trivially, a region of the efivars hold the boot table (a list of payloads and where to find them. It contains both pointers to .efi files on the EFI System partition like Grub or Stub or the Windows bootloader or Memtest or the extended EFI Shell, and pointers to payloads embed into the firmware like Setup, minimal EFI Shell, full blown FlashROM manager, etc.

Another part of the efivars contains indications like "on the next reboot, boot a special entry in the boot table instead of the usual one" (and carry parameters for that payload).

This part is extremely used by the desktop versions of Windows.

(specially the fast booting laptops that boot straigh into windows without any "BIOS" screen or "press DEL for Setup" prompt. To reach setup on these, you need to first boot into Windows, then use a control pannel thingy to ask the laptop to boot windows's boot loader into a special menu mode instead of loading Windows itself. Then reboot. Then in the menu of the boot loader pick-up the option to run Setup. That in turn switch EFI to boot the Setup for the next reboot. One final reboot later and you're in Setup. This whole circus requires a couple of writes to efivars).

(similarily, windows update might use efivars to give special instuctions to the boot loader for the next reboot. That means that at least for every major releases of Windows 10 builds, efivars are going to get written)


Luckily, as I said before, these SuperMicro motherboards are specially popular to build your own server (not as much in desktops).
i.e.: they are more frequently found on machines that will run CentOS or Ubuntu.
and Linux is a lot less likely to constantly write shit on efivars.
If any information needs to be passed between reboots, usually either the /boot directory (or in rarer circumstances: the RAM) are used to carry information.
So these machine put a lot less stress on such buggy EFI firmwares.
- that's good, they are less likely to brick the motherboard
- but on the other hand, it means QA are even less likely to notice the bug. (a buggy *desktop* or *laptop* with such a bug will get noticed very easily given how often the windows update is at risk of triggering it).

In my case, I was simply editing the boot table using the official editor in the EFI firmware.
And that managed to b0rk the table to the point that the firmware wasn't even able to boot it's own Setup payload (because it got stuck much early in the process at the step where it tries to make sense out of the corrupted boot table).