Debian To Consider Changing How It Treats Closed-Source Firmware

[Waethorn]

That is the case right now. Vendors provide closed source drivers and closed source firmware. So now you just added a way to make the situation worse.

Only NVIDIA does this. ARM chipset vendors don't do anything except provide closed-source drivers for Android because Android doesn't fall under the GPL. And it took years before Intel and AMD would commit to open-source drivers at all. UEFI firmware/drivers can be open or closed source. There's the whole Tianocore/EDK2 project behind system-level firmware. But it's on the hardware, not in the software, so the responsibility is now firmly in the realm of the hardware vendor.

No. The UEFI doesn't play a role at all there. Linux have an AHCI driver that sends AHCI commands to the PCIe bus to the right device. UEFI is out of the picture.

This happens because the UEFI firmware on the board supports many PCIe NVMe chipsets via their "drivers" (what they call UEFI option ROM support). The UEFI does all the work to initialize the NVMe controller, and it needs its own driver for supporting the vendor-specificity of the NVMe controller, otherwise it can't initialize it. The PCIe NVMe controllers supports the higher-level AHCI command-set in their own firmware. This is EXACTLY how I propose video cards should work with higher-level API commands using feature set levels.

But that's only useful for very simple, standardized devices. For a start it would add a lot of overhead to talk to UEFI for everything. But further, you get a complete inability to upgrade. OpenGL can't be implemented in UEFI for the simple reason each card uses a custom bytecode and you need a compiler for it. You don't feed OpenGL, you don't even feed SPIR-V or Vulkan. You need to implement all that in OS/userspace software, and that's the biggest part anyway.

You don't have to talk to UEFI for everything - you let the controller firmware do that. You let the OS talk to the controller firmware through a standard set of commands and feature levels by building an API driver into the firmware, not vendor-specific drivers in the OS. The UEFI firmware can handle basic on-board controller firmware ("drivers") like AHCI, network, and should be required to handle all other on-board components already. A discrete graphics card can have its own firmware, supported by the UEFI driver (again, this is what they call "option ROM support" in UEFI).

I never understood the logic behind delegating hardware access to user space because "if it crashes....(insert excuses here)".

Here's a thought: make it BETTER so that it DOESN'T! <- this thought is apparently not on developers minds for some reason. It's all about "let's bring this hardware up as shoddily as we can and worry about bugs later" and we see the result of that where hardware takes years for OS support to become stable or mature, or potentially never because OS hardware drivers are broken by updates or some new software only works with a specific version of driver. How many jobs are there where you're allowed to do something wrong the first dozen or hundred times or so, even though you know it's wrong or know that it's eventually going to fail, and then have the ability to do it right later?

And yes, there is a high-level shader language.

That doesn't describe reality at all. For a start, even if it were possible, which it isn't because software is never perfect, it would require the mythical "sufficiently disciplined programmer" and also a "sufficiently funded provider", when their priority is always to cut costs and ship fast. The cards behave a certain way. The drivers contain bugs and updates, thus, bring bug fixes. Those fixes will stop coming downstream as soon as your OEM decides not to ship further UEFI updates. Everybody loses.

How many times does a UEFI firmware get updated? How many times do drivers? Stop the insanity and support hardware by feature levels and make the hardware vendors responsible for supporting it. They will, if they get to do things the way they want. This is why ARM hardware vendors won't touch Linux with a 10-ft pole, yet there are how many Android handsets on the planet right now? And how often do Android drivers get updated? And how often does the firmware on Android handsets get updated? How many times does an Android OS in a handset even get updated, despite software continuously rolling out for it?

What happens when you freeze your ZFS driver? It goes out of sync with Linux, it doesn't get fixes, it doesn't get optimizations. Feature levels are irrelevant here. Sometimes features get standardized _after_ some card ships them, sometimes new features are technically implementable in older hardware. It's not about the theoretical interface, it's about the reality of hardware and software distribution.
When you write code _right now_ you can, in theory, forget about the particulars of drivers and cards too, but in reality you can't. And that's what happens in UEFI as well, as it is simply software stored in an EEPROM.

ZFS is the just the command set. This is the part that goes into the OS, just like OpenGL and Vulkan. It doesn't need to know about what drive controller you have, or how to communicate with it on a hardware level, so this statement is moot. If you're upgrading your ZFS storage server just because there's a new feature flag addition, you'd be promptly fired from your enterprise job.

[sinepgib]

Only NVIDIA does this. ARM chipset vendors don't do anything except provide closed-source drivers for Android because Android doesn't fall under the GPL.

Userspace doesn't fall under the GPL, but userspace is irrelevant there. The kernel is still Linux and still under the GPL. However, nothing stops vendors from making closed source modules, such as NVIDIA and ARM. They just can't upstream kernel drivers without the userspace half.
The reason ARM vendors only provide drivers for Android is that they only support Android and it's cheaper to target one given kernel release and forget about it, which is what they actually do.

And it took years before Intel and AMD would commit to open-source drivers at all. UEFI firmware/drivers can be open or closed source. There's the whole Tianocore/EDK2 project behind system-level firmware. But it's on the hardware, not in the software, so the responsibility is now firmly in the realm of the hardware vendor.

1. It's still software.
2. How many boards ship TianoCore or EDK2?
My case is not theory, it's reality. The reality of the issue is that vendors provide closed source only for UEFI.
NVIDIA is pretty much the most relevant closed source driver. Intel's and AMD's aren't reverse engineered (though AMD's used to), so they don't have the problem you mention. Most other drivers are either for generic, well documented and standardized interface, or actually supported by the vendor. Of course, speaking of non-SoC which are a different beast where everything is ad-hoc.

Putting the responsibility on the OEM (_not_ the hardware vendor) is a terrible idea. That's my entire point. Just look at how many crappy BIOS/UEFI implementations are there and how long those are maintained and do the math.

This happens because the UEFI firmware on the board supports many PCIe NVMe chipsets via their "drivers" (what they call UEFI option ROM support). The UEFI does all the work to initialize the NVMe controller, and it needs its own driver for supporting the vendor-specificity of the NVMe controller, otherwise it can't initialize it. The PCIe NVMe controllers supports the higher-level AHCI command-set in their own firmware.

I'm not sure about NVMe, but we're talking about how the OS uses drives. UEFI of course has drivers for basic initialization and to find the EFI partition. But that's irrelevant here.

This is EXACTLY how I propose video cards should work with higher-level API commands using feature set levels.

Unless you include the compiler and all that stuff, you simply can't do that. And you'd still need to talk to multiple cards, many of which you don't even know because they don't exist yet.

You don't have to talk to UEFI for everything - you let the controller firmware do that. You let the OS talk to the controller firmware through a standard set of commands and feature levels by building an API driver into the firmware, not vendor-specific drivers in the OS.

It's still an extra layer. Of course I'm not assuming your application will talk directly to UEFI, that's so backwards I didn't even think it was what you proposed.

The UEFI firmware can handle basic on-board controller firmware ("drivers") like AHCI, network, and should be required to handle all other on-board components already. A discrete graphics card can have its own firmware, supported by the UEFI driver (again, this is what they call "option ROM support" in UEFI).

I never understood the logic behind delegating hardware access to user space because "if it crashes....(insert excuses here)".

Here's a thought: make it BETTER so that it DOESN'T! <- this thought is apparently not on developers minds for some reason. It's all about "let's bring this hardware up as shoddily as we can and worry about bugs later" and we see the result of that where hardware takes years for OS support to become stable or mature, or potentially never because OS hardware drivers are broken by updates or some new software only works with a specific version of driver. How many jobs are there where you're allowed to do something wrong the first dozen or hundred times or so, even though you know it's wrong or know that it's eventually going to fail, and then have the ability to do it right later?

I never understood the logic behind separating user space and kernel space because "if it corrupts memory... (insert excuses here)
Here's a thought, make it better so it only accesses its own memory.

Software is complex and you need failsafes because it's practically impossible to cover all corner cases in the first attempt. And there's an economic reality in play, the industry expects fast cycles and that means taking shortcuts. Unless you formally verify everything, which would take ages, you'll ship bugs, end of story.

But whether or not it's in userspace is irrelevant: you need a big chunk of code that won't fit in your ROM.

And yes, there is a high-level shader language.

Yes, so?

How many times does a UEFI firmware get updated? How many times do drivers? Stop the insanity and support hardware by feature levels and make the hardware vendors responsible for supporting it. They will, if they get to do things the way they want. This is why ARM hardware vendors won't touch Linux with a 10-ft pole, yet there are how many Android handsets on the planet right now? And how often do Android drivers get updated? And how often does the firmware on Android handsets get updated? How many times does an Android OS in a handset even get updated, despite software continuously rolling out for it?

Maybe the drivers get updated more often for a reason, you know? Maybe you can get away with not updating the UEFI firmware as often because its task is much simpler. Add the complexity of those drivers and you created a problem.
Regarding ARM hardware vendors, they won't touch _mainline_ with a 10-ft pole because they already need to support Android for phones and tablets and it's cheaper to support only one OS and not run around in circles trying to get their code in shape for upstream. Open sourcing doesn't add value for them and a static base where they can just make a code drop is simpler. That doesn't mean you get bug free drivers, it just means you'll learn to live with the quirks. And oh boy there are quirks.
Also, how long is the shelf life for an Android phone and how long for a desktop or laptop? You're comparing apples and oranges.

ZFS is the just the command set. This is the part that goes into the OS, just like OpenGL and Vulkan. It doesn't need to know about what drive controller you have, or how to communicate with it on a hardware level, so this statement is moot. If you're upgrading your ZFS storage server just because there's a new feature flag addition, you'd be promptly fired from your enterprise job.

Most of the driver for OpenGL and Vulkan is in the translation part. The thin layer you'd get by using UEFI doesn't save you a lot. And now you're unable to handle the quirks your particular vendor inserted. The ZFS command set comes with a data and metada layout just as any file system. There are abstraction layers for hardware, more news at 11. That doesn't mean it needs to (and in practice doesn't) go through UEFI. There's an AHCI driver in your OS. As there's an old IDE one that can also run ZFS.

Regarding my enterprise job, maybe there's more to computing than just enterprise, and each segment has different priorities. And maybe even firmware has bugs, if you checked you'd see some drives do. Now, convert that into a single block in UEFI that also carries drivers and enjoy your mess.

[piorunz]

So newbies don't count then? That's the kind of thinking that makes FLOSS a fringe target. We need to welcome newbies, otherwise we'll end up circle jerking about how our 10 years old box with all free software runs like a 20 years old box. We need users to count as a market segment. We need to actually have something to negotiate with.

Debian is not for newbies. Newbies should go to Linux Mint, Ubuntu, and so on. IDK what you on about. "Debian is not good for newbies, not idiot-proof, we're doomed". Why you cry like that?

I think you didn't understand my post. I said, this change will affect newbies. Maybe for the better. That's why Debian devs are discussing these things right now, because newbies don't know that non-free ISO image exists. So it's a good thing that they do that. Debian may improve for newbies when they reach consensus and change now non-free firmware is present in Debian.
But I expressed opinion from my perspective. For power users, nothing change. 100% FOSS images will be available. Debian will be still Debian. I am fine with that.

[hamishmb]

Canonical devs have controlled over half the Debian project voting power for quite a few years now. Debian will do whatever Ubuntu decides it wants it to do. This has been going on for 8-9 years, maybe longer.

I hadn't heard this before, and it might scupper my plans to distro hop to LMDE in the summer. I can't find any information on this though. Source?

[sinepgib]

Debian is not for newbies. Newbies should go to Linux Mint, Ubuntu, and so on. IDK what you on about. "Debian is not good for newbies, not idiot-proof, we're doomed". Why you cry like that?

I think you didn't understand my post. I said, this change will affect newbies. Maybe for the better. That's why Debian devs are discussing these things right now, because newbies don't know that non-free ISO image exists. So it's a good thing that they do that. Debian may improve for newbies when they reach consensus and change now non-free firmware is present in Debian.
But I expressed opinion from my perspective. For power users, nothing change. 100% FOSS images will be available. Debian will be still Debian. I am fine with that.

I misunderstood your post indeed.

[sinepgib]

I hadn't heard this before, and it might scupper my plans to distro hop to LMDE in the summer. I can't find any information on this though. Source?

I won't take the time of checking whether the particular assertion is true, but you can find the names of the people in the committee here: https://www.debian.org/intro/organization#tech-ctte

FWIW I don't think it is entirely true, considering how the systemd vs Upstart discussion went. Canonical was deeply vested in making Upstart at least a first class citizen in Debian to offload some of the work. They ended up using systemd because of that vote.

[PublicNuisance]

Personally I don't see this as a big deal. Users who don't care about FLOSS principles will see better hardware support and most users who do care probably don't use Debian to begin with. Nobody should be mad at Debian, they should be mad at the hardware manufacturers who force you to choose between using closed source firmware or to have poor hardware support.

[jabl]

4.- Publish your firmware source as free software. Problem solved.

Sure, in an ideal world. In the current reality however, the vast majority of devices have non-free firmware. I mean, look at the Debian wiki list of FOSS firmwares: https://wiki.debian.org/Firmware/Open

There's some dozens of devices there. How many different devices are there on the market today and have Linux drivers? Thousands certainly?

Anyway, my point was that all the three options I mentioned are more or less equivalent from a software freedom perspective. Your device runs free or non-free firmware regardless of the particular method the firmware is provided to the device. Claiming that some of the alternatives are Ok from a free software perspective while others are evil is just sticking your head in the sand. Particularly so when there are strong technical reasons for preferring the particular options that FSF think are evil. Heck, even from a software freedom perspective I'd prefer updateable firmware as that at least allows hackers to reverse engineer it and create a free replacement firmware.

[edgmnt]

Why can't the same be done for OpenGL and other higher-level interfaces? Let the hardware do the work instead of flakey software drivers. It should just be a device with XYZ features that the OpenGL/Vulkan/DX driver doesn't need to know what brand it is or how to communicate with it other than to pass OpenGL/Vulkan/DX commands to it.

GPUs don't execute public APIs, which are way too complex and different to handle in firmware. Drivers are typically full-fledged compilers that translate OpenGL, DirectX, video acceleration APIs and everything else into a GPU-specific command stream. Doing it on the graphics card would require a fast general purpose CPU with a huge firmware or a very complex instruction decoder. The complexity is much higher than for an HDD.

They could come up with a vendor-agnostic or at least front-facing command stream, but that is either very difficult to generalize or it locks them into a specific GPU architecture. I don't really like the ISA proliferation, but I'm not sure how that could be improved.

The only other sensible option is to expose the internal workings of the GPU to allow developers to write open source drivers.

[michaelb1]

Part of the reason I'm still back on a creaky old Athlon II X2 270 is that I can't afford a Raptor Systems machine and don't like the idea of having to run a closed-source Ring -3 hypervisor like the Intel ME or AMD PSP just to get the system to boot.

I'd like to let you know there are newer AMD boards without PSP which are also supported by the opensource coreboot BIOS firmware. I.e.: Lenovo G505S with A10-5750M, ASUS A88XM-E with A10-6800K, ASUS AM1I-A with Athlon 5370. They are from 15h / early 16h generations which didn't have AMD PSP. Although these quad-core CPUs are from 2013-2014, they are still powerful enough for the modern tasks.