The Arm SoC/Platform Changes Finally Sent In For Linux 5.3: Jetson Nano, New SoCs

[starshipeleven]

Well, uboot can dynamically alter the dtb, as uboot is in a position it knows a few things (same SoC, different model).

It could, but for most hardware I've seen (for example all boards supported by Armbian, and all hardware supported by OpenWrt) this is far from a requirement, and it can just boot with a board-specific dtb that is not touched by the bootloader.

And yes, it is setting up: you prepare the image at a board specific memory position, and hand over the pointer to the kernel.

That's just loading stuff in RAM, not "setting up", but maybe I'm just nitpicking.

On ARMv8 everything needs to be decompressed by the bootloader as well, and the bootloader also needs to know the more esoteric compression algorithms. ARMv8 also doesn't allow concatted initramfs images AFAIK.

I'm not sure how a processor arch (ARMv8) can dictate what the bootloader does, or that the Linux kernel must be uncompressed.
I mean, the bootROM only executes raw code from some source (loading it to RAM first or not, usually yes nowadays). It's far too dumb to load an actual OS, and it's not supposed to. From that point on the bootloader can do whatever it wants.
an example of initramfs boot on ARMv8 http://weng-blog.com/2017/03/ARMv8-manual-boot/

Anyway: the Arm's I know probably load the first stage bootloader in cache, because you can't initialize a memory controller if you have no clue about how it is attached.

Afaik devices with a bootROM do a execute-in-place from the internal ROM as it's fast and small enough (and they don't have much choice anyway).
I've seen enough cases of L2 caches being disabled on boot while RAM most definitely was usable, and caches had to be enabled by the u-boot bootloader by writing some magic to a register.

My intel NAS

Are you saying Intel ARM? That's some pretty old and discontinued stuff.
If it's x86 then all the initialization is done by FSP blobs and redboot is just a bootloader tasked to load a kernel.

the point is: redboot initializes memory to be able to download to.

My point is, in more modern SoCs the RAM and peripheral init is handled by the bootROM which is a true ROM containing some unknown code and is inside the SoC. The bootloader (usually uboot) is not doing initialization and sits on a storage media like NOR/NAND flash or USB/SDcard/eMMC/Sata.

Final point: generically a SoC doesn't know the memory that's connected to it, so it would be unable to initialize and use it,

The only unknown is the RAM size. The controller is known. Atheros SoCs just run a full RAM address space scan when initializing RAM controller, and this allows them to auto-sense the RAM size.

There is a 99.9% chance that if you can download to memory it is by using a second stage bootloader like uboot's (pre) init that has already initialised memory. Or uboot is the third stage bootloader.

uboot can be even a tenth stage bootloader (and if we are talking of modern ARM where there is also Trusted Boot yes it gets funny https://chromium.googlesource.com/ex...ware-design.md ), the point I'm making is that most of the SoC init is handled by the bootROM, as the "TTL recovery" or "USB OTG recovery" functionality functions even if you hose or physically remove the whole flash, see here an example for the Allwinner hardware. https://linux-sunxi.org/BROM that when you trigger the recovery mode https://linux-sunxi.org/FEL makes it show up as USB on a PC, so you can load the bootloader image.

Look at the output, it has initialized RAM and it knows its size
HELLO!
BOOT0 is starting!
boot0 version : .3.0
dram size =1024
Succeed in opening nand flash.
Succeed in reading Boot1 file head.
The size of Boot1 is 0x00036000.
The file stored in 0X00000000 of block 2 is perfect.
Check is correct.
Ready to disable icache.
Succeed in loading Boot1.
Jump to Boot1.

On RPI the main CPU loads the second stage from microsd, which in turn initialises memory and then continues loading, and finally starts the application processor (the ARM part).

yes, the "bootcode.bin" which is what, 50kbytes? What's the size of the bootROM in the Allwinners again? 32 KiB. Are you seeing a pattern yet?

[Ardje]

I'm not sure what "set up the dtd" means here. It's either loading a dtb file and either appending it to the kernel (legacy dtb boot) or using a standard interface to pass it to the kernel, end of what it does.

Well, uboot can dynamically alter the dtb, as uboot is in a position it knows a few things (same SoC, different model). And yes, it is setting up: you prepare the image at a board specific memory position, and hand over the pointer to the kernel. On ARMv8 everything needs to be decompressed by the bootloader as well, so you need a lot more of setting up, and the bootloader also needs to know the more esoteric compression algorithms. ARMv8 also doesn't allow concatted initramfs images AFAIK.
Anyway: the Arm's I know probably load the first stage bootloader in cache, because you can't initialize a memory controller if you have no clue about how it is attached. My intel NAS "loads" redboot from nor flash (it's just runs it), then redboot needs to initialize the memory controller before it can do anything else. The setup it uses to run redboot from nor flash also prevents me from going full force and change redboot into accepting 1GB DIMM instead of 512GB DIMM. Because if I fuck that up, I need to use jtag to flash the rom loader.
The rom based redboot loads the ram based redboot, which gives me tftp access if I need it. It in turn loads a stripped kernel from nor flash, which can finally access the sata bus and then that linux kernel can load the real operational linux kernel from disk and boot that using kexec. I can use the rom loader to download images using X-modem, but the point is: redboot initializes memory to be able to download to.
Final point: generically a SoC doesn't know the memory that's connected to it, so it would be unable to initialize and use it, unless you have told it somehow using more flash, or if it is a vendor specific system (POP's), or if it has on board sram. And it's all SoC dependant. So there is a 99.9% chance that if you can download to memory it is by using a second stage bootloader like uboot's (pre) init that has already initialised memory. Or uboot is the third stage bootloader. On RPI the main CPU loads the second stage from microsd, which in turn initialises memory and then continues loading, and finally starts the application processor (the ARM part).

[starshipeleven]

The SoC bootloader usually does nothing more than load the first page of selectable known storage device (nand, emmc or nor flash), verifies it's signature and starts that.

Afaik on most things I've seen it's more advanced than that, and it is initializing more or less everything on its own. Marvell and NXP hardware for example. Atheros, Allwinner and Rockchip hardware.
For example on Marvell hardware starting from Kirkwood onwards (old ARMv5 single core) it's also able to talk over TTL serial and (slowly) load a bootloader over the serial using xmodem protocol (you use a tool called kwboot from a PC to send this bootloader over), and I've seen their uboot source, most init code is skipped and the uboot is just a bootloader.
On Armada and more recent SoCs the BootROM can load the payload bootloader from USB, SPI chip, Sata or eMMC and you select that with pins you pull up/down.
This is also true for old Atheros SoCs like the AR9331, and it allows you to do the same for deciding if USB is initialized as host/guest (and the Linux system can't swap this later). And also the RAM init is generic and you can freely swap RAM chips with bigger ones and they will be detected at the correct size and used, and again uboot code is skipping init of that.

uboot will set up the DTD (like ACPI, but more advanced, less bugs, but very very dynamic), load the kernel and start it.

I'm not sure what "set up the dtd" means here. It's either loading a dtb file and either appending it to the kernel (legacy dtb boot) or using a standard interface to pass it to the kernel, end of what it does.

[Ardje]

Are you also aware that in most SoCs the initialization phase is 99% dealt with by a bootROM which is integrated in the SoC itself and can already initialize the system and load a payload (usually a bootloader to boot an OS) from any available storage medium connected?

The SoC bootloader usually does nothing more than load the first page of selectable known storage device (nand, emmc or nor flash), verifies it's signature and starts that.
That portion will probably set up some things and load the nexts loader which might just be uboot.
Uboot's initialization must already know RAM to setup the RAM controllers, and to be able to load itself. Then it needs to know how to setup the storage controllers and other busses wherever the payload may be. uboot will set up the DTD (like ACPI, but more advanced, less bugs, but very very dynamic), load the kernel and start it.
Starting with ARMv8 there were some bolt statements about bootloaders afaik, so now uboot also has to decompress any initramfs, because the kernel is incapable of unzipping the initramfs nor is it capable to unzip the kernel.
So yeah, no, it really is outside the scope of any generic software developer. Every system has it's own very specific way of booting. The defacto bootloader is uboot.
Anyway, what it boils down to: the loader in a SoC is in 99% of the cases comparable of jumping to FFFF0h in real mode to start the BIOS. Incapable of doing anything because there is no memory, no stack, nothing.
What happens next used to be open, but is now 100% proprietary on any intel architecture. It's the same on ARM.
Heh, I even have a dual stage redboot bootloader on one of my devices, and I think a saw ecos somewhere, but it might have been abused as operating system even.

[tuxd3v]

If you think that sounds bad, I suspect RISC-V won't be much better. That may be open source, but the tweaks companies make to it don't have to be.

Taking the example from ARM...
I'm Afraid that it will be a lot worse then with ARM

[tuxd3v]

All SoCs that are running Linux, can't speak for others.
You are wrong, at most there is some hardware outside the SoC (like a switch or a modem in a embedded modem-router) where the bootloader is doing initialization, but it's not initializing the SoC hardware itself.

The Boot Loaders that exists in some SoCs in the form of a ROM, are very limited in functionality!
Also,
They are not standard across all Soc Vendors, sometimes they are not even Standard on all SoCs from the same vendor..
The ROM bootloader,
Is has a tinny ROM space.. some times around 20kB,
And it usually only has the ability to initialise some core components( like the ability to load, the boot loader from spi flash, or usb, or sd card, or even in FEL mode flash a new boot0 ROM( to sdcard, flash, etc )

So its not that it initialise all hardware,
That is wrong, it is a tinny peace..
Compile bootloader like uboot, and you will see that it has >500KB of size.

Bottom Line, does it initialise all hardware in the SoC?
No it doesn't,
Because it only has facilities to to basic stuff that permits him to load a payload, that will assume control from there on..

So it doesn't do 99% of the requirements,like you stated..
It does 10-20% or even less than that..the rest is done by the payload it loads( at that time will be loaded to the Global RAM )

[tuxd3v]

What do you mean they exist in part? They've existed since 2014, it's just up to the companies that make the SoCs to implement them, and the consumers that buy them to demand it.

I mean exactly that,
They are not a global standard for all boards..so they 'exist in part'..
On Servers probably all( or maybe not all, but majority ), they implement it, but for developers that buy a sbc, its a nightmare..

I own several SBCs, and I am in need to create a build system, that I am finishing, or almost, exactly because of that..
In 99% of the cases you get, something that is lacking drivers, or have outdated kernels, or even a mix of this, with kernel headers missing, or a buggy DTB..

How will you develop a kernel driver in this environment?
You need to roll out your own image,
But some boards doesn't even have mainline kernels, or if have, they lack a complete DTB, or drivers for it..

[schmidtbag]

You are wrong, at most there is some hardware outside the SoC (like a switch or a modem in a embedded modem-router) where the bootloader is doing initialization, but it's not initializing the SoC hardware itself.

I'm beginning to question if we're talking about the same thing...

Are you stupid? How does it work on x86 then? The kernel reads ACPI tables and knows what hardware is there.

Are you stupid? Because I didn't say anything that suggested that the kernel doesn't read ACPI tables. You aren't paying much attention today.
I'm saying that ARM devices don't normally have an ACPI table to begin with and anything you to do provide one (or something similar) is not supplied by the device itself (again, not talking about server hardware here). For noobs, this is a problem. For experienced users, this is tedious.

Generic ARM images have the dtb subsystem enabled, and therefore come with all dtbs of all boards that were upstreamed in a folder located in the /boot folder, as long as the bootloader provides the dtb name (or provides the full dtb on its own) it will work the same.

Yes, I've noticed that. As far as I'm aware, DTB doesn't automatically do all the work for you; you still need to configure it for each platform you put it on. DTB makes life much easier, but not as easy as a BIOS. When the hardware itself provides the means for an easy selection of boot devices or hardware addressing, you can boot up something generic and it "just works" (again, assuming you have the right drivers). The average consumer-grade ARM platform doesn't have that luxury, and that's what tuxd3v was complaining about.

You said "The purpose of the BIOS is to act like a medium between the mainboard and the kernel.", which is not as clear as you might have thought.

I didn't specify the duration or when it happens. That doesn't make me wrong.

This was not clear from your original post though.

It was made abundantly clear by tuxd3v, who is the person my original post was addressing.

Still, manufacturers like Librecomputer are doing this for example, their uboot is using the efi subsystem (which is upstream) and passing the dtb file to Linux kernel on boot, so they can boot a generic EFI image https://libre.computer/2019/02/18/ub...ato-with-uefi/

But... that's still a device-specific disk image. It's a great step in the right direction, but the topic at hand is trying to have a true generic cross-compatible image; that link does not fit such a description.

No we don't need ACPI and full UEFI unless we need to boot Windows.

I agree; ACPI and a full UEFI by themselves aren't necessary. My point is we want the device to provide such things, so people can more easily use a generic installer or disk image (for any OS) with relative ease. With a BIOS, you don't need to implement ACPI, DTB, UEFI, or whatever else yourself. Do you not understand this?

It's an ACPI equivalent, duh. Of course each board has its own dtb. ... To create your board's dtb you only need to enable or disable what was actually connected or not in the end device.

Then you've completely missed the point of this whole discussion. I'm not denying anything you're saying there, but it's not fixing the underlying problem, which I feel I have stated way too many times by now.

Yes they do. If you replace the stock uboot with a more modern one that can boot efi and embed in it the dtb file (or use a device where the board has a dtb that was upstreamed) you will be able to use a generic EFI image.

You're still not getting it...
You can't take the SD card from something like a Raspberry Pi and put it in a Cubieboard and expect it to boot up with all the hardware working. You have to make changes. It's possible to have the OS carry over just fine (assuming you don't have something like blacklisted drivers or a very specific xorg.conf) but in most cases, you only have a single SD card to boot from, which includes your OS. That makes portability cumbersome. But hypothetically, let's say your Linux install is on a USB hard drive, where you're using the SD card just for booting, DTB, and the EFI. That makes a very dramatic improvement in portability, since you're basically substituting the SD card for what could have been an on-board BIOS chip. But that brings us full-circle to the original problem:
Why can't the manufacturers just provide that functionality out-of-the-box, directly on the hardware, like you find on x86 BIOS chips? We know it's possible because ARM servers do it.

The outlier is x86 with its standardized ecosystem, mostly because of Windows.

Well if you account for all devices and architectures then yeah, x86 is very much an outlier. But in terms of most PCs and rack-mounted servers, x86 is not an outlier. As far as I'm aware, POWER (or at least certain variants of it) and SPARC have at least some firmware. And like I said, ARM servers do it too. I'm not sure about anything else, since it isn't really something I extensively look into.

[starshipeleven]

Most SoCs or most new SoCs?

All SoCs that are running Linux, can't speak for others.

Because there's plenty out there that don't do this.

You are wrong, at most there is some hardware outside the SoC (like a switch or a modem in a embedded modem-router) where the bootloader is doing initialization, but it's not initializing the SoC hardware itself.

Booting an ARM device with a generic image, though tedious, is easy. In some cases, you can get by using a single dd command and you're good to go. Getting all of your hardware to work properly on a generic image isn't so easy.

Are you stupid? How does it work on x86 then? The kernel reads ACPI tables and knows what hardware is there.

Generic ARM images have the dtb subsystem enabled, and therefore come with all dtbs of all boards that were upstreamed in a folder located in the /boot folder, as long as the bootloader provides the dtb name (or provides the full dtb on its own) it will work the same.

I never said it did any more than that...

You said "The purpose of the BIOS is to act like a medium between the mainboard and the kernel.", which is not as clear as you might have thought.

Again... you're missing the point. The problem is expecting the user to do this themselves.

This was not clear from your original post though.

Still, manufacturers like Librecomputer are doing this for example, their uboot is using the efi subsystem (which is upstream) and passing the dtb file to Linux kernel on boot, so they can boot a generic EFI image https://libre.computer/2019/02/18/ub...ato-with-uefi/

This will NOT work with Windows because it's using dtb and not ACPI, even if it's EFI boot.

Anyway, to my understanding, only ARMv8 servers seem to have "built-in" ACPI support.

Servers have full UEFI firmware in a "BIOS chip", but UEFI is a huge PITA to work with so it's 100% impossible that anyone outside of high-end hardware will ever want to deal with it unless forced.
see the various UEFI implementations for raspi https://github.com/andreiw/RaspberryPiPkg

That's what we want to see in stuff like phones and developer boards in order to make it easier for home users any hobbyists to get into ARM.

No we don't need ACPI and full UEFI unless we need to boot Windows.

Don't DTBs still need to be manually configured per-device?

It's an ACPI equivalent, duh. Of course each board has its own dtb. They are usually drived by a "master dtb" published by the SoC manufacturer as part of the basic Linux support for that SoC. To create your board's dtb you only need to enable or disable what was actually connected or not in the end device. It's a piece of cake if compared to Coreboot.

For decent boards they are usually merged upstream, but the uboot bootloader can also store the board's dtb in itself and then pass it to a booting Linux kernel, similar to what a BIOS does with ACPI tables.

Remember, the problem at hand is there's no such thing as a fully generic and fully functional ARMv8 image that will work cross-platform with any device you put it in. Stuff like DTB or custom UEFIs don't fix that.

Yes they do. If you replace the stock uboot with a more modern one that can boot efi and embed in it the dtb file (or use a device where the board has a dtb that was upstreamed) you will be able to use a generic EFI image.
Also on ARMv7 because uboot can do that also there.

Everything on the Linux side of things is fine. The board manufacturers (or ARM themselves) are the ones who need to change something.

Yes it's correct, but since ARM has a very neat split between the board initialization and the bootloader, and the former is usually done by a bootROM which you can't touch anyway, you can usually replace the bootloader, it's not as arcane and hard as with Coreboot.

I don't think ARM ever predicted their CPUs were going to be used in this way. They were originally for embedded devices, really crappy phones, and routers.

The outlier is x86 with its standardized ecosystem, mostly because of Windows.

Everything else is a Wild West of random stuff where none cares about running a generic OS image.

[schmidtbag]

But with armv7, or armv8, they should have addressed a lot of standard problems, that are still a plague today..

I agree, but keep in mind the largest markets for arm: phones, tablets, network devices, embedded devices, and servers. ARM servers already have everything you're asking for. As for the rest of the stuff, those come with an OS pre-installed that the user is not expected to ever replace. So, even though development boards and hobbyists are a decent sized market, we're kinda an outlier. I feel your pain, but ARM doesn't really have much of an obligation to support us.

Everybody tough that because ARM will license cores,
It will be a common standard there, something "wonderful",
The problem is that there are now, a zillion standards there( one per CPU vendor, and one per board..sometimes more, depending on the board revision..),
Or resuming it, no standard at all..

If you think that sounds bad, I suspect RISC-V won't be much better. That may be open source, but the tweaks companies make to it don't have to be.