A ZSTD-Compressed Linux Kernel Could Be Up Next

[sdack]

Aircraft use a mix of operating systems. ...

Don't get hung up on one single factor such as the boot up time. If you're the maker of a product and your entire sales strategy relies on having the booting kernel compressed with one particular algorithm, or else you won't be able to sell it, then you're not fit for the market you're selling to.

Imagine the guy who comes to an aircraft manufacturer and proudly tells them how he managed to get his device booting in under 1 second just so they can finally accept it. ... Sounds like a Hollywood story really. They are more likely to bend the 1-second rule in favour of a solid product than to accept a device onboard their planes that fulfils all criteria, but that was developed with a lot of risks just to get there. And I have worked for the space industry by the way. People there do look beyond their paper work and if there's something they don't like, or do like, then they will of course adjust the rules. These aren't set in stone.

[cj.wijtmans]

I use xz compressed kernels and mine are 6.3MB with all firmware compiled in. I don't have any modules, just the bits I need compiled in.

thats huge mines 2MB.

[oiaohm]

Don't get hung up on one single factor such as the boot up time. If you're the maker of a product and your entire sales strategy relies on having the booting kernel compressed with one particular algorithm, or else you won't be able to sell it, then you're not fit for the market you're selling to.

Imagine the guy who comes to an airline manufacturer and proudly tells them how he managed to get his device booting in under 1 second just so they can finally accept it. ... Sounds like a Hollywood story really. They are more likely to bend the 1-second rule in favour of a solid product than to accept a device onboard their planes that fulfils all criteria, but that was developed with a lot of risks just to get there. And I have worked for the space industry by the way. People there do look beyond their paper work and if there's something they don't like, or do like, then they will of course adjust the rules. These aren't set in stone.

When it comes down to winning contracts in competitive parts the vendor who can tick the boxes without needing the rules adjusted wins. Think about it if you have two products on the table. Both are equally solid in operation and one is meeting the 1 second rule because the time has been spent to-do that area properly along with everything else and the other is not. The one that fails the 1 second limit loses. In a lot of the devices for aircraft everyone is doing under the 1 second reset.

To be fit in market means meeting the requirements of that market. Did I say that the 4 factors around compression were the only factors device makers had to worry about. The answer I did not. If you are not getting 1 second boot times with everything else about product being quality and your competitors are getting 1 second boot time with equal quality you lose unless you are cheaper and going cheaper will not risk lives.

I like how you always get people who say hey it has to be Hollywood. There are a lot of risks flying aircraft and information screens for pilots not working when they should just adds more risk. 1 second rule in the aircraft information screens is based on studies. The studies say horribly that 0.5 seconds is ideal so 1 second is double ideal this is why 1 second is kind of brick wall.

Space industry is very interesting because you see rules bent there so many times it not funny. Its a lot simpler to make a control console for the international space station than for a standard aircraft because you have more wiggle room on the space based stuff.

Basically the list of 4 I gave are what you are going use when you are choosing what compression that your product in fact requires. To be fit for market those 4 requirements may be part of the critical requirements list. This is why I said there are 15 configurations of those 4 options. Because all 4 of those requirements could be on your requirements list or only 1 of them or even none of them.

To be including in the Linux kernel mainline it has to pass peer review and you would hope that quality of the compression implementation would not be a problem any more.

sdack are you saying to me and something that in the customer requested feature list said boot time of 1 second that you would not attempt to-do and if you failed do-to not be surprised if you lost the contract. I am not saying there would not be other requirements that the device has to perform perfectly to win contract either.

Failure to meet the market prescribed requirements is a very fast way to fail in that market. Be it boot time, be it power usage, be it quality.... Of course not all the requirements will apply to kernel development like layout of graphical screen. There are ones that do apply to kernel development because those making products using the kernel need them.

[sdack]

When it comes down to winning contracts in competitive parts the vendor who can tick the boxes without needing the rules adjusted wins. ...

No, that's not what I'm saying. Rules themselves can be wrong and they do need to be reasonable and have to evolve, too. You don't just make them up and pass them on like these were given to you by Moses himself. It may only seem this way especially when you're a small contractor among a lot of others and bigger ones. You then often get judged quickly, get sorted out against other competitors, but your work will also not be of much significance when it's this easy to rule you out. Don't let rules dominate you or you only will be insignificant.

And yes, I've seen stuff that still cracks me up today just thinking about them, but in the end did it have a good reason for decision makers to choose them. I.e. the use of an embedded PC with Windows NT in a real-time and safety-critical area and whenever it booted up did it run a virus scanner as the first thing! It felt out of place to watch it do this, but the decision for it was made based on cost and all other factors were thrown out the window just for that. A contractor had build a system with all bells'n'whistles for a very low price and it ended up beating all the others who could only meet the requirements while being very expensive. One could say that the requirements were excessive when in the end these got thrown out the window, or that the decision makers were bought and sold by its very low cost design, but who gets to judge the judges? ...

[FireBurn]

thats huge mines 2MB.

That includes firmware for skylake, tonga, ath10k, webcam and audio. Here's my config: https://github.com/FireBurn/KernelSt...dot_config_tip

[caligula]

3 seconds is already far too much for most customers. If you then want to argue over cases where its 3 vs 10 seconds, not including the remaining boot process of course, then you're just theorizing. Customers, who have to wait 15 or 22 seconds for a device to boot aren't going to complain about it. Depending on what this device does will they either not buy it or they'll just leave it on or not care for it. Such a device, with a low spec CPU, is obviously designed not to be fast, but to be cheap.

Note that the 3 seconds comes from my own experiment with RPi. It's merely the result of 15 minutes of work: tuning the kernel defconfig to use lz4 and disabling unnecessary drivers and modules. Would an embedded vendor spend more than 15 minutes on their embedded OS? Hell yes. I don't know, I'm guessing they could easily shave off 1-2 seconds by doing some normal tricks known to most embedded Linux developers. There's even talk about slimming down Linux. They can't be this clueless, of course they know how to do it better than a total noob buying RPi and compiling their first kernel..

[oiaohm]

No, that's not what I'm saying. Rules themselves can be wrong and they do need to be reasonable and have to evolve, too. You don't just make them up and pass them on like these were given to you by Moses himself.

The 1 second rule on aircraft interface stuff is in fact defined as reasonable. It also on some automotive it is also defined as reasonable.

The reality here is 1 second is not a unreasonable requestion for total boot time. 0.5s is classed as unreasonable in most cases.

Note that the 3 seconds comes from my own experiment with RPi. It's merely the result of 15 minutes of work: tuning the kernel defconfig to use lz4 and disabling unnecessary drivers and modules. Would an embedded vendor spend more than 15 minutes on their embedded OS? Hell yes. I don't know, I'm guessing they could easily shave off 1-2 seconds by doing some normal tricks known to most embedded Linux developers. There's even talk about slimming down Linux. They can't be this clueless, of course they know how to do it better than a total noob buying RPi and compiling their first kernel..

https://www.youtube.com/watch?v=KTRA1PRJWH8

This video is a walk though to getting a 1 second boot out of Linux using mainline options. I like that some chips using internal clock instead of having a clock crystal can cost your over 1 second.

https://embexus.com/2017/05/16/embedded-linux-fast-boot-techniques/

This has a graphical Linux using a beaglebone black to have a graphical interface on line in under 1 second. It starts off requiring 12 seconds.

Most of the processes to optimise down to 1 second are not using special stuff. Making sure the hardware was designed for fast booting like if you are losing a few seconds for the soc chip to calibrate clock every boot fast boot is off the table.

The documentation on fastboot experienced software implementers are able to-do everything they can for fastboot in a about 3 hours. Where up to 2000+ hours can come in when you have nightmare cases of having to scrap proto boards and start over with faster soc this can mean redesigning the boards, QA on boards...This is hour after hour disappearing out the window . If you have a 1 second limit and you need to do more than you can optimise to it means faster chip and a faster chip equals higher cost. This is where fastboot tricky and time expensive because you don't want too powerful of a chip or too weak of a chip. Different soc chip designs have different performance profile and this can require different optimisations.

What makes fastboot expensive to-do is not really software optimisation it is the hardware optimisation to get the sweet spot for the task required. This can also by why if you grab any generic bit of hardware and attempt to make it fastboot why you don't stand a chance.

caligula on RPi you run into straight up 1+ second from VC4 gpu hardware init. 2 seconds on RPi is about the same level of optimisation to get 1 second on proper fastboot hardware like the beaglebone black. Sub 1 second booting is very hardware dependant. So one of the tricks from embedded developers going for speed with the RPi would be throw it over shoulder and get something more suitable. Being able to throw board over shoulder and start over is what makes producing fast boot devices expensive.

[sdack]

The 1 second rule on aircraft interface stuff is in fact defined as reasonable. It also on some automotive it is also defined as reasonable.

Nobody doubts it. You just keep following the rules, son.

[caligula]

caligula on RPi you run into straight up 1+ second from VC4 gpu hardware init. 2 seconds on RPi is about the same level of optimisation to get 1 second on proper fastboot hardware like the beaglebone black. Sub 1 second booting is very hardware dependant. So one of the tricks from embedded developers going for speed with the RPi would be throw it over shoulder and get something more suitable. Being able to throw board over shoulder and start over is what makes producing fast boot devices expensive.

Yes, it was just an example. I don't have access to real US air force embedded computers, but I have RPi..

[DrYak]

Does anyone compress the kernel with zoplfi? Seems like you could get at least a few extra percent with maybe a few hundred iterations

Compared to Zstd, it's not interesting.

Zopfli is just plain old Deflate (like Gzip and is compatible), but is excruciatingly more thorough in its searches for perfectest match/frequency modelling.
It compresses a tiny bit better at the cost of horrendous compression time (thus it's only useful for things that you compress once and re-use multiple time - kernel would be a candidate). It's only main advantage is that it's compatible with Deflate, thus doesn't require anything more than stock Gzip support (thus, it's NOT interesting for kernel - the kernel could afford more modern decompressors, even bzip2. It's more useful for distributing static content over the web, without requiring an update of the clients: browsers)

Zstd is like "deflate++" : it replace a few core technologies with more recent one : e.g. the Huffman entropy coding is replaced with FSE (Finite State Entropy - a type of tANS table Assymetrical Number System) which is as efficient as Range Encoding / Arithmetic coding, but is as fast as Huffmen coding.
With a few other similar tweak (more exhaustive but still fast match searches, etc.) it manage to be better as Gzip at similar speed (or faster at similar compresion).

You are making the mistake of attempt of think Linux is run in all embedded when it does not.

Well, given the quasi-monopoly of Linux on "anything-but-the-desktop", I would say that this is an acceptable rounding error :-D

Aircraft and in general critical systems don't use Linux as it isn't Real Time nor safety rated, so Linux boot times are irrelevant. They use VXWorks or any other decently well-known RTOS with proper safety certifications.

Oh common. We all know that even the mundane modern cars are data-center on wheels due to the amount of computers they have.
At that scale an airplane would probably qualify as "a university campus worth of different HPC centers" :-D

Thus there are guaranteed more than 1 computer, and certainly more than 1 single OS.
I would suspect that Linux would be popular on some user-friendly touch interface. But again, just like the infotainment certer in cars, these probably aren't crucial and probably aren't covered by the "1 sec to reboot" requirement anyway.

Starshipeleven knows and has observed many embedded systems boot through their debug serial port and seen that of 8-10 seconds of boot time the bootloader itself is wasting 1-2 seconds (uboot for example can be slimmed down to boot more or less instantaneously and I know of some opensource projects that do that, did I ever see any OEM do that? nope, they all blindly ship the hardware manufacturer's bootloader with 0 modifications), decompressing is also a second tops if the kernel image is bare, or much more if the kernel image has also an initramfs embedded into it (dumb choice, but easier development), and then you have bulk of the time taken by the kernel/OS itself starting up (again because it's crap and uses scripts and hacks and is therefore 100% sequential even when it could just be parallelized like systemd does).

The same device booting in 8-10 seconds boots in like 4 seconds if I replace its firmware with LEDE, and if I replace bootloader I can shave another second or two.

Speaking of boot process optimisation (and systemd involvement), Jolla did release their official "Sailfish X" together with Sony Open Device for their Xperia X.
I find it surprising how much faster Sailfish will boot (thanks to a parallel systemd boot) compared to the stock Android.
And that is including the 5 seconds warning about unlocked firmware that the Bootloader displays before booting.

I'm waiting to see a zstd mode for zram. Would be really useful there.

That would be definitely interesting !
(specially on ram-constrained embed device)