Linux 6.0 Adding Run-Time Verification For Running On Safety Critical Systems

[phoronix]

Phoronix: Linux 6.0 Adding Run-Time Verification For Running On Safety Critical Systems

Another big ticket feature has made it for the Linux 6.0 kernel: the Runtime Verification infrastructure for running Linux on safety-critical systems...

https://www.phoronix.com/news/Linux-6.0-Runtime-Verification

[Volta]

Linux is becoming increasingly used within safety critical systems.

Which cannot be said about Windows.

[Ananace]

I figure that this is equal parts useful just for security against hacking and the like in important infrastructure systems etc - things like power plants or hospitals.
But I imagine that it'd be just as useful for hardware that's running in very exotic environments, being able to detect execution errors due to stellar radiation for example.

[marlock]

I have not investigated further, but at a glance I'd bet this is a fruit of AGL - Automotive Grade Linux and similar initiatives:

https://www.automotivelinux.org/

[Linuxxx]

Wait a minute!

With this, and if PREEMPT_RT lands in time, Linux 6.0 will have the option to become a verifiable hard real-time OS, right?

If true, I'd like to hear the opinions of the usual suspects around here who loved to claim that Linux could never get successfully transformed into a trustworthy RTOS!

You peers very well know who you are...

[coder]

Wait a minute!

With this, and if PREEMPT_RT lands in time, Linux 6.0 will have the option to become a verifiable hard real-time OS, right?

I don't know enough to comment in any depth, but I'd caution against throwing around such terminology without knowing precisely what it means or what the PREEMPT_RT patchset actually accomplishes. It's really these details that matter, not what label you slap on it.

[Linuxxx]

I don't know enough to comment in any depth, but I'd caution against throwing around such terminology without knowing precisely what it means or what the PREEMPT_RT patchset actually accomplishes. It's really these details that matter, not what label you slap on it.

If it is really meant to run Linux on safety-critical systems, doesn't this then also automatically imply a hard real-time OS as a requirement?

[oiaohm]

With this, and if PREEMPT_RT lands in time, Linux 6.0 will have the option to become a verifiable hard real-time OS, right?

If true, I'd like to hear the opinions of the usual suspects around here who loved to claim that Linux could never get successfully transformed into a trustworthy RTOS!

This is a lot more complex.

I don't know enough to comment in any depth, but I'd caution against throwing around such terminology without knowing precisely what it means or what the PREEMPT_RT patchset actually accomplishes. It's really these details that matter, not what label you slap on it.

https://www.youtube.com/watch?v=BKkX9WASfpI

There is a nice video on the topic from the developer in PREEMPT_RT patch.

PREEMPT_RT stated by developer that its real-time. The other RT options in the Linux kernel would be called soft real-time. PREEMPT_RT define of real-time is pure deterministic. This is not your hard real-time. Hard real-time OS does not have to error if a real-time objective is not meet. PREEMPT_RT is designed to raise error in case of determinism not being meet for some reason.

Yes its a interesting video to say the least even that it getting old. Its interesting because if you measure by the rules of hard real-time being max latency the difference between PREEMPT_RT and the other options that had Linux kernel on top of a microkernel the max latency risk was basically the same yes "Worst Case Execution Time"(WCET) value. Yes that a 2017 video and this has not really changed in all this time.

Now as in 2017 video shows at the time the average execution latency with PREEMPT_RT is worse for generic application.

Also interest enough the 2017 video demos that restricting the PREEMPT_RT and using platform only features you can in fact get lower average latency than the other options with the PREEMPT_RT and microkernels. Yes platform only features + PREEMPT_RT can end up giving you a better "Worst Case Execution Time".

https://etr2021.ensma.fr/files/p-ficheux-realtime-linux.pdf

Yes we still see people doing presentations like this one. Where is PREEMPT_RT with user space using generic code not using platform particular extensions. Also they are not running the test for long enough to in fact find the "Worst Case Execution Time"(WCET). Yes that 2017 video states that you have to run the test for at least 12 hours that what is required to be at SIL2. 30% to 300% worse is claimed the reality is for WCET that not the case its more they have not run the test long enough to see where the Microkernel with Linux kernel design stalls. If you are talking soft real-time performance where absolute determinism is the important instead the average lattency yes the other solutions vs generically source code PREEMPT_RT are generally faster by 30% to 300%.

Its something to remember you put x=1+1+1; into gcc in execution that will be x=3 so faster now gcc sees the equal optimisation for realtime it does not do it.

Coder you are reality right the details matter. Problem here when you get into real-time the details get horrible complex really quickly. This difference is not always what it appears. Most people would not be thinking that the difference is really the applications source code+the compiler. RTAI and other microkernel+Linux kernel real-time solutions are basically using a smarter build application method. 2017 presentation demoed the existence of this problem.

PREEMPT_RT getting mainline is really only the start. There is lot more userspace work required like making gcc and llvm compilers smarter for the real-time use case so developers don't need to know as much platform particular knowledge to get the best out of PREEMPT_RT. Yes that is the advantage of the microkernel+Linux kernel solutions for real-time at the moment you get away with knowing less platform knowledge to get to particular performance levels.

But remember there is a upside to this absolutely generic code of the current PREEMPT_RT you have a broader range of parts your solution will work with.

If it is really meant to run Linux on safety-critical systems, doesn't this then also automatically imply a hard real-time OS as a requirement?

This is a no a SIL4 certified OS that is the highest rating under IEC 61508 does not in fact mandate hard real-time support. Yes you can have SIL4 and totally not be a deterministic OS so not even a soft real-time OS just never crash in a uncontrolled way for the us case of the system. Think about it a SIL4 system for cataloguing and comparing medical outcomes cannot afford to screw up the processing but the speed is not that critical.

Safety-critical requirements can mandate that a solution has hard real-time support. Safety-critical requirements can also mandate what people call soft real-time as well. What is safety-critical changes on the use case. Yes most Safety critical certifications are for the final product.

Run-time validation will help the real-time kernel work for finding where drivers and other things are doing things they should not be. Also Run-time validation helps all SIL4 use cases. Run-time validation could be left running on some non-realtime SIL4 use cases to error out the system if particular events happen.
Safety critical is not one size fits all.

Run-time validation does also allow creating a formal model for more parts of the Linux kernel and running tests to make sure the formal model and what Linux kernel does does happen to match. Yes parties like Nvidia who make closed source drivers could end up hating this due to making it simpler to find kernel feature miss usage.

[marlock]

I'm by far not an expert, but I'd risk saying "probably but not necessarily"...

you can have non-RT components doing important coordination work (eg. onboard central AI unit defining route, desired steering and speed, etc and the human user UI unit on a self-driving car) over RT components (eg: a bunch of specific-purpose units like LIDAR sensor arrays that need produce very accurately time-labeled data and stream it to the driving AI)

the hard RT bits might run a different minimal OS and have interfaces linking them together with the main units

even the user UI can be considered safety-critical in some aspects, but it definitely can afford to run with small time-irregularities in its unlike the sensor arrays

This is in broad terms what I have picked up a few years ago from early public discussions on AGL, when the project was younger... but of course they might or might not have found a way to go beyond this limitation into the realm of "hard RT" and not just some clever minimization of processing jittter which can be nicknamed RT despite not fully achieving it

[Linuxxx]

I'm by far not an expert, but I'd risk saying "probably but not necessarily"...

When I read a statement like the following, then I'm pretty sure that Linux can be considered a hard real-time OS, which just means that it will always react on time to an incoming event in a pre-defined timeframe;
nothing more, nothing less:

The mission of the Enabling Linux In Safety Applications (ELISA) project is to make it easier for companies to build and certify Linux-based safety-critical applications – systems whose failure could result in loss of human life, significant property damage or environmental damage. ELISA members are working together to define and maintain a common set of tools and processes that can help companies demonstrate that a specific Linux-based system meets the necessary safety requirements for certification.

https://elisa.tech/