Linux 6.0 Adding Run-Time Verification For Running On Safety Critical Systems

[oiaohm]

When I read a statement like the following, then I'm pretty sure that Linux can be considered a hard real-time OS, which just means that it will always react on time to an incoming event in a pre-defined timeframe;
nothing more, nothing less:

https://elisa.tech/

Nothing is this black or white.

https://elisa.tech/wp-content/uploads/sites/75/2021/05/ELISA_Advancing-Open-Source-Safety-Critical-Systems_Whitepaper_050521.pdf

Page 9

As you can see, the ELISA WGs collaborate and work towards providing resources for
System integrators to apply and use to analyze qualitatively and quantitatively on their
systems. You might ask, “Why can’t ELISA qualify safety-critical systems?” We can’t,
as we don’t know the product details to be able to do so. It is the System integrator’s
responsibility to use the resources ELISA is developing to analyze their systems.

Safety critical is a final solution thing. Yes

https://www.mdpi.com/2079-9292/10/11/1331/pdf

The above write up is clear PREEMPT_RT is a better hard realtime than the default Linux kernel by quite a bit. The question for safety critical is how many μs until something bad happens. The default Linux kernel without PREEMPT_RT is good enough to do hard realtime where 1 ms/1000 μs jitter is not a problem on most hardware. PREEMPT_RT basically halves this. Note most hardware is taking very broad brush.

The question with doing safety critical hard real-time is what is the pre-defined max timeframe to be safe. Pre-defined timeframe its possible that a safety critical hard real-time is that tight that you cannot use micro-controller instead have to use hard wired circuit board logic for the real-time actions. Think emergency stop button on a lot of machines here lot of these do not make sense to put though a Micro controller.

Now think room lighting control 1ms delay on lights turning on or off in reality this will make no over all difference. Yes lighting is safety critical so is emergency stop button.

We can even go to medical lets take a machine given a person Insulin injection if the time of the injection is different by a few seconds the results are not going to be deadly yes this is a few million μs of acceptable jitter. Part of ELISA is safety critical for medical working group this does include insulin injections where the maths being right is more critical than real-time behavour as too high of dosage or too small of dosage can be fatal.

Linux kernel does have suitable hard realtime behavour for particular use cases without PREEMPT_RT and with PREEMPT_RT is suitable for more use cases. There are always going to be the hard real-time requirements that the Linux kernel and all RTOS can never meet.

Linux kernel for hard real-time OS usage still has room for improvement. The reality is the Linux kernel is used in many places today where a historic RTOS operating system would be used. Current Linux kernel with PREEMPT_RT has lower jitter than some of the historic RTOS options on the same hardware.

I would say that the Linux kernel is coming the worst line for those making decanted RTOS options. As in if their RTOS option is not better than the Linux kernel in some way they might as well stop development and move on to Linux. We have seen over the past 5 years many RTOS makers leave the business. This will increase as the Linux kernel comes able to be more validated. Yes some of the RTOS options on sale in commercial don't have the same validations elisa is working on for the Linux kernel.

[rleigh]

With this, and if PREEMPT_RT lands in time, Linux 6.0 will have the option to become a verifiable hard real-time OS, right?

If true, I'd like to hear the opinions of the usual suspects around here who loved to claim that Linux could never get successfully transformed into a trustworthy RTOS!

I'll be one to view this with scepticism, despite being a Linux user and developer for over 25 years. Linux is an OK desktop and server OS, but real-time requirements are mostly incompatible with a time-sharing general-purpose OS. Use the right tool for the job, and for real-time use there are much better tools available. Better performance in this area is of course a good thing, but it's also important to be realistic about what this enables, and what it does not.

"Real-time" requires concretely defining based upon the specific application requirements, including timing and safety requirements. In some cases Linux might be a suitable fit, in other cases not so much.

If you want to achieve highly accurate and consistent timing, then you aren't going to do this on a regular application processor, let alone with Linux. Ignore the specific OS for a moment. A modern CPU with speculative execution and branch prediction and multi-level caches isn't going to provide the necessary guarantees. All these features are about speeding up program execution in various ways, and reducing latency of memory accesses. They aren't about consistency. That's why we have dedicated microcontrollers which deliberately omit these features, and which behave deterministically. For example, take a look at all of the MCUs with an ARM Cortex M4 core, certified for automotive, medical, and other safety-critical uses. All memory accesses are consistent and deterministic. There is no speculative execution or branch prediction. So the same code paths will always use the same number of CPU cycles under all circumstances, so your timing can be accurate.

When it comes to Linux, verifying and validating it is extraordinarily difficult. It's simply too big and complex. We have small RTOSes which can and are painstakingly validated for safety-critical use. They are small, simple and completely understandable. All the interactions are clear and unambiguous. A single person can understand the whole system.

Even with PREEMPT_RT, Linux has serious timing deficiencies. Recently, I noticed spurious outages on an I2C bus during transactions, with SDA held low before and low or high after ACK/NAK for variable periods up to several tens of microseconds. Sometimes it was completely fine, other times variable delays were present. Presumably the I2C driver was pre-empted by something considered a higher priority. But on a properly-configured microcontroller, I would expect jitter-free operation with timing accuracy down to well below a microsecond, and if you use careful hardware and software configuration down to tens or hundreds of nanoseconds, because the system behaviour is deterministic by design at both the hardware level and the software level.

Sometimes even the most ardent Linux fanatics should have a reality-check. It's not suitable for every computational job in existence, and nor should it be. When it comes to life-critical applications, why take the risk when there are more appropriate hardware and software solutions for these tasks. I do feel that shoehorning Linux into these roles is somewhat reckless and irresponsible, but Linux fanboys are busy doing just that in companies around the planet because for them it's the hammer for every nail and for them using Linux at all costs is a higher priority than user safety. When a bug, unexpected interaction or system misconfiguration could result in serious harm or death, maybe look at the bigger picture.

[coder]

I do feel that shoehorning Linux into these roles is somewhat reckless and irresponsible, but Linux fanboys are busy doing just that in companies around the planet because for them it's the hammer for every nail and for them using Linux at all costs is a higher priority than user safety.

I think a lot of it is simply comes down to Linux being the tool they know, and shoehorning it to fit their need seems like it would be easier than learning something completely new.

BTW, I have somewhat similar thoughts about Raspberry Pi's being pushed into jobs they're not really suited for.

[oiaohm]

Sometimes even the most ardent Linux fanatics should have a reality-check. It's not suitable for every computational job in existence, and nor should it be. When it comes to life-critical applications, why take the risk when there are more appropriate hardware and software solutions for these tasks. I do feel that shoehorning Linux into these roles is somewhat reckless and irresponsible, but Linux fanboys are busy doing just that in companies around the planet because for them it's the hammer for every nail and for them using Linux at all costs is a higher priority than user safety. When a bug, unexpected interaction or system misconfiguration could result in serious harm or death, maybe look at the bigger picture.

This is also multi sided problem. You have RTOS fans pushing RTOS solution where its not need as well. There are other bigger pictures here.

I think a lot of it is simply comes down to Linux being the tool they know, and shoehorning it to fit their need seems like it would be easier than learning something completely new.

BTW, I have somewhat similar thoughts about Raspberry Pi's being pushed into jobs they're not really suited for.

Its more than learning something new.

Issue 1 that is over looked Software Maintenance. The more unique solutions the more complex your software maintenance comes a problem..
Yes just because you choose a RTOS does not mean you can skip out on the maintenance problem.

https://www.csoonline.com/article/3411971/critical-vxworks-flaws-expose-millions-of-devices-to-hacking.html

Yes horrible things like this happens.

Software Maintenance is a big. Linux kernel real-time behavour does depend on the hardware it being used on. So does your other open source RTOS options.

https://forums.freertos.org/t/xtasknotifyfromisr-causing-interrupt-jitter/10767

For example, take a look at all of the MCUs with an ARM Cortex M4 core, certified for automotive, medical, and other safety-critical uses. All memory accesses are consistent and deterministic. There is no speculative execution or branch prediction. So the same code paths will always use the same number of CPU cycles under all circumstances, so your timing can be accurate.

Yes this shows a person who does not know problem. Yes "ARM Cortex M4" is exactly the core type being used in that freertos bug. Not all certified ARM Cortex M4 is created equally, Yes odd jitter on particular hardware with particular configurations is a problem of the RTOS world choosing RTOS operating system over Linux kernel does not magically make this problem go 100 percent away.

Yes choosing particular ARM core types does not make this problem go away either. Having in-order processor just removed one form of issue. ARM Cortex M4 has a trap.

Intelligent Energy Management (IEM) A technology that enables dynamic voltage scaling and clock frequency variation to be used to reduce power consumption in a device.

So yes ARM Cortex M4 can jitter quite badly due to CPU clock frequency changes run to run. Horrible part here you can end up very high jitter in fact. ARM Cortex M4 if used to specification is only max 200Mhz processor at best. Now you need to remember this is a dynamic clock device and will down clock very long way if overheated.

Several tens of microseconds can in fact come from a ARM Cortex M4 overheating and clocking down.

Coder there are cases where the Raspberry Pi people think is a bad choice but they have not considered that the Raspberry PI you start off with a higher clockspeed with a higher min clockspeed than quite a few RTOS hardware solutions. Yes we end up in a Apple and Oranges problem here due to hardware difference. So a Raspberry PI with Linux on may in fact have a lower worst case jitter when put against existing RTOS on existing hardware and in fact using less power.

There are a lot of cases where the Raspberry PI with Linux RT is a good enough solution. In fact a better solution than the hardware that is being replaced.

Next issue 2: is your inventory of spares. The more unified your solution is less spare parts you have to have.

Coder I do agree at times you see Raspberry Pi used in solutions they are not 100 percent built for but you also need to consider the hardware the Raspberry PI are replacing. Lot of cases it vendor unique hardware that in fact performs the role worse than the Raspberry PI. Yes so you have both issue 1 and 2 in the existing solution with at worse no software maintenance option at all and needing a stack of unique spares to keep everything working that has been solved changing to Raspberry Pi replacement.

I see the Raspberry PI + Linux RT kernel coming the baseline that solutions need to perform better than.

rleigh yes it really easy to put to the Linux kernel and say it bad problem here this is not why Linux kernel RT is being deployed so much. Reason why Linux kernel RT+ generic hardware like the Raspberry Pi is being deployed so much is how badly the existing solutions suxed in real world with your logistics(spare part acquirement and storage), maintenance and performance.

Something to consider you can make every part in a factory or where ever that is safety critical perfect for it role and end up with spare parts problem or you can pick a generic good enough solution. Generic good enough solution a repair tech going to a location in factory to fix fault could possible have a suitable replacement parts on him all the time now in a perfect matched solution repair tech has to diagnose fault then get part because repair tech cannot physically carry all the different spare parts.

The reality like it or not Linux kernel is being used in more safety critical usages so we are needing better verification of behavour for the Linux kernel. It really simple to say its Linux fanboys doing these deployments so missing there are some quite big problems in the safety critical hardware world in general. Yes its software, hardware and logistics. Logistics turn out to be quite a major driving force in companies deploying hardware and software combinations that appear not to fit well. Yes software and hardware wise they are not ideal fit but logistics wise they turn out to be absolutely perfect. Yes perfect logistics fit equals less downtime and more production so more profit generally.