OnLogic Factor 201 Fits The Raspberry Pi CM4 For Industrial Needs & Other Innovative Uses

[coder]

Non-negotiable in an industrial setting. You need non-software safeguards, regardless of costs. If a manager is unable to figure it out, then change the manager. Industrial accidents result in permanent disability, death, many other problems, and lastly financial damages of rework.

Yes, but also having industrial equipment break because a bit flipped and that caused a sacrificial mechanical safeguard to have to do it job would lead to costly downtime and equipment replacement. So, it's a false economy to say you don't need ECC because you have mechanical safeguards to fall back on.

Well, yes, if the hardware is under operating environmental range. If it's over 250F or 120C, the chips start bubbling.

We've establish you need a mechanical safeguard, but I would hope an industrial controller would also enter thermal shutdown, if it exceeded its operating temperature range.

Something like a Raspberry Pi 4 will reduce its clock speed, when it overheats. That was a big issue, when it first launched, due to the performance impact. But that shows us that not only does it have built-in temperature monitoring, but it also has active clock-throttling (and probably thermal shutdown, I presume).

Then they will fail because the chip will fall off from the solder bubbling.

Also, before they get nearly that hot, you might find that the software can't meet realtime guarantees, due to the reduced clock speed. That's another scenario under which it might voluntarily shutdown.

[AndyChow]

The only way to do this is to have non-software safeguards. It's usually doable, but adds cost and will probably incur some degree physical damage, in event of a failure.

Non-negotiable in an industrial setting. You need non-software safeguards, regardless of costs. If a manager is unable to figure it out, then change the manager. Industrial accidents result in permanent disability, death, many other problems, and lastly financial damages of rework.

With ECC RAM, you get an interrupt on memory errors. For single-bit, correctable errors, this is usually informative and typically just shows up as a message in syslog. In the case of double-bit, uncorrectable errors, your process will typically abort with a SIGBUS signal. Presumably, these behaviors can be customized, as desired.

And more than double-bit errors are exceedingly rare, in terrestrial applications. I think satellites and space probes need to handle them, but the rest of us are much more likely to encounter other sorts of hardware failures than a triple-bit memory error.

Well, yes, if the hardware is under operating environmental range. If it's over 250F or 120C, the chips start bubbling. Then they will fail because the chip will fall off from the solder bubbling. And if and when that happens, you better have a non-software solution to prevent the milling machine, or chemical boiler, or whatever it is, from violently deviating from the program.

[coder]

have to design a system that doesn't require ECC ram,

The only way to do this is to have non-software safeguards. It's usually doable, but adds cost and will probably incur some degree physical damage, in event of a failure.

because RAM failure routinely happens even with ECC RAM.

With ECC RAM, you get an interrupt on memory errors. For single-bit, correctable errors, this is usually informative and typically just shows up as a message in syslog. In the case of double-bit, uncorrectable errors, your process will typically abort with a SIGBUS signal. Presumably, these behaviors can be customized, as desired.

And more than double-bit errors are exceedingly rare, in terrestrial applications. I think satellites and space probes need to handle them, but the rest of us are much more likely to encounter other sorts of hardware failures than a triple-bit memory error.

You have to design the system so that when a catastrophic failure happens, it's not catastrophic.

Yeah, but ECC RAM is somewhat akin to having anti-lock brakes, in a car. They're so useful at avoiding collisions and thus the deployment of other safety measures that many places require them by law.

In industrial control, you'd much rather have a simple warning of single-bit ECC errors and have to replace the RAM or the entire controller box than to rely on other safety measures, likely involving sacrificial safeguards and subsequent downtime until you can do repairs and replace the parts.

[AndyChow]

I heard a story (first hand) about an industrial milling machine, where memory corruption in a lookup table resulted in a several-ton table being ejected. Had the tech operating it been standing in a different place, he could've been killed. The root cause turned out to be a software bug, but a hardware memory failure could've had the same effect.

I remember reading a chapter of a book that explained why you could not have manual and digital milling machine at the same time. I forget the book, but it may have been "When things start to think", by Neil Gersenfeld. Anyway there's a whole chapter where they want a milling machine that you can sometimes program by mechanical settings, sometimes digital. But they want a machine that does both. They go around the world visiting the greatest manufacturers, that tell them it's impossible. Then, they find a small manufacturer only a few blocks from their lab, that does just this. I guess this is the machine that almost killed someone in your story.
Do not think regular scripts work in an industrial environment. When the temps "sometimes" go above 80C, and things of this nature, then you have to design a system that doesn't require ECC ram, because RAM failure routinely happens even with ECC RAM. You have to design the system so that when a catastrophic failure happens, it's not catastrophic.

[stiiixy]

'Industrial PC' is as varied as the PC world in general. End point monitoring, control, display kiosk, image server, the list goes on. ECC may or may not be required. It shouldn't even be a point of discussion it's so 'base'. ECC where you need a readout to absolutely work; regular where you're simply watching a display. Most of the industrial shit like pumps and sensors only have two wires and punch out one of four protocols.

[coder]

I guess I'm saying that these chip shortages suck and I'm tired of them. They often seem to have "full dev kits" with cheap microSD and cheap power supplies, but never just the board or CM.

Yup. Chip shortages and I guess they'd do well to work on their distro channel, so that the available parts are harder for scalpers to buy up.

Everything needs ECC. It's only a 15% premium (13% the extra ram chip, 2% the other validator chip that's on ECC ram) if everyone got it.

Yeah, but anything for industrial usage especially needs it. You don't want industrial equipment malfunctioning because a DRAM cell degraded* and no longer holds charge as well.

* Contrary to popular belief, most ECC failures aren't caused by cosmic rays. The practical benefit of ECC is for early warning of DRAM-related hardware failures.

these industrial devices probably require less ECC than any other device. They store data remotely, boot remotely, and process very little data. Because of their nature, it's expected that they can fail and the usual solution is a reboot and PXE to a fresh image, or change the hardware with a service call, without being disturbed by the data corruption/loss.

I heard a story (first hand) about an industrial milling machine, where memory corruption in a lookup table resulted in a several-ton table being ejected. Had the tech operating it been standing in a different place, he could've been killed. The root cause turned out to be a software bug, but a hardware memory failure could've had the same effect.

Therefore it's better to hash every file, understand how to detect failed data, and be able to compensate on failed data, rather than think "It's ECC, we are safe".

Uh, I was talking about RAM, not persistent storage. It goes without saying that NAND or whatever needs strong error correction. And no objection to using a filesystem with checksums, but neither are a substitute for using ECC memory because memory errors can basically invalidate any and all assumptions made by software.

I've had the experience of trying to debug segfaults caused by software reaching an "impossible state", where it turned out the machine had bad (non-ECC) RAM. After banging my head on it for a while, it occurred to me to try a memtest. Once we replaced the bad DIMM, the failure stopped occurring. You don't ever want to waste your time trying to debug software on a machine with bad RAM.

[AndyChow]

I've been on waiting lists in Canada to get my hands on raspberry pi zero 2w for over six months, they are never available (no, I'm not paying $90 on amazon resellers). Same with Rock 5 which I bought a pre-order coupon several months ago.
I guess I'm saying that these chip shortages suck and I'm tired of them. They often seem to have "full dev kits" with cheap microSD and cheap power supplies, but never just the board or CM.

Neat, but needs ECC memory.

Everything needs ECC. It's only a 15% premium (13% the extra ram chip, 2% the other validator chip that's on ECC ram) if everyone got it. Because people aren't aware, in practice it's a much higher premium. I never understood the mentality. I think non-ECC ram should be a niche market for gamers that want higher frequencies, and therefore the minority share. But people don't understand data corruption until it happens to them.

That said, these industrial devices probably require less ECC than any other device. They store data remotely, boot remotely, and process very little data. Because of their nature, it's expected that they can fail and the usual solution is a reboot and PXE to a fresh image, or change the hardware with a service call, without being disturbed by the data corruption/loss.

The environment of industrial cannot rely on "local accuracy". Therefore it's better to hash every file, understand how to detect failed data, and be able to compensate on failed data, rather than think "It's ECC, we are safe".

I agree industrial equipment, and all equipment, should use ECC, but in the scenario of industrial equipment, you should assume the devices will fail or self-corrupt.

[stiiixy]

Would be interesting if Mich tested in some actual 'industrial' environments; if I recall he/you live in northern USA, and it's 'summer' there. Pretty mild.

I want to see a constant high temp (40+C) from a factory with 'foreign particulates' (gluten from a wheat processor could be fun), or so cold monkey's and brass et cetera, holding 12,000 tinnies of freshly brewed beer. A mine processing plant whereby there's constant vibration, the air is safe for humans, but constant exposure is bad for 'certain parts' because of reagents. I could go on, but...tinnies.

[coder]

Neat, but needs ECC memory.

[JEBjames]

Michael

correction:
"one USB 3.2 port, two USB 3.0 ports" should be "one USB 3.2 gen 1 port, two USB 2.0 ports" according to the source web site specs.