Linus Torvalds On The Importance Of ECC RAM, Calls Out Intel's "Bad Policies" Over ECC

[coder]

Either prove why my source is bad or accept it.

I explained why it's not relevant. You're being irrational. If I cited some bogus source and you had nothing comparable to cite, I'm sure you wouldn't simply accept data you knew to be bad.

If you don't accept the fact that the modules tested don't necessarily reflect modern, consumer-grade DRAM that's manufactured on a different process node and with other unknowns regarding the cell design, then we'll have to leave it as an irreconcilable point. But you'd be well-advised to consider that clinging to bad data damages your credibility, in addition to your argument.

Puget says that RAM is only getting more reliable, so the outdated sources I showed were a worst-case scenario.

No trend goes on forever. If DRAM is only getting more reliable, then why must DDR5 now embed ECC within the modules?

Because I'm finding myself repeating things to you I've already said to others. You aren't reading everything.

There are other explanations for the sense that you need to repeat yourself. For instance, if I don't accept some claim you've made, it might be that I saw it and dismissed it, in which case repeating yourself isn't going to change anything.

2. Getting ECC support on Intel isn't that expensive and therefore not worth complaining about.

Here's my concern about Intel's position and where I agree with Linus: by reserving ECC for niche processors that are sometimes significantly more expensive that their consumer counterparts (in cases where they exist), Intel is shrinking the market for ECC support in mass-market motherboards and unbuffered DIMMs. At the low end, that's where it really hits the average consumer. And not only are these products more expensive and available in more limited selection, but ECC UDIMMs traditionally lag their non-ECC counterparts in terms of their rated speed, often leaving buyers with an unnecessary and exaggerated choice between speed and reliability (granted, that tradeoff always exists at some level, but there's no good reason why ECC UDIMMs lag quite as much as they do).

And while I agree that (so far) ECC hasn't been a practical necessity for the unwashed computing masses, it's not without benefits. Therefore, it should be as accessible as possible -- not used as a differentiation point for premium products!

I already cited a source, so wtf else do you want?

The claim was ridiculous on its face, so we really don't need to debate it.

I mentioned more than just bad sectors, you're just reading what you want to read (or, just evidence you haven't actually read everything I wrote).

You said:

Doesn't change the fact that data corruption errors on disks happen more often than ECC errors [for home users].

By your own data, it's simply not correct!

But let's for a moment assume I only mentioned bad sectors: you are getting annoyingly pedantic over insignificant details.

Only if you define insignificant details as any details that are factually inaccurate! If you're not going to say what you mean and stand by what you say (or concede when it becomes clear that you've overreached), then we cannot have a meaningful conversation.

I simply request that you remain focused on the original claim, instead of trying to twist the point into something that's more defensible.

It can but as I've said multiple times: ECC isn't a miracle worker and it won't save you from everything.

Nobody is saying it is. That is a stawman.

f you have frequent memory errors, getting ECC RAM is not a sensible solution.

We're not saying that it can be used as a substitute for normally-functioning DRAM. It's like how you're not supposed to drive recklessly, just because you're wearing a seatbelt.

You have to fix the underlying problem first.

And how do you even know you have an underlying problem, without it? Yours is a purely reactive approach: waiting until data or productivity is lost. I chose to pay a little more and have an early warning system that can pay big dividends by saving my time and data. It's both an early-warning mechanism and an insurance policy for your data.

By your logic, that's like getting a bullet wound and you decide to fix it by taping a cotton ball over the hole and downing a bottle of painkillers.

No, ECC is like a bulletproof vest. It keeps you from getting punctured, and lets you survive long enough to take evasive action.

Depending on ECC to fix frequent memory errors is just plain irresponsible.

It would be, if you just sat there and ignored them. But, if I'm seeing a disproportionate number of single-bit errors, what I do is replace the DIMM (or whatever else is failing), before I start getting uncorrectable double-bit errors!

That said, even someone who ignores the logs and continues to operate without regard for any errors still gets far more reliability out of the same underlying DRAM. Since your proposition amounts to simply playing the odds, I don't see how you can conclude that ECC doesn't significantly improve them.

[coder]

ZFS solved this issue via software, not hardware.
Why can't ECC be solved via software checksums?

You can't (practically) code around something so fundamental as unreliable RAM. Probably the closest one could come is doing/storing everything at least twice and compare the result (using a 3rd run as a tie-breaker), but anyone unwilling even to pay for ECC RAM would accept that kind of performance penalty.

Furthermore, many types of software cannot really do a tie-breaker, at least without storing yet a 3rd copy of the data. However, depending on how much data and for how long it's stored, you might end up just doing some sort of ECC scheme in software.

I know performance would drop, but seems like it should be an option you enable in the BIOS if you're willing to lose performance.

High-end servers, mainframes, and other types of mission-critical systems have support for mirroring in hardware. Again, much more costly than simple ECC.

ECC memory can still go bad and doesn't even address the worst errors in the first place

It's about mitigating by far the most common errors, and ECC is just the first tier of fault-tolerance.

This isn't just system memory. There's GPU memory, CPU L1, L2, and L3 cache, SSD and hard drive cache, RAID card cache, highspeed network card cache.

While I don't know about enterprise storage or networking, all of the others are available with ECC. Even PCIe has a CRC mechanism to help protect data in transit to/from the peripherals you mentioned.

Getting back to storage, while even consumer SSDs and HDDs have had error-correction schemes for literally decades. For more on protection beyond the persistent media layer, I would refer you to Intel's explanation of their "end-to-end data protection" capability:

https://www.intel.com/content/dam/www/public/us/en/documents/guides/intel-ssd-technology-terminology-guide.pdf

[schmidtbag]

I explained why it's not relevant. You're being irrational. If I cited some bogus source and you had nothing comparable to cite, I'm sure you wouldn't simply accept data you knew to be bad.

It's not relevant, in your opinion. Opinions aren't rational. So, you're arguing irrationality with irrationality.
You do have something comparable to cite: find me a modern source about RAM reliability. You're questioning the age of my sources. Find something newer.

If you don't accept the fact that the modules tested don't necessarily reflect modern, consumer-grade DRAM that's manufactured on a different process node and with other unknowns regarding the cell design, then we'll have to leave it as an irreconcilable point. But you'd be well-advised to consider that clinging to bad data damages your credibility, in addition to your argument.

As I already stated, Puget suggest RAM is getting more reliable, so, my credibility is fine. Your lack of evidence does nothing good for your credibility.

No trend goes on forever. If DRAM is only getting more reliable, then why must DDR5 now embed ECC within the modules?

You say that as though ECC is coming on all DDR5 modules (it's not). Not sure what embedding has to do with anything. It's likely a cost-saving measure, since ECC is such a critical mass-produced product in the server industry.

Here's my concern about Intel's position and where I agree with Linus: by reserving ECC for niche processors that are sometimes significantly more expensive that their consumer counterparts (in cases where they exist), Intel is shrinking the market for ECC support in mass-market motherboards and unbuffered DIMMs. At the low end, that's where it really hits the average consumer. And not only are these products more expensive and available in more limited selection, but ECC UDIMMs traditionally lag their non-ECC counterparts in terms of their rated speed, often leaving buyers with an unnecessary and exaggerated choice between speed and reliability (granted, that tradeoff always exists at some level, but there's no good reason why ECC UDIMMs lag quite as much as they do).

They're not that niche, at least in desktops. The average consumer doesn't give a shit, because significant RAM failures just simply aren't enough of a problem to such people, hence my point.

And while I agree that (so far) ECC hasn't been a practical necessity for the unwashed computing masses, it's not without benefits. Therefore, it should be as accessible as possible -- not used as a differentiation point for premium products!

It is plenty accessible. $300 for a 6c/12t Xeon with ECC support is accessible. You can go cheaper, or go with AMD.

The claim was ridiculous on its face, so we really don't need to debate it.

I'm not the one who brought it up, asshole. Remember: you're the one who responded to me first and inserted yourself in a dialogue you weren't apart of.

By your own data, it's simply not correct!

IT'S NOT MY DATA. I cited a source proving how disks are less reliable than RAM. You'd have to be an idiot of a PC enthusiast to think otherwise. And as usual, here you go questioning my sources without having anything to back up your claims. Either accept my sources, provide your own, or shut up. You're accomplishing nothing.

Only if you define insignificant details as any details that are factually inaccurate! If you're not going to say what you mean and stand by what you say (or concede when it becomes clear that you've overreached), then we cannot have a meaningful conversation.

Holy shit dude you are obnoxious... You're telling me to concede over an argument you invented out of your own mental gymnastics. I've made my point very clear and you're derailing this to such an absurd extreme.

I simply request that you remain focused on the original claim, instead of trying to twist the point into something that's more defensible.

How about you do that, asshole? I made 2 points, and here you are nitpicking the differences between drive failures vs bad sectors, old citations, and your own opinions (which is ironic considering how much you seem to care about data).

Nobody is saying it is. That is a stawman.

Do you know what a strawman fallacy is? You were saying to use ECC in a situation where RAM failures are common. What I said is a direct argument against that. It's really not that hard to follow...

We're not saying that it can be used as a substitute for normally-functioning DRAM. It's like how you're not supposed to drive recklessly, just because you're wearing a seatbelt.

Then why suggest it when the scenario was frequent RAM failures?

And how do you even know you have an underlying problem, without it? Yours is a purely reactive approach: waiting until data or productivity is lost. I chose to pay a little more and have an early warning system that can pay big dividends by saving my time and data. It's both an early-warning mechanism and an insurance policy for your data.

I take it you've never actually had a real RAM issue because it's very apparent when there's a problem.
lol and here you are, accusing me of strawmanning. If you're talking "big dividends" and "saving time and data", we're not talking about the average tablet or family PC anymore. I don't know how to get it through your thick head that I agree ECC is critical to workstations and servers.
I repeat: ECC is critical to workstations and servers.
Again: ECC is critical to workstations and servers.
So tell me: what big dividends and critical data I'm I losing right this moment writing this to you on my $400 laptop? Oh, that's right - none. So take your stupid strawman and burn it.

No, ECC is like a bulletproof vest. It keeps you from getting punctured, and lets you survive long enough to take evasive action.

Ok fine, let's use bulletproof vest instead. The way I saw the scenario: you have a constant memory issue, and your solution was to replace the non-ECC RAM with ECC. If the RAM wasn't the problem, what you're doing is putting the bulletproof vest on after you already got shot.
If what you meant to say was to already have ECC from the very beginning, well, you didn't make that clear. But even then, that's like walking around in your day to day life with Kevlar. Why does the average person need to wear Kevlar all day every day?

I don't see how you can conclude that ECC doesn't significantly improve them.

That's because I never said or implied that.

[coder]

Soo.. With that in mind, performance UDIMM for AM4 platform..
Kingston seems to have 3200mhz Udimm available - but the question is will it work on x570 boards?

If you want to play it safe, limit yourself to the memory specified in you mobo's QVL (Qualified Vendor List), on their website. Here's the QVL for a board that's caught my fancy as a likely upgrade to my fileserver:

https://www.asrockrack.com/general/p...4U-2L2T#Memory

Unfortunately, they typically test only a very small subset of the available memory and memory brands often refresh their lineup more than once per year, leading to a situation where most of the qualified memory can no longer be purchased for boards nearing the end of their product cycle.

So, the next best thing to try is finding RAM according to their guidelines about what speeds are supported at what ranks, and with what channel occupancy. And obviously make sure the voltage matches. For an example, check out page 22 of that board's user manual (under the section titled "DDR4 UDIMM Maximum Frequency Support"):

https://download.asrock.com/Manual/X570D4U-2L2T.pdf

If they don't break out that level of detail, then look up the specs on some UDIMMs that are on their QVL and try to buy memory that is specs-equivalent.

not going to link website since I don't know phoronix rules regarding links.

I don't know if it's any different for new accounts, but links are allowed (and appreciated, if on-topic), but the forum software will withhold your post for review by a moderator if you have too many links. I think it just looks at the number of links, because even if they're all wikipedia, your post can still get quarantined.

how would one go by to check if ECC is indeed working as intended?

Ah, now that's the question. There are better places to discuss such details, but I'll link one individual who is planning to try error-injection and promising to post their findings:

https://www.realworldtech.com/forum/...rpostid=198781

Also, welcome!

[coder]

With memory patrolling the memory controller constantly reads/writes a test pattern to the memory to catch any errors before the memory is (hopefully) being used by the system.

My understanding of patrol scrub is that it slowly walks through RAM and verifies the checksums of the existing contents. While this is not as efficient at finding memory faults as writing carefully-patterned data, it's a lot simpler and you just hope that repeating the process on a continual basis will turn up any faults, eventually.

That's how RAID scrubbing (sometimes called "consistency-checking") works, anyhow.

[coder]

If the bit flipped while you were editing it, you would either notice the change (and since you said it was encrypted, a single flipped bit is going to have a cascading effect, making it more noticeable) or it would trigger the application to crash. So, you wouldn't get to the point where you save it in the first place.

This is perhaps too trivial an example, but if we take the case of a document, what if the error occurred on a different page than where the user is editing?

Also, a bit-error can have a disproportionate impact, such as shifting a memory address or array index by a large amount. Usually, such changes would result in a program crash, but they could instead have the effect that part of the document goes missing or is replaced by a copy of some other part.

Furthermore, consider that software sometimes maintains more than one copy of the data, for various practical reasons. So, when an error occurs in the primary, you might not see it reflected on screen, immediately.

Finally, the very process of saving a document usually involves a number of copies and transformations, during any of which a memory error could corrupt what's eventually persisted to nonvolatile storage.

I don't think you know how low-level RAM goes, because when RAM is fucked, you know it immediately.

You seem to be contradicting your earlier position that "simple" computer users don't need ECC memory because errors are likely to occur in unimportant data, and therefore will go unnoticed.

Usually, bad RAM gets noticed only when it's so bad that it leads to program or OS instability. However, even before that point, it quite plausibly could've corrupted some of a user's data. I see it as a continuum, rather than the sort of sudden cliff that you suggest.

As I've said a hundred times already: the average home user doesn't need to worry about such things.

His basic scenario is as legitimate for them as anyone else. It's just a matter of how much content they're editing, how long it sits in RAM, and how susceptible it is to memory errors. Uncompressed image data is probably the most resilient to errors, while anything that's highly-structured is probably the least.

[coder]

THERE ARE HUNDREDS OF THOUSANDS OF PEOPLE THAT USE "REGULAR" COMPUTER FOR PROFESSIONAL USE,

Yes, I do get it. The Industry's "solution" is to market different computers for business vs. gaming use, but it should surprise no one that even corporate-oriented PCs are frequently available with non-ECC memory.

My employer used to purchase all of the engineering PCs with ECC RAM, but I guess someone drove too hard a bargain, in some round of price negotiations, and they now come with non-ECC RAM! But, it's not these bean-counters who have to miss family vacations and work nights + weekends, if data corruption should jeopardize a product release date.

While I agree with you and Linus that ECC shouldn't be used to differentiate premium products, I'm not hopeful that he can shame the industry into changing its ways. Even Microsoft couldn't do it, near the zenith of its power (in the days leading up to the launch of Windows Vista). Mainstream hardware is a price-sensitive business.

Anyhow, we're all expending a lot of energy on this, just when the DDR5 spec already has integrated ECC support. So, let's see how that plays out. I fear it'll somehow get derailed by the same forces, but perhaps not.

On a side-note, I'm becoming more than a little alarmed at the recent price-inflation of workstation CPUs, and the increasing divergence between PC and server hardware.

[coder]

ECC seems like the simplest solution, but it's still only system memory. There's still the issue of software errors elsewhere and other devices and firmware. It seems like there should be a more total solution available than ECC which doesn't even fix the worst errors.

Well, I think we've already touched on RAM elsewhere in the system (and interconnects, too), as well as more reliable options than "plain ECC". As far as firmware goes, it always has some sort of CRC to ensure its integrity.

Not to seem glib, but if you're interested in the topic at large, this might be a good place to start:

https://en.wikipedia.org/wiki/Data_integrity

[coder]

this means a law who makes non-ECC against the law would have postive effect.

Though I doubt you're being serious, I disagree. Videogame consoles and video streaming devices seem to do alright without it, and it's difficult for me to see what positive impact it can have for them that would offset the downsides of its added cost.

The law is a very blunt instrument and undermines (potentially) intelligent decisions and tradeoffs made by designers and engineers. If anything, laws should focus on disclosure of a device's data integrity properties. It's not so crazy, if you consider we have energy-efficiency labeling requirements for automobiles and appliances, and we have nutrition labels on food. But, since politicians skew older, less technical, and tend to have non-engineering backgrounds, they'd probably manage to bollocks it up.

[coder]

It's not relevant, in your opinion. Opinions aren't rational. So, you're arguing irrationality with irrationality.

Save the post-modern relativistic BS for literature class essays, please.

Your lack of evidence does nothing good for your credibility.

Evidence of what? I'm not the one making the claim, here. You did, and you cited a source which doesn't support it. That's different than saying your claim is incorrect.

Puget suggest RAM is getting more reliable, so, my credibility is fine.

I find it interesting that you didn't even link it. That's hardly the mark of an excellent source.

You say that as though ECC is coming on all DDR5 modules (it's not).

According to Anandtech's latest DDR5 coverage, it is.

https://www.anandtech.com/show/16289/teamgroup-announces-intentions-for-ddr5-elite-ddr54800-in-2021

If you have a better source to the contrary, please share it for our collective education.

Not sure what embedding has to do with anything.

It's evidence that single-bit error frequencies are indeed becoming too high, with newer cell sizes. Otherwise, why would they burn the overhead on it? In this case, it's yet worse than DDR4, with an overhead of 25%, instead of a mere 12.5%!

It's likely a cost-saving measure, since ECC is such a critical mass-produced product in the server industry.

That doesn't even make sense. The consumer market is large enough to support a different set of DRAM chips, to the extent it makes sense to do so. For each client CPU sold, there will be about a couple dozen DRAM chips accompanying it. Compared to CPUs, DRAM chips are tiny and simple. If Intel can justify at least 4 different CPU dies in each generation (or about 7, if you include laptops), then surely DRAM makers can afford to design a separate chip for servers vs. clients!

They're not that niche, at least in desktops.

Given that Intel's standard desktop CPUs do not support them (with a few exceptions), how are they not niche? Probably no more than 10-15% of the desktop board models out there support ECC. And those are mostly premium models that cost much more than average. I don't know what you consider niche, but I think that fits most people's definition.

It is plenty accessible. $300 for a 6c/12t Xeon with ECC support is accessible. You can go cheaper, or go with AMD.

There are two ways to look at accessibility. First, there's the minimum entry price, which must take into account not only the CPU, but also the motherboard and the ECC memory, itself.

Second, we can look at the price premium over an equivalent non-ECC setup.

Finally, as I mentioned, even AMD Ryzen CPU users will suffer the inflated prices of the motherboards and DIMMs.

The point isn't that it's cost-prohibitive for most, but that it's a nontrivial difference, for many.

I'm not the one who brought it up, asshole. Remember: you're the one who responded to me first and inserted yourself in a dialogue you weren't apart of.

This is a community forum, where all users can read, post, and reply to all messages. If you think my behavior is out of line, you're free to take it up with the mods, but I'll point out that I'm not the one hurling insults.

If you want to have a private conversation, you're free to use direct-messages, email, etc.

As for not being the one who brought it up, it's your exact words that I quoted. Don't say things you can't back up, or at least be a decent person and admit when you've done so.

IT'S NOT MY DATA. I cited a source proving how disks are less reliable than RAM.

In an earlier post, you said:

Back in 2009, Google showed that 1-5 bits would flip per hour per 8GB

Are you now saying storage sectors fail more frequently than that?

Holy shit dude you are obnoxious...

If you consider expecting you to stand by your words obnoxious, then I guess so.

You're telling me to concede over an argument you invented out of your own mental gymnastics. I've made my point very clear and you're derailing this to such an absurd extreme.

All I expect is for people not to play fast and loose with the facts, to be accountable for their statements, and to maintain a basic level of decorum. You don't have to concede anything, but it's hard to have a productive discussion when one party is twisting and shifting their position and refusing to be pinned down. We can certainly agree that a point is irreconcilable and move on, but that at least takes some agreement on what point is in dispute.

As for mental gymnastics, you'd have to cite an example for me to understand what you mean.

How about you do that, asshole? I made 2 points, and here you are nitpicking the differences between drive failures vs bad sectors, old citations, and your own opinions (which is ironic considering how much you seem to care about data).

For opinions to carry weight, the details matter. That's why I'm focusing on deails. I trust you know the difference between an informed and uninformed opinion? There can also be misinformed opinions and underinformed opinions. What I'm trying to do is help nail down the details, so that more people can hopefully hold more informed opinions (myself included).

Maybe if your interest extended beyond winning what you perceive as arguments, you'd see a different theme in my contributions to this thread. Maybe not. But, if you're only reading my replies to you, and if you view this as a zero-sum interaction, then it can't help but shade your perspective.

Do you know what a strawman fallacy is? You were saying to use ECC in a situation where RAM failures are common. What I said is a direct argument against that. It's really not that hard to follow...

I had not proposed to use it in the way you suggested. Nobody did. It looked to me like you setup that strawman and burned him down.

Then why suggest it when the scenario was frequent RAM failures?

That is either a misunderstanding or mischaracterization of what I said. If you want to quote it, I can hopefully clarify whatever statement gave you that impression.

I take it you've never actually had a real RAM issue because it's very apparent when there's a problem.

Like we keep telling you, the reactive approach puts your data at risk. I get that you're fine with the level of risk, but it's nonzero for sure.

lol and here you are, accusing me of strawmanning. If you're talking "big dividends" and "saving time and data", we're not talking about the average tablet or family PC anymore.

It's up to the individual to value their own time and data. I know that if my ECC RAM saves me from data loss, I'd indeed consider it a good return on investment.

So tell me: what big dividends and critical data I'm I losing right this moment writing this to you on my $400 laptop? Oh, that's right - none. So take your stupid strawman and burn it.

As I said, it's like an insurance policy. Insurance doesn't always get claimed. But, when it does, it's usually much appreciated.

If what you meant to say was to already have ECC from the very beginning, well, you didn't make that clear. But even then, that's like walking around in your day to day life with Kevlar. Why does the average person need to wear Kevlar all day every day?

I apologize that it wasn't clear. Not everyone needs Kevlar attire, but we're all at some risk of being shot (in this analogy). I think we agree that the level of risk and exposure varies.

That's because I never said or implied that.

You did say that "Depending on ECC to fix frequent memory errors is just plain irresponsible." Perhaps I was reading too broadly into your use of the word "depending". If you meant specifically for cases where ECC is required, then I agree.