Linux 5.2+ Hit By AVX Register Corruption Bug - Affecting At Least Golang Programs

[down1]

I hope this fixes the weird system hangs I've been having.

[jabl]

They ended up making that clearer in the next C standard, but there are still cases where it can't be fixed (when accessing types below the lowest read/write resolution of a specific architecture).

Due to this, in practice, as of C11 byte addressing is a required architectural feature (as long as the C compiler claims support for threads/atomics and thus the concurrency memory model, via lack of __STDC_NO_THREADS__ and __STDC_NO_ATOMICS__ macros, that is). The alternative of faking it via RMW atomics or an LL/SC loop is too slow to be usable in practice.

Even outside concurrency, the main reason why the original DEC Alpha sucked at string processing was lack of byte addressing, necessitating a lot of RMW code for otherwise simple operations. They fixed it in later iterations though.

[knweiss]

The bug is fixed: https://bugzilla.kernel.org/show_bug.cgi?id=205663#c2

BTW: The long version of this debugging story can be found in this Go issue: https://github.com/golang/go/issues/35326

[discordian]

debian testing provided the Linux 5.3 Kernel yesterday, compiled with gcc9 (peviously gcc8 was used). I am back to 5.2 after the graphics seems to be horribly molested after going into standby, and me being unable to log back in.
Of course this can be a caused by a change in amdgpu (or elsewhere), or we are in for some more pesky bugs like this showing up in the future...

[Weasel]

Just write the thread local variable accessor in inline asm and problem solved? How hard can it be?

[carewolf]

IIRC, kernels handle this problem with compiler flags that disable generation of those specific instructions, as if the target hardware did not have them. That definitely moves us further away from the C standard, but is still well-defined in the eye of the compiler.

Yeah, it is all compiler flags, architecture defined behavior or explicit agreements with the compiler maker. At least in theory.

A "funny" case I remember from almost a decade back was the kernel getting a random value on read of a int in one thread. It turned out to be caused by another threading reading a neighbour int with a 64-bit read, then changing his part, then writing the whole thing back overwriting any changes of the other part in another thread. That was all legal for the compiler at the time because the value changed wasn't declared volatile or atomic, but it caused 'random value out of nowhere'.

They ended up making that clearer in the next C standard, but there are still cases where it can't be fixed (when accessing types below the lowest read/write resolution of a specific architecture). So while that is fixed now, making it explicit when it is defined and when it is architecture dependent. I like to use it as an example of how much we can take "sane" behavior for granted, and that there almost always are implicit things we rely on we never know could be broken until they are.

[HadrienG]

True, but there are many types of CPU state that a compiler is technically allowed to change because it is within the calling convension that it may do so, but that a kernel would assume it won't change. For instance not changing any x87 state or SSE registers if the kernel specifically avoids using floating point operations. All kinds of code could break if the compiler started optimizing integer division with fp divisions that trashed some FPU registers the compiler was allowed to change, but the kernel didn't think it would.

IIRC, kernels handle this problem with compiler flags that disable generation of those specific instructions, as if the target hardware did not have them. That definitely moves us further away from the C standard, but is still well-defined in the eye of the compiler.

[carewolf]

Note that C also provides you with semi-standard tools to defer to system-specific semantics, like assembly and volatile. If you're writing a kernel, using those is definitely more sustainable than relying on undefined behavior being compiled in a certain way.

True, but there are many types of CPU state that a compiler is technically allowed to change because it is within the calling convension that it may do so, but that a kernel would assume it won't change. For instance not changing any x87 state or SSE registers if the kernel specifically avoids using floating point operations. All kinds of code could break if the compiler started optimizing integer division with fp divisions that trashed some FPU registers the compiler was allowed to change, but the kernel didn't think it would.

[HadrienG]

Well, relying on system specific behaviour that lies outside the C/C++ standard is sort of necessary for a kernel, but yes, it still could be a programming mistake.

Note that C also provides you with semi-standard tools to defer to system-specific semantics, like assembly and volatile. If you're writing a kernel, using those is definitely more sustainable than relying on undefined behavior being compiled in a certain way.

[carewolf]

Not necessarily. The existing code may have been relying on undefined behaviour. The compiler may have the full right to optimize this and just by the fact that GCC 8 did not do it, doesn't mean that GCC 9 is buggy.

Compilers are under constant pressure of improving their optimizations and the consequence is that they more strictly follow the C/C++ standard as to what they are allowed to optimize and what not. Buggy code that assumes the compiler will compile it in a certain way, while the standard doesn't give that guarantee often becomes a victim of newer compilers.

Well, relying on system specific behaviour that lies outside the C/C++ standard is sort of necessary for a kernel, but yes, it still could be a programming mistake.