Lenovo Discovers Situation Of Linux Dropping PCIe Gen 5 NVMe SSDs To Gen 1 Speeds

[phoronix]

Phoronix: Lenovo Discovers Situation Of Linux Dropping PCIe Gen 5 NVMe SSDs To Gen 1 Speeds

A change made to the Linux kernel in June 2023 has led to a situation where PCIe Gen5 NVMe solid state drives could potentially drop down to Gen1 speeds... Lenovo engineers spotted this issue and bisected the problem along with coming up with a solution...

https://www.phoronix.com/news/Lenovo-Linux-NVMe-Gen5-To-Gen1

[JEBjames]

Michael

Typo

"being dealth with by Lenovo" should be "dealt"

[mikelpr]

dealth soundeth the cool'st

[geerge]

I know this is for servers and fixing the speed is great, but I'm going to give my 2 cents on pcie for consumers anyway because this is the internet baby I can rant about whatever whenever.

For 99% of consumers 1GB/s is plenty for storage, that can be done with pcie 3.0 x1. Lets say 4GB/s should be the standard to cover 99.9% of consumers, achieved with pcie 3.0 x4, 4.0 x2, 5.0 x1. Consumers don't really care about speed except muh number bigger. M.2 having 4 lanes is overkill going forwards, unless it becomes the new pcie (for dGPU) and there's a new 1 lane standard that becomes the next M.2. I'd rather 4x 1 lane connectors directly on the mobo than have to break out M.2 into single lanes in the same way you can break out pcie x16 into 4x M.2. A lot of 2230 M.2 slots already only provide 1 lane meant for wifi modules, get rid of that crap and instead make it standard that consumer boards have at least a pcie 5.0 x8, an M.2, 4x "single lane standard TBD".

tl;dr give us a standard single lane pcie connector that motherboards can have many of (like sata used to be) that would at least cover wifi modules and most consumer storage needs. PCIe 5.0 x1 is plenty fast for most things and single lane is only getting faster every generation.

[coder]

For 99% of consumers 1GB/s is plenty for storage,

I once had a 6 core machine with a SATA SSD and 32 GB of memory. I could do parallel builds and never saw any significant amount of I/O wait. However, with many more cores and less RAM per core, I think that would start to change.

What a typical consumer probably notices most is latency, because a lot of the programs they use still do a lot of serial, synchronous I/O. Lower bandwidth means longer latency, because the amount of time it takes for the sends and receives is higher. Moving to NVMe was great for latency, because it has lower overhead than SATA, but bandwidth is still relevant.

That's not to refute your point that most people could use lower bandwidth links to their SSDs and not see performance utterly tank, but I'm just making the argument that dropping all the way back to 1 GB/s wouldn't be an immeasurable or even imperceptible difference, in normal use cases.

M.2 having 4 lanes is overkill going forwards,

PCIe 3.0 x4 is lower power than PCIe 4.0 x2. PCIe 4.0 x4 is lower power than PCIe 5.0 x2. Because the frequency doubled at each step, power increases nonlinearly. That makes the slower, wider option better for power consumption and it's why laptops have usually lagged behind desktops in adopting faster PCIe speeds.

the same way you can break out pcie x16 into 4x M.2.

In order to split PCIe x16 into quad x4, the PCIe root must support that bifurcation. Otherwise, you need a bridge chip, which adds cost, power, and latency overheads, as well as complicating/breaking things like GPU Direct I/O.

an M.2, 4x "single lane standard TBD".

They don't like to cut M.2 connectivity, because it disproportionately affects older/cheaper SSDs with a lower PCIe revision. That's what makes it hard to cut back on connectivity. Some boards will dynamically reduce lane count, if you populate additional slots. That's probably the best option.

tl;dr give us a standard single lane pcie connector that motherboards can have many of (like sata used to be) that would at least cover wifi modules and most consumer storage needs. PCIe 5.0 x1 is plenty fast for most things and single lane is only getting faster every generation.

It basically sounds to me like you want to go back to the days when the PCIe root node was in a North Bridge that had a high speed connection to the CPU. Then, you can fan out into potentially more lanes than could be directly integrated into the CPU or supported by the current lower-bandwidth connections between CPUs and their current chipsets. The 890FX chipset in my old AMD board could support a whopping 44 lanes of PCIe, and that was a mainstream desktop socket - not HEDT, workstation, or server!

Intel is closest to this model, by having a PCIe 4.0 x8 link between its CPU and chipset. That enables more fanout options from the chipset.

[hwertz]

To be honest when I saw this involved link retraining aftwr a hot swap, I assumed it'd be like 'retraining happens too early, while the device is being inserted and the link is probably noisy' as opposed to 'we read the wrong registers',. Well either way nice it got fixed!

[marlock]

why does it need training anyway?

[coder]

why does it need training anyway?

Speaking as someone who's not a hardware engineer, I think it's to calibrate both sender and receiver to compensate for variations in the electrical and timing parameters of both the physical link and remote endpoint. I've noticed that pretty much all high-speed communication protocols now seem to incorporate some sort of training phase.