Testing The First PCIe Gen 5.0 NVMe SSD On Linux Has Been Disappointing

[Linuxxx]

I use a SATA SSD which is plenty fast and I don't quite understand people's obsession with newer PCIe Gen X SSDs. How often to you copy gigabytes of data? What for do you need this speed?

True, throughput speed is already plenty fast on a SATA SSD for most people.

However, where there can be a noticeable difference is in IO-latency:

That's because SATA SSDs are handled by only a single core through the AHCI driver, whereas the interrupt handling is perfectly spread-out onto every core by Linux on a NVMe SSD.

One can see for themselves with the following command:

cat /proc/interrupts

This is the reason why most Linux distros are not using any IO-scheduler at all on a NVMe SSD, because even under heavy IO-loads most of them will still be able to react fast enough to incoming tasks (although this obviously depends on the particular drive model).

On the other hand, using an IO-scheduler such as BFQ can still markedly improve the IO-latency on a SATA SSD, albeit that will result in slower read & write performance as a trade-off.

[avis]

True, throughput speed is already plenty fast on a SATA SSD for most people.

However, where there can be a noticeable difference is in IO-latency:

That's because SATA SSDs are handled by only a single core through the AHCI driver, whereas the interrupt handling is perfectly spread-out onto every core by Linux on a NVMe SSD.

One can see for themselves with the following command:

cat /proc/interrupts

This is the reason why most Linux distros are not using any IO-scheduler at all on a NVMe SSD, because even under heavy IO-loads most of them will still be able to react fast enough to incoming tasks (although this obviously depends on the particular drive model).

On the other hand, using an IO-scheduler such as BFQ can still markedly improve the IO-latency on a SATA SSD, albeit that will result in slower read & write performance as a trade-off.

For heavy database users - yes, surely. For home users? I just cannot image any workloads requiring that many (concurrent or not) IO operations.

OK, with NVMe SSD PCIe 5.0 your system, Firefox and a couple of electron applications will boot in 20 seconds, with an old SATA SSD it will be ~23 seconds. Is it really worth chasing such speeds and ... heatsinks then? Again, the vast majority of users absolutely don't need this kind of performance unless they have money to burn. And you absolutely need the fastest CPU on Earth to be able to read that much data and execute it cause on a sufficiently old system your CPU and RAM will easily bottleneck any random IO throughput improvements. And sequential IO even for PCIe 2.0 SSDs is already insanely fast.

[Mike Frett]

How often to you copy gigabytes of data?

Every day. Sometimes all day. AI models are quite large.

[unwind-protect]

For heavy database users - yes, surely. For home users? I just cannot image any workloads requiring that many (concurrent or not) IO operations.

Loading the DCS game comes to mind.

[L_A_G]

IMHO PCIe 5.0 SSD's are kind of pointless.

At the speed even a x4 bus operates you need to be working almost entirely out of the DRAM cache to take advantage of the speed advantage over a PCIe 4.0 bus of the same width. Even today's most/worst speed-over-density-and-wear-resistance type NAND flash just isn't fast enough to saturate even a 4.0 bus.

These benchmarks, which are purpose-designed to give the drive a proper workout and heavily spill out of the DRAM cache, demonstrate this quite well. Real life use cases that are far less taxing may perform a bit better, but those aren't the heavy usage scenarios you really want a high end SSD for.

Not that the PCIe 5.0 standard couldn't be used for storage. If somebody used it to build an old school ramdisc (remember those?) that could be an absolute monster of a performer and with the now fairly low cost of DDR4 memory not cost you an arm and a leg.

[M@GOid]

I use a SATA SSD which is plenty fast and I don't quite understand people's obsession with newer PCIe Gen X SSDs. How often to you copy gigabytes of data? What for do you need this speed?

Well, I upgraded from a (then) fast SATA SSD to a modern "as budget as it gets" NVMe SSD, but of much greater capacity (240GB to 2TB), and, there were some performance gains loading things, since modern NVMe drivers had much better random reads. And mind you, budget NVMe SSD, not a "top of the benchmarks" one.

But I agree with you in the matter that ultra fast SSDs are a waste for most users. You cannot benefit from such high speeds if you only transfer data from a home network, USB or even SATA sources. There is the "Directx Storage" thingy from MS for modern games, but initial benchmarks are not impressive.

I mostly got a M.2 NVMe driver because they take less space inside a case and you don't need to run cables for them. Also, from now on, they will be compatible with more devices.

[schmidtbag]

I use a SATA SSD which is plenty fast and I don't quite understand people's obsession with newer PCIe Gen X SSDs. How often to you copy gigabytes of data? What for do you need this speed?

Same here. In real-world tests, there's very little benefit in having anything faster than SATA. Sure, maybe a game takes another 3 seconds to load, or if you do UHD video editing you're definitely going to need a faster drive, but otherwise I'd rather just get what's cheapest and doesn't need a fan to cool it.

My laptop came with an Optane drive, which I use to store my OS and applications. It is noticeably faster (particularly with updates) but in day to day usage, it hardly makes a difference to me.

[Weasel]

Not that the PCIe 5.0 standard couldn't be used for storage. If somebody used it to build an old school ramdisc (remember those?) that could be an absolute monster of a performer and with the now fairly low cost of DDR4 memory not cost you an arm and a leg.

Omg yes, I want a ramdisk in PCIe even with DDR2 speeds is plenty. And it should be as big as possible and cheap, since I don't want performance out of it, I just want unlimited write endurance. Since I want it for temporary storage. The key part is that it doesn't wear over time, and the auto wipe on shutdown is a bonus (instead of something undesired).

My biggest gripe with SSDs is their limited write endurance, especially since the cheap ones use TLC or even QLC. Slow ramdisk please.

[Weasel]

My laptop came with an Optane drive, which I use to store my OS and applications. It is noticeably faster (particularly with updates) but in day to day usage, it hardly makes a difference to me.

Not gonna lie, I was considering Optane since it has almost limitless write endurance. Too bad it got discontinued... also it would be half a waste considering it's blazing fast and yet I just want something big and slow, cheap but with limitless write endurance. Something that doesn't have moving parts, so not a hard disk. And also hard disks tend to have limited write endurance as well (and even reads), though not as low as SSDs.

[schmidtbag]

Not gonna lie, I was considering Optane since it has almost limitless write endurance. Too bad it got discontinued... also it would be half a waste considering it's blazing fast and yet I just want something big and slow, cheap but with limitless write endurance. Something that doesn't have moving parts, so not a hard disk. And also hard disks tend to have limited write endurance as well (and even reads), though not as low as SSDs.

Well, if you do what I did and use it as your boot drive, be aware some distros will take some extra work to boot from a purely NVMe drive. Pretty nice though - KDE gets to the login screen from POST in about 3 or 4 seconds.