IO performance on Linux in the gutter off late - downward trend across versions

[devsk]

So, what exactly has happened to Linux IO in recent kernels? I am comparing three LTS kernels:

5.15.145
6.1.68
6.6.8 (purported next LTS)

I have a RAIDZ1 of 4 NVME SSDs. Same ZFS version 2.2.2. The only thing I change is the kernel version, rest everything is same. The kernel config has been brought forward with 'make oldconfig'. So, in almost all aspects, it should be identical for common config elements between 5.15, 6.1 and 6.6.

Scrub times:
5.15.145 - 10m
6.1.68 - 12m
6.6.8 - 20m

Attached kdiskmark shots from 5.15.145 vs 6.6.8 (I did not run for 6.1.68). All numbers are lower for 6.6.8.

I was expecting 6.6 to be higher numbers because of all the IO enhancements work that has gone in since 5.15.

Has anybody else done any comparison like this across Linux versions? Is there a thread I can refer if so?

Is there anything I need to tune specifically to double up the sequential IO speed in 6.6.8?

[devsk]

Has anyone done any IO performance comparisons across Linux versions?

[devsk]

Can someone please run kdiskmark with 5.15.xxx and 6.6.xx to confirm or dispute my findings above? This looks like a major IO performance regression.

[devsk]

Same issue with 6.7.3. Half the speed!

How is it possible that I am the only one who has hit this nasty regression from 5.15 to 6.6 to 6.7? There are 2 LTS kernels sitting between 5.15 and 6.7. So, someone must have hit this issue for sure. Maybe people are not looking for performance regressions? But I thought Michael had a test bed for this kind of testing.

[devsk]

OK, found the solution. It was because of kernel cmd line option "quiet". Removing it restored the performance.

[devsk]

OK, found the solution. It was because of kernel cmd line option "quiet". Removing it restored the performance.

This is obviously not the solution...:-)

One of my friends recently said that people on the internet generally don't want to help but they will definitely correct you if you post something obviously wrong. Looks like my friend was wrong! And he has now been corrected....

I am going to be stuck with 5.15 for a while I guess.