DM-Crypt "High Priority" In Linux 6.10 Makes For Better Throughput & Latency

[phoronix]

Phoronix: DM-Crypt "High Priority" In Linux 6.10 Makes For Better Throughput & Latency

The Device Mapper changes were merged last week into the Linux 6.10 kernel, including a new "high priority" option for DM-Crypt...

https://www.phoronix.com/news/Linux-6.10-DM

[varikonniemi]

would be nice if such changes came with even a simple benchmark result demonstrating it's effect.

[yump]

Yeah, dm_crypt seems to be growing "this makes it faster sometimes probably, IDK, tune it for your workload yourselves" parameters at an unsustainable rate. I think this'll be #5 by my count. Five bits of "whatever" is too much. They need to start enabling these things by default if they're secular improvements, or flipping them on and off with heuristics or online exploration if they're not.

[Anux]

Yeah, dm_crypt seems to be growing "this makes it faster sometimes probably, IDK, tune it for your workload yourselves" parameters at an unsustainable rate. I think this'll be #5 by my count. Five bits of "whatever" is too much. They need to start enabling these things by default if they're secular improvements, or flipping them on and off with heuristics or online exploration if they're not.

... but at the risk of reducing responsiveness of the system at large.

That is a perfect valid reason to not enable it per default.

[yump]

That is a perfect valid reason to not enable it per default.

But is not, I don't think, a valid reason to pawn it off on the user/sysadmin instead of understanding what makes it beneficial, and when, and at least documenting some concrete guidelines and benchmarks if you can't figure out how to auto-tune it or sort out the underlying problem.

I went spelunking into the history of this tweak, and the whole thing smells.

2017 April: A Google employee on the Android team moves dm-crypt to WQ_HIGHPRI in order to work around a performance problem on "bullhead devices". That's the Nexus 5X, which is a potato with a lot of CPUs. (Aside: I know what a pipeline bubble is in the context of CPU hardware, but I don't grok the analogy in this commit message.) "Duck season!"
2017 November: This is discovered to cause audio glitches under I/O load on a Thinkpad x61s, which is a potato with only a few CPUs (benchmark comparison). The problem is also documented on the kernel bugtracker.
2019 May: Another Google employee on the Chromebook team discovers that CPU pressure from dm-crypt is causing failed SPI transactions on "veyron" (a potato with a middlin' number of CPUs). They fix this by moving hard-realtime stuff to the realtime scheduler, and then...
2019 November: The WQ_HIGHPRI change is reverted; apparently proprietary shitbox vendors have more LKML clout than desktop users who have had broken sound for 2 years. "Rabbit season!"
2023 February 26: It is discovered [1] that dm-crypt is sitting on the CPU long enough to trigger the soft-lockup watchdog in some circumstances, in spite of the normal priority. Maybe only a problem for non-preemptible kernels? Anyhow, fixed by adding a cond_resched().
2023 February 28: Huawei employee raises concern [2] about the effect of that on write performance under heavy CPU load.
2023 March 6: A patch is sent that adds the high_priority tuning option, citing [2].
2023 March 11: Huawei guy reports benchmark result [3] showing that raised priority actually makes I/O performance worse under CPU load??? Note that the CPU load in question is stress-ng saturating the machine, which is not doing any useful work. So unlike the real world, anything that gives more CPU time to dm-crypt should be """better""".
2024 April 8: high_priority tuning patch is resent, again citing [2] but with no reference to [3]. Patch makes it in under a slightly different name.

Personally, my hunch is that enabling this tuning option will be counterproductive if the threads dm-crypt would steal CPU time from are actually doing something that contributes to forward progress (of whatever workload the machine is running). Possibly even counterproductive in all cases, seeing [3].

I like Vito's theory that the core problem is dm-crypt not inheriting the priority of whatever thread actually caused the I/O.