"Nest" Is An Interesting New Take On Linux Kernel Scheduling For Better CPU Performance

[ATLief]

Like all nice things, this probably comes with some obscure security vulnerability.

[gamerk2]

I've been thinking about that for years, it's strange no one has raised the issue earlier. The Linux kernel is notorious for its penchant for juggling tasks between CPU cores for no reasons and that results in emptying whatever you had in L1/L2 caches prior and non-zero delays considering new CPU cores could be at their absolute lowest power settings when they are given a task to execute.

You could simply run:

7z b -mmt1

And see in top or any graphical process manager how the task is thrown between CPU cores.

There's a reason for that: You want to typically try and maximize thread uptime. Remember that there are literally hundreds if not thousands of threads fighting for CPU time; your thread of interest is going to get bumped at some point. You then have two options: Either wait for whoever bumped it to finish and reschedule on the same core, or bump someone else and reschedule on another core ASAP. That's why you see instances of high workload threads jumping between cores: The OS is trying to maximize their uptime, even if the core allocation is considered suboptimal.

[Granted, in a situation where you only have a handful of high workload threads, it would be "more" ideal if those were locked to their cores and the rest of the threads fought over the remaining ones. This does incur a latency hit though, and breaks down as the number of "high workload" threads increases.]

[ms178]

BORE is just patched CFS. None of the schedulers I know address this task assignment problems. They are mostly focused on efficiency of task prioritization but none of them actually take into account CPU cores.

Thanks for your insight. I don't know much about these internals, hence I am shocked to get to know that such information is not already accounted for in today's schedulers as boost frequency behavior has a huge impact on performance. Also the scheduler needs to know about power management and core/cache layout, too. I guess that means that there are some performance gains still to be had in the future.

[V1tol]

I use BORE as of late, but I don't know if that already accounts for this.

BORE is just patched CFS. None of the schedulers I know address this task assignment problems. They are mostly focused on efficiency of task prioritization but none of them actually take into account CPU cores.

[CochainComplex]

But this implies that the die layout has to be hardcoded for any cpu or at least any generation especially for the large server cpus...or maybe its given by staying in a numa node (for the lager ones)? maybe it is less problematic then i think

edit. There is another culprit aswell.

Lets imagine the scheduler chooses for performance the best high clockable core. Statistically one out of n is the best and will it always be. This would result in a potential overusage of one core and its neighbours. This will result in more thermal wearout on this particular core (and its neighbour.

[Volta]

Maybe some of the many custom schedulers are already more clever than CFS? I use BORE as of late, but I don't know if that already accounts for this.

Maybe there are, but it depends on workload. From what I'm seeing Nest is just CFS enhancement when comes to task placement decisions.

[Volta]

Volta but is it coming to upstream?

Who knows? We'll see.

[user1]

I've been thinking about that for years, it's strange no one has raised the issue earlier. The Linux kernel is notorious for its penchant for juggling tasks between CPU cores for no reasons and that results in emptying whatever you had in L1/L2 caches prior and non-zero delays considering new CPU cores could be at their absolute lowest power settings when they are given a task to execute.

Interesting.. What you described here is exactly the same as the behavior of the Windows scheduler someone described in another forum. In that forum I've even seen people calling the Windows scheduler "dumb", when it turns out the Linux scheduler actually works in a similar way.

[birdie]

I haven't tested it much but on my first run taskset -c 0 7z b -mmt1 is faster than just running 7z b -mmt1 (almost completely idle Ryzen 5800X here).

[ms178]

Basically they found out that you get better performance by tweaking the scheduling of the workload in a way that makes use of higher boost frequencies when using less cores. Wow, who would have thought?!

I am not surprised that this only affects low to mid-level workloads the most either, as your frequencies fall down the cliff on core-heavy workloads. And yes, as I have a undervolted 18-Core-Xeon with an unlocked turboboost, that CPU is easily hitting its TDP limit, mind you. So I've made use of this behavior to get better performance out of the CPU for a long time now (and am not the only one doing so). It would be great if the Linux Scheduler would be aware of this behavior and make more optimal choices, of course. I only wonder why this is not the case yet, as things like TDP limitations and different base/turbo frequencies are nothing new.

Volta Maybe some of the many custom schedulers are already more clever than CFS? I use BORE as of late, but I don't know if that already accounts for this.