Linux 5.18 Released With Intel SDSi, New CPU & GPU Features

[Volta]

Yeah, but thread director and HFI is just an interface which provide detailed performance/power consumption information for each core as they now have small-big cores and thus each core may have different performance/power consumption.

I think it was an only piece missing, but we'll see in benchmarks.

[NobodyXu]

I doubt that. In Windows something notifies the kernel which process is in foreground/background and the kernel then decides which cores this process could/should use. The Linux kernel has no notion of background/foreground processes, it doesn't know that your web browser is more important than some compilation/rendering/compute/etc. background task either. Moving these features into the kernel scheduler is a strict no-go.

On a second thought, the existing scheulder in kenrel might be able to directly utilise information provided by HFI.

The current task scheduler CFS (completely fair scheduler) measures the cpu time to decide which tasks to run next, by trying to ensure all processes are run with roughly the same time (for different priority, their time slice is adjust accordingly).

It could take advantages of HFI when deciding to migrate one task to another core, by placing it onto another core that provides better performance based on HFI if it keeps using up its allocated time slice.

[NobodyXu]

I thought thread director is hardware based and HFI was needed to support it?

https://www.fudzilla.com/news/pc-har...dware-feedback

Yeah, but thread director and HFI is just an interface which provide detailed performance/power consumption information for each core as they now have small-big cores and thus each core may have different performance/power consumption.

To take advantages of it, you need software support, especially support in the scheduler in kernel and maybe also daemon in userspace.

[Volta]

Hmmm, I don't think it says that HFI is used by the scheduler, so without direct support of HFI in the scheduler, we will still have to wait for distro support for it.

I thought thread director is hardware based and HFI was needed to support it?

https://www.fudzilla.com/news/pc-hardware/54369-linux-kernel-will-support-intel-hardware-feedback

[Volta]

Microsoft with its deep pockets and tens of thousands of excellent programmers spent almost a year optimizing the Windows 11 scheduler for ADL and in Linux it will work right away by itself ... magically I presume. No tuning, no rules, no intelligence required, it'll just work.

Do you mean microsoft with their mediocre programmers and legacy OS:

I'm a developer in Windows and contribute to the NT kernel. (Proof: the SHA1 hash of revision Favorite Open/Closed Source Games of [Edit: filename redacted] is [Edit: hash redacted].) I'm posting through Tor for obvious reasons.

Windows is indeed slower than other operating systems in many scenarios, and the gap is worsening. The cause of the problem is social. There's almost none of the improvement for its own sake, for the sake of glory, that you see in the Linux world.

Granted, occasionally one sees naive people try to make things better. These people almost always fail. We can and do improve performance for specific scenarios that people with the ability to allocate resources believe impact business goals, but this work is Sisyphean. There's no formal or informal program of systemic performance improvement. We started caring about security because pre-SP3 Windows XP was an existential threat to the business. Our low performance is not an existential threat to the business.

See, component owners are generally openly hostile to outside patches: if you're a dev, accepting an outside patch makes your lead angry (due to the need to maintain this patch and to justify in in shiproom the unplanned design change), makes test angry (because test is on the hook for making sure the change doesn't break anything, and you just made work for them), and PM is angry (due to the schedule implications of code churn). There's just no incentive to accept changes from outside your own team. You can always find a reason to say "no", and you have very little incentive to say "yes".

There's also little incentive to create changes in the first place. On linux-kernel, if you improve the performance of directory traversal by a consistent 5%, you're praised and thanked. Here, if you do that and you're not on the object manager team, then even if you do get your code past the Ob owners and into the tree, your own management doesn't care. Yes, making a massive improvement will get you noticed by senior people and could be a boon for your career, but the improvement has to be very large to attract that kind of attention. Incremental improvements just annoy people and are, at best, neutral for your career. If you're unlucky and you tell your lead about how you improved performance of some other component on the system, he'll just ask you whether you can accelerate your bug glide.

Is it any wonder that people stop trying to do unplanned work after a little while?

Another reason for the quality gap is that that we've been having trouble keeping talented people. Google and other large Seattle-area companies keep poaching our best, most experienced developers, and we hire youths straight from college to replace them. You find SDEs and SDE IIs maintaining hugely import systems. These developers mean well and are usually adequately intelligent, but they don't understand why certain decisions were made, don't have a thorough understanding of the intricate details of how their systems work, and most importantly, don't want to change anything that already works.

These junior developers also have a tendency to make improvements to the system by implementing brand-new features instead of improving old ones. Look at recent Microsoft releases: we don't fix old features, but accrete new ones. New features help much more at review time than improvements to old ones.

(That's literally the explanation for PowerShell. Many of us wanted to improve cmd.exx, but couldn't.)

More examples:

We can't touch named pipes. Let's add %INTERNAL_NOTIFICATION_SYSTEM%! And let's make it inconsistent with virtually every other named NT primitive.
We can't expose %INTERNAL_NOTIFICATION_SYSTEM% to the rest of the world because we don't want to fill out paperwork and we're not losing sales because we only have 1990s-era Win32 APIs available publicly.
We can't touch DCOM. So we create another %C#_REMOTING_FLAVOR_OF_THE_WEEK%!
XNA. Need I say more?
Why would anyone need an archive format that supports files larger than 2GB?
Let's support symbolic links, but make sure that nobody can use them so we don't get blamed for security vulnerabilities (Great! Now we get to look sage and responsible!)
We can't touch Source Depot, so let's hack together SDX!
We can't touch SDX, so let's pretend for four releases that we're moving to TFS while not actually changing anything!
Oh god, the NTFS code is a purple opium-fueled Victorian horror novel that uses global recursive locks and SEH for flow control. Let's write ReFs instead. (And hey, let's start by copying and pasting the NTFS source code and removing half the features! Then let's add checksums, because checksums are cool, right, and now with checksums we're just as good as ZFS? Right? And who needs quotas anyway?)
We just can't be fucked to implement C11 support, and variadic templates were just too hard to implement in a year. (But ohmygosh we turned "^" into a reference-counted pointer operator. Oh, and what's a reference cycle?)

[birdie]

Volta continues to impress with uttering the things he absolutely doesn't know or understand.

Microsoft with its deep pockets and tens of thousands of excellent programmers spent almost a year optimizing the Windows 11 scheduler for ADL and in Linux it will work right away by itself ... magically I presume. No tuning, no rules, no intelligence required, it'll just work.

[NobodyXu]

It can:



https://www.kernel.org/doc/html/late...ails-for-linux

Hmmm, I don't think it says that HFI is used by the scheduler, so without direct support of HFI in the scheduler, we will still have to wait for distro support for it.

[NobodyXu]

I doubt that. In Windows something notifies the kernel which process is in foreground/background and the kernel then decides which cores this process could/should use. The Linux kernel has no notion of background/foreground processes, it doesn't know that your web browser is more important than some compilation/rendering/compute/etc. background task either. Moving these features into the kernel scheduler is a strict no-go.

It is indeed impossible for the kernel to know about foreground/background processes, but maybe we can use priority (nice value) + the type of the process as a way to differentiate interactive and batch processes?

Interactive processes often has a low nice value, batch processes often has a high nice value and sometimes set to SCHED_BATCH or SCHED_IDLE.

While it may not provide the best performance, it is at least better than not using HFI at all.

[Volta]

Can't the task scheduler in kernel directly take advantage of HFI?

It can:

The kernel or a userspace policy daemon can use these capabilities to modify task placement decisions. For instance, if either the performance or energy capabilities of a given logical processor becomes zero, it is an indication that the hardware recommends to the operating system to not schedule any tasks on that processor for performance or energy efficiency reasons, respectively.

https://www.kernel.org/doc/html/latest/x86/intel-hfi.html#implementation-details-for-linux

[birdie]

Can't the task scheduler in kernel directly take advantage of HFI?

I doubt that. In Windows something notifies the kernel which process is in foreground/background and the kernel then decides which cores this process could/should use. The Linux kernel has no notion of background/foreground processes, it doesn't know that your web browser is more important than some compilation/rendering/compute/etc. background task either. Moving these features into the kernel scheduler is a strict no-go.