Multi-Core, Multi-OS Scaling Performance

[baryluk]

So was PC-BSD devlopers notified about this important problem? It looks like not well tested scheduler on multicore.

[baryluk]

HT is a real nightmare. If you have lots of data to calculate and wouldn't even run into the cache hell, it is fine.

However if a high priority thread is the HT you're screwed. HT is basicaly a technique that lets a second thread utilize a part of the CPU that isn't being consumed by another thread.

Great for round robin scheduling and number crunching. For everything else... not so much...

For number crunching? I will say in most cases not. It is good for number crunching with irregular structure of memory access and working set of memory is bigger than L2 cache. Then using HT will help, as it will hide memory latencies. But for pure number crunching (like computing complicated function of 100 floating operations on array of 1000000 doubles, for example when solving PDE), it will not help that much (even if 1000000 doubles is 8MB - much more than L2), as cache prefetching and actual FPU computations are longer than memory transfer time. For simple computations, or irregular access yes - for example bigmultiuser databases, or raytracing with accelerations strctures (BVH or kd-tree) - it can give even 20-30% speed increase.

[baryluk]

Looks like kind of scheduler bug in PC-BSD. It cannot be so bad. Obviously wasn't tested well on multicore.

[V!NCENT]

Common wisdom in the sysadmin field about 6 years ago or so was that HT was a real mixed bag.

HT is a real nightmare. If you have lots of data to calculate and wouldn't even run into the cache hell, it is fine.

However if a high priority thread is the HT you're screwed. HT is basicaly a technique that lets a second thread utilize a part of the CPU that isn't being consumed by another thread.

Great for round robin scheduling and number crunching. For everything else... not so much...

[Garp]

Common wisdom in the sysadmin field about 6 years ago or so was that HT was a real mixed bag. At the time the HT thread shared the CPU cache with the main thread. For highly repetitive work on the same data, e.g. multimedia processing, that was fine. However if the HT thread picked up an instruction that didn't have data in cache it would flush and reload. That could end up wasting a lot of cycles and could noticeably slow things down. Safest was to assume that it would be bad, but if you could, do benchmarking. I haven't seen any technical details to suggest whether that weakness still remains or not.

[kebabbert]

Gah, the edit limit here is truly microscopic. Anyway.
The only way to really isolate that effect would of course be to run the linux distros through twice, with different schedulers ... but then it'd only be fair to do the same with PC-BSD (BSD vs the default ULE) and openindiana (if they have more than one). It would more than double the work, of course.

Solaris has several schedulers, that you can change on-the-fly without rebooting. Most probably OpenIndiana also has those schedulers. OpenIndiana is in alpha stage, and has not hit beta version yet. OpenIndiana is the fully open sourced version of OpenSolaris, so it should have every feature that OpenSolaris had (which also was in alpha version).

This year Solaris 11 will hit beta version (it is in alpha now), and later this year Solaris 11 will finally be done and released for production! Yey!

[dnebdal]

That would be a completely different test - of the schedulers. This was a test of the operating systems. A more useful one, in my opinion. Though i do agree that determing whether the FS was to blame might be nice.

Well, in this case just listing the respective schedulers would be nice - my point was that scheduler choice is as interesting in this benchmark as the filesystem used is for DB benchmarks.

[smitty3268]

Gah, the edit limit here is truly microscopic. Anyway.
The only way to really isolate that effect would of course be to run the linux distros through twice, with different schedulers ... but then it'd only be fair to do the same with PC-BSD (BSD vs the default ULE) and openindiana (if they have more than one). It would more than double the work, of course.

That would be a completely different test - of the schedulers. This was a test of the operating systems. A more useful one, in my opinion. Though i do agree that determing whether the FS was to blame might be nice.

[dnebdal]

It would be explanatory information, though - much like how the filesystem is often relevant when there are large gaps in database performance. (The different kernel versions here are of course a confounding effect - but it the two linuxes are also using different schedulers, that might be part of the explanation.)

Gah, the edit limit here is truly microscopic. Anyway.
The only way to really isolate that effect would of course be to run the linux distros through twice, with different schedulers ... but then it'd only be fair to do the same with PC-BSD (BSD vs the default ULE) and openindiana (if they have more than one). It would more than double the work, of course.

[dnebdal]

I don't see people asking for the scheduler information for PC-BSD or OpenIndiana. The scheduler decisions made by Ubuntu and Red Hat (CentOS, Fedora) are no different than the scheduler decisions made for other OSes. Only those who regularly hack and play with schedulers really care. I have never heard anyone say distro-X with CFQ's scalability sucks. It's invariable either the scheduler as the primary point of interest or the distribution.

If the schedulers were given a head-to-head, then people would say "well, the distribution or compiler choices make the benchmarks pointless". Of course we all know what happens when compilers are compared...

It would be explanatory information, though - much like how the filesystem is often relevant when there are large gaps in database performance. (The different kernel versions here are of course a confounding effect - but it the two linuxes are also using different schedulers, that might be part of the explanation.)