Tweet
Good time-of-day, everyone!
The vast majority of benchmarks published these days, in the mainstream, focuses
on pretty degenerate cases that most people won't even care about, most of the
time. Although, said media goes somewhat far enough on the success metric towards
convincing the public that's all the rage.
The problem is, the benchmarks published on the forefront (with notable exception
of games) are all quite scalable (sometimes perfectly), and one could say, almost
carefully oriented towards highlighting the strong side of very-many-core CPUs.
One could say -- hey, that's a good thing -- we now have proper competition (even
if those highly scalable benchmarks are at least somewhat misleading) -- and that
benefits all of the developers, too!
Well, if you look at the numbers for the staple of programmer's sustenance, the
venerable C family of languages -- most prominently the Linux kernel and Clang
compilation benchmarks, then indeed, that's hard to dispute. Even Linus himself
switched to a ridiculously multi-core Ryzen machine recently, and I'm sure it
wasn't on impulse.
However, as much fun as that is (and hey, developers love fun!), the C language
family isn't particularly, well, forward-looking, in terms of progress.
A number of different modes of linguistic expression exist, all spanning different
parts of the multi-dimensional spectrum between performance, interactivity and,
importantly, assurance.
A particular feature of "grown-up" languages, that are also not hard-core
system-oriented, is relieving programmers from manual memory management -- that is
use of automatic garbage collection. Mature implementations, of course, have
highly optimised, parallel, cache-friendly, low-latency and otherwise many
different forms of those garbage collectors.
And that appears to open a whole new can of worms, when it comes to benchmarking
modern systems for their ability to cater to the developer needs.
So, there -- we live in a world, where there is a number of widely-used
higher-level languages, that have memory usage patterns nontrivially-different
from traditional C-based languages -- and that difference translates to rather
surprising compiler performance characteristics.
What is important, though, is that's where the developer world moves to: Java, C#
(or, rather, JVM and CLR -- with their plethora of languages) and yes, even such
odd things like Haskell. Indeed C and C++ are losing their relevance in many ways
because of them (although that's also partly driven by Rust, a non-GC language).
Oh boy, was that a long introduction! The purpose, though, was to motivate a
certain rethinking of compilation benchmarking -- to better reflect what the
non-C-based developer demographics might be interested in.
So, long story short -- despite the fairly obvious (and tremendous) improvement
that the new wave of highly-cored offerings bring to compilation of C/C++
projects, the same improvement doesn't necessarily translate to compilation of
GC-based languages.
Currently, we don't have very many such benchmarks on Openbenchmarking.org: we
used to have one for the Java Gradle build, and we have recently gained the
Haskell/GHC-based compilation of the clash-lang.org's Haskell-to-VHDL compiler
(heh).
The story, so far, seems to be that the high core counts don't translate to
performance gains all that well -- instead, we see much better correlation with
lower memory latencies and higher core frequencies.
A particularly enlightening synthetic benchmark in this respect is Google's
multichase: https://openbenchmarking.org/test/pts/multichase. As can be seen in
the second set of results, i9-10900K is reigning supreme.
The results translate pretty well to both the Java Gradle build and the Haskell/GHC
'clash-lang' benchmark:
- https://openbenchmarking.org/showdow...va-gradle-perf
- https://openbenchmarking.org/test/pts/build-clash-1.0.0
So, what are the conclusions?
Maybe the extreme multi-core isn't the be-all, end-all we sought for, even for
people who we thought would traditionally gain from high parallelism --
developers. And especially so, developers using more modern, higher-level
languages with automatic memory management.
First is -- it'd be nice to see more non-C/C++ compilation benchmarks for common
GC-based languages. That's pretty much a given.
Second, as much as that's counterintuitive in today's IT landscape, it'd be still
interesting to see performance reviews for some Intel CPU's as well.
For instance, there's the Xeon E-2288G (which is a tuned version of i9-9900K) and
seems to do quite well on the Java Gradle build benchmark, and the upcoming
i9-10900K-derived Xeon W-1290P, which is already being made available for
preordering.
And, of course, the upcoming Zen 3!
The vast majority of benchmarks published these days, in the mainstream, focuses
on pretty degenerate cases that most people won't even care about, most of the
time. Although, said media goes somewhat far enough on the success metric towards
convincing the public that's all the rage.
The problem is, the benchmarks published on the forefront (with notable exception
of games) are all quite scalable (sometimes perfectly), and one could say, almost
carefully oriented towards highlighting the strong side of very-many-core CPUs.
One could say -- hey, that's a good thing -- we now have proper competition (even
if those highly scalable benchmarks are at least somewhat misleading) -- and that
benefits all of the developers, too!
Well, if you look at the numbers for the staple of programmer's sustenance, the
venerable C family of languages -- most prominently the Linux kernel and Clang
compilation benchmarks, then indeed, that's hard to dispute. Even Linus himself
switched to a ridiculously multi-core Ryzen machine recently, and I'm sure it
wasn't on impulse.
However, as much fun as that is (and hey, developers love fun!), the C language
family isn't particularly, well, forward-looking, in terms of progress.
A number of different modes of linguistic expression exist, all spanning different
parts of the multi-dimensional spectrum between performance, interactivity and,
importantly, assurance.
A particular feature of "grown-up" languages, that are also not hard-core
system-oriented, is relieving programmers from manual memory management -- that is
use of automatic garbage collection. Mature implementations, of course, have
highly optimised, parallel, cache-friendly, low-latency and otherwise many
different forms of those garbage collectors.
And that appears to open a whole new can of worms, when it comes to benchmarking
modern systems for their ability to cater to the developer needs.
So, there -- we live in a world, where there is a number of widely-used
higher-level languages, that have memory usage patterns nontrivially-different
from traditional C-based languages -- and that difference translates to rather
surprising compiler performance characteristics.
What is important, though, is that's where the developer world moves to: Java, C#
(or, rather, JVM and CLR -- with their plethora of languages) and yes, even such
odd things like Haskell. Indeed C and C++ are losing their relevance in many ways
because of them (although that's also partly driven by Rust, a non-GC language).
Oh boy, was that a long introduction! The purpose, though, was to motivate a
certain rethinking of compilation benchmarking -- to better reflect what the
non-C-based developer demographics might be interested in.
So, long story short -- despite the fairly obvious (and tremendous) improvement
that the new wave of highly-cored offerings bring to compilation of C/C++
projects, the same improvement doesn't necessarily translate to compilation of
GC-based languages.
Currently, we don't have very many such benchmarks on Openbenchmarking.org: we
used to have one for the Java Gradle build, and we have recently gained the
Haskell/GHC-based compilation of the clash-lang.org's Haskell-to-VHDL compiler
(heh).
The story, so far, seems to be that the high core counts don't translate to
performance gains all that well -- instead, we see much better correlation with
lower memory latencies and higher core frequencies.
A particularly enlightening synthetic benchmark in this respect is Google's
multichase: https://openbenchmarking.org/test/pts/multichase. As can be seen in
the second set of results, i9-10900K is reigning supreme.
The results translate pretty well to both the Java Gradle build and the Haskell/GHC
'clash-lang' benchmark:
- https://openbenchmarking.org/showdow...va-gradle-perf
- https://openbenchmarking.org/test/pts/build-clash-1.0.0
So, what are the conclusions?
Maybe the extreme multi-core isn't the be-all, end-all we sought for, even for
people who we thought would traditionally gain from high parallelism --
developers. And especially so, developers using more modern, higher-level
languages with automatic memory management.
First is -- it'd be nice to see more non-C/C++ compilation benchmarks for common
GC-based languages. That's pretty much a given.
Second, as much as that's counterintuitive in today's IT landscape, it'd be still
interesting to see performance reviews for some Intel CPU's as well.
For instance, there's the Xeon E-2288G (which is a tuned version of i9-9900K) and
seems to do quite well on the Java Gradle build benchmark, and the upcoming
i9-10900K-derived Xeon W-1290P, which is already being made available for
preordering.
And, of course, the upcoming Zen 3!
Comment