The Effort To Parallelize GCC With Threads Is Starting To Take Shape

[phoronix]

Phoronix: The Effort To Parallelize GCC With Threads Is Starting To Take Shape

Back in April we wrote about a proposal for providing better parallelization within GCC itself to address use-cases such as very large source files. That effort was accepted as part of this year's Google Summer of Code and the student developer pursing this parallelization with threads has issued his first progress report...

http://www.phoronix.com/scan.php?page=news_item&px=Parallelize-GCC-With-Threads

[wizard69]

I’m all for speed ups but how much parallelization can one achieve in a single source file? Is the goal to compile functions (or other well defined sections of code) in parallel?

[archkde]

I hope this change will not negatively affect reproducibility.

[hubicka]

I’m all for speed ups but how much parallelization can one achieve in a single source file? Is the goal to compile functions (or other well defined sections of code) in parallel?

With link-time optimization parallelism is important since you no longer compile single source file
However even w/o LTO parallelising single file compilation should be a win. On 256 thread box I am now using for testing builds are typically waiting for compilation of one or two largest files in the source tree to finish. In the GCC case those are auto-generated files that does pattern matching and also libasan which has many instrumentation functions in one source.

[caligula]

I’m all for speed ups but how much parallelization can one achieve in a single source file? Is the goal to compile functions (or other well defined sections of code) in parallel?

The problem with parallel makefiles is that you need to serialize to a file. If the compiler works in a multi-threaded way, you can share preloaded stuff from memory. That's one way to make it faster even if you can't utilize more parallelism than before.

[schmidtbag]

Although there's nothing especially bad about this change, I would think in most cases you're better off just compiling each of your source files in a separate thread. I'm sure you'd see better overall performance that way.

[oiaohm]

Although there's nothing especially bad about this change, I would think in most cases you're better off just compiling each of your source files in a separate thread. I'm sure you'd see better overall performance that way.

Yes you start off with all you source files as single parts. Its once all of them are glued into 1 huge block as LTO process at this point multi threading the optimisations will be a god send. Very large source files and very large LTOs are what will gain most from this.

Please remember CPUs are not getting that much faster. Either we will have to update software to multi thread well. Or have kernel/cpu auto multi thread our programs to get more speed. Really after the mess of speculative execution with hyper-threading as a 2 threads on a single core I really don't want to think the mess a cpu could make taking a single threaded program and breaking it up to run in like 128 cores.

[andreano]

This should make unity builds (compiling everyting as one compilation unit, as a poor man's link time optimization) less painful. Meson has an option for it.

I think Cargo (the Rust buildsystem) is basically doing unity builds all the time – it just recently got support for incremental builds (!) and it also parallelizes beyond the number of files, though splitting up big files still speeds up compilation in my experience.

If modules get accepted into C++20, then that's goodbye to separate compilation units in the traditional sense, also for C++.

[xorbe]

This isn't going to help build farms that already run lots of gcc at a time per socket.

[Thaodan]

I hope memory usage while linking with PGO and LTO gets better.
Firefox hits 20GB and more at its peak while linking.