Red Hat Developing "eu-stacktrace" For Profiling Without Frame Pointers

[phoronix]

Phoronix: Red Hat Developing "eu-stacktrace" For Profiling Without Frame Pointers

While last year we saw Fedora to no longer omit the frame pointer to help in debugging/profiling Fedora packages and Ubuntu 24.04 LTS also enabled frame pointers for better debugging/profiling, among other distributions, there is the known performance implications of no longer omitting the frame pointer. But now in aiming to make the best of both worlds, it turns out Red Hat has been developing eu-stracktrace as a new means of profiling without relying on frame pointers...

https://www.phoronix.com/news/Red-Hat-eu-stacktrace

[EphemeralEft]

At first I thought this was about the GDPR.

[Kjell]

Fedora users cried their eyes out over fno-omit-frame-pointer

Arch Linux sneaked it in a couple of weeks ago and no one bats an eye

https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/commit/a84bd5de18533069262a2b402f6aba088d218c1c

[dralley]

Fedora users cried their eyes out over fno-omit-frame-pointer

Arch Linux sneaked it in a couple of weeks ago and no one bats an eye

https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/commit/a84bd5de18533069262a2b402f6aba088d218c1c

And Ubuntu 6 months ago https://ubuntu.com/blog/ubuntu-perfo...ers-by-default

Because frame pointers are a nonissue. The performance regression rounds to zero 90% of the time, and most of the regressions that people did note were easily fixed more than a year ago. The doomers were completely wrong about the significance of any regressions, meanwhile the benefits on the other hand are pretty significant. The xz-utils backdoor was caught partly due to frame pointers. Lots of performance improvements have been made based on profiling with frame pointers.

[Kjell]

The performance regression rounds to zero 90% of the time

There's no point in shipping binaries with debug flags for 90% of end-users, especially if you care about latency.

This is precisely why Gentoo is the most stable and snappiest Desktop Linux distribution out there - there's no debug flags by default.

Here's a quick latency test:

1.22user 0.27system 0:01.48elapsed 100%CPU (0avgtext+0avgdata 3840maxresident)k
1.19user 0.29system 0:01.47elapsed 100%CPU (0avgtext+0avgdata 3840maxresident)k
1.20user 0.29system 0:01.48elapsed 100%CPU (0avgtext+0avgdata 3584maxresident)k

0.40user 0.15system 0:00.54elapsed 102%CPU (0avgtext+0avgdata 3336maxresident)k
0.40user 0.15system 0:00.54elapsed 102%CPU (0avgtext+0avgdata 3336maxresident)k
0.40user 0.15system 0:00.54elapsed 102%CPU (0avgtext+0avgdata 3336maxresident)k

[dralley]

There's no point in shipping binaries with debug flags for 90% of end-users, especially if you care about latency.

This is precisely why Gentoo is the most stable and snappiest Desktop Linux distribution out there - there's no debug flags by default.

Here's a quick latency test:

This is a completely nonsense benchmark. Besides the fact that I have basically no idea what you even tested because you didn't provide it (I'm going to assume it was the startup time of some program) things like "what scheduler is being used" and "does my kernel already have the pages in memory" are going to have orders of magnitude more impact on that than an extra two or three simple instructions per function call.

Also frame pointers are only one of, probably, dozens of different compile flag differences between the two. This is not a serious comparison.

[Kjell]

I'm going to assume it was the startup time of some program

It's a nifty test to see how quickly Bash can print lines in TTY

$ echo '#!/bin/bash
for i in $(seq 1 1000);
do bash -c ":" ;
done' > bash_benchmark_script.sh
$ /usr/bin/time bash bash_benchmark_script.sh

Debian has a unfair advantage in this test since it uses PGO:

0.34user 0.13system 0:00.46elapsed 102%CPU (0avgtext+0avgdata 3072maxresident)k
0.36user 0.12system 0:00.48elapsed 102%CPU (0avgtext+0avgdata 3072maxresident)k
0.36user 0.11system 0:00.46elapsed 103%CPU (0avgtext+0avgdata 3072maxresident)k

0.39user 0.12system 0:00.50elapsed 102%CPU (0avgtext+0avgdata 3164maxresident)k
0.39user 0.13system 0:00.50elapsed 103%CPU (0avgtext+0avgdata 3164maxresident)k
0.37user 0.14system 0:00.50elapsed 102%CPU (0avgtext+0avgdata 3332maxresident)k

0.55user 0.17system 0:00.70elapsed 102%CPU (0avgtext+0avgdata 2816maxresident)k
0.53user 0.17system 0:00.69elapsed 102%CPU (0avgtext+0avgdata 2816maxresident)k
0.55user 0.18system 0:00.72elapsed 102%CPU (0avgtext+0avgdata 2816maxresident)k

[dralley]

Debian is by far fastest out of the box (due to PGO):

Cool, so like I said, this comparison has nothing to do with frame pointers, and proves nothing.

[Kjell]

Cool, so like I said, this comparison has nothing to do with frame pointers, and proves nothing.

Correct, my point remains the same:
Debug flags don't belong in production binaries

Anecdotally I find this test awfully accurate when comparing responsiveness of different distros
That's coming from someone who used Linux since 2009

[illwieckz]

The Orbit profiler already knows how to profile without frame pointers, and this tool is truly awesome:

https://github.com/google/orbit

Though having more tools able to do this would be good.