Fedora Decides After All To Allow Default Compiler Flag To Help Debugging/Profiling

[phoronix]

Phoronix: Fedora Decides After All To Allow Default Compiler Flag To Help Debugging/Profiling

The past several months saw much discussion over a proposal to add "-fno-omit-frame-pointer" as a default compiler flag to Fedora Linux that would improve profiling/debugging but with possible performance implications that can vary based on the application/workload. While just over one month ago FESCo rejected that change, they re-voted today and decided after all to allow this change to happen but to ensure that packages can easily opt-out if they find performance regressions. By Fedora 40 they will also re-visit the matter to determine if the benefits and performance costs are justified...

https://www.phoronix.com/news/F38-fno-omit-frame-pointer

[elduderino]

If anyone is interested in the why's about this, I wrote up something last week that explains all the crazy amount of edge cases that have to be handled to get really useful eBPF tooling to work in the absence of frame pointers. The ones out there today are not robust and would fail to handle a significant portion of how Fedora is used today especially at the rate (thousands of samples per second) that are done in system-wide profiling.

Linux has been the odd-penguin in this one, with macOS and Windows both still using frame-pointers, in part ensure good profiling results in the presence of many of the same types of issues noted in the blog post.

https://blogs.gnome.org/chergert/2022/12/31/frame-pointers-and-other-practical-near-term-solutions/

[MadCatX]

At least the knob seems to be an opt-in rather than opt-out for now. If this is going to get flipped on for all system packages, not opting in will probably not help very much though.

[RahulSundaram]

If anyone is interested in the why's about this, I wrote up something last week that explains all the crazy amount of edge cases that have to be handled to get really useful eBPF tooling to work in the absence of frame pointers. The ones out there today are not robust and would fail to handle a significant portion of how Fedora is used today especially at the rate (thousands of samples per second) that are done in system-wide profiling.

Linux has been the odd-penguin in this one, with macOS and Windows both still using frame-pointers, in part ensure good profiling results in the presence of many of the same types of issues noted in the blog post.

https://blogs.gnome.org/chergert/202...erm-solutions/

I read this write up earlier. It's detailed and helpful. Thanks

[avis]

If anyone is interested in the why's about this, I wrote up something last week that explains all the crazy amount of edge cases that have to be handled to get really useful eBPF tooling to work in the absence of frame pointers. The ones out there today are not robust and would fail to handle a significant portion of how Fedora is used today especially at the rate (thousands of samples per second) that are done in system-wide profiling.

Linux has been the odd-penguin in this one, with macOS and Windows both still using frame-pointers, in part ensure good profiling results in the presence of many of the same types of issues noted in the blog post.

https://blogs.gnome.org/chergert/202...erm-solutions/

I only care about performance and a little less about memory consumption. If this flag doesn't affect performance and increases memory consumption only slightly, whatever.

[Linuxhippy]

hopefully it will be applied / monitored carefully. fedora is already one of the slower distributions with very conservative compiler/build-settings, additional options with the potential to slow it down further won't help for sure. just compare the fedora-built firefox with the official builds, not to even mention what clear linux has shown to be possible over the last few years.

[skeevy420]

I only care about performance and a little less about memory consumption. If this flag doesn't affect performance and increases memory consumption only slightly, whatever.

It can effect performance negatively according to the benchmarks 6 months back. What I wonder is why they can't have release and debug repositories or why they can't glue release and debug together with something like FatElf?

Normally I'm on that same performance>memory boat as yourself. This doesn't look to be it.

Edit: Although, for some irony, the short-term hit for performance might mean a long-term increase in performance assuming they get good information and make good use of it as well as flag things that have notable drops in performance.

[markg85]

I'm using fedora on - specifically - VPS nodes.
I don't care about debug-ability. I do care about speed optimized packages and - preferably - the smallest packages possible (speed and size don't always go hand-in-hand).

I've already been annoyed by fedora's docker vs podman annoyances.
Polkit plagued me for a long time too (though not an issue right now).

I suppose the redhat derived linux distributions are now also going to have this "-fno-omit-frame-pointer" nonsense in production or have it already?
Guess i'm off to Arch on the VPS side of things too. It's a fine OS if you keep it up to date but still a bit risky...

[skeevy420]

Guess i'm off to Arch on the VPS side of things too. It's a fine OS if you keep it up to date but still a bit risky...

CachyOS is pretty nice if you want Arch with v3 optimizations and some other QOL and performance enhancements. I've been running it for almost a month now and so far so good. ptr1337 is behind it.

[elduderino]

It can effect performance negatively according to the benchmarks 6 months back. What I wonder is why they can't have release and debug repositories or why they can't glue release and debug together with something like FatElf?

Normally I'm on that same performance>memory boat as yourself. This doesn't look to be it.

FatELF would be *considerably* larger foot print on disk as you'd basically more than double every binary (duplicated+extra instructions, duplicated eh_frame/debug_frame/etc). But beyond that, you simply can't get around how much more memory it takes to upload duplicated unwind tables to the kernel when you need to profile vs frame-pointers. Having them inline in the instruction cache is the fastest way to get both unwinding and not tip over faulting when profiling under memory pressure.

As for performance, if you don't have register pressure, then it's not going to affect you to keep using %rsp. If you do have register pressure, there is a chance you're already doing a lot to work around these sort of things already.

As for release vs debug, it's important tenant to be triaging the same code people are using in production. Particularly if you want people experiencing the issue to contribute to the problem solving effort.