Benchmarking Mercury As The "Fastest Firefox Fork" With AVX, AES, LTO + PGO

[guglovich]

I've wanted a test like this for a long time. Now I'm interested in Chromium vs Thorium.

[stormcrow]

Not when it's built by Debian's maintainers as that build definitely sucks as it much slower than the build built by Mozilla.

Debian uses Firefox ESR anyway. It's a compatibility thing that's perfectly understandable. I've seen new mainstream versions of Firefox break websites that were working fine the version/day before (and no, Chrome is no better at this). If you're dependent on a browser to do work, school, finances, etc having it maintain a steady compatibility profile long term is far more important than half second differences in Javascript performance.

However, if you're one of the many people that want the rapid release cycle mainstream Firefox, then that's what flatpak is for and it works swimmingly on Debian.

[yoshi314]

I don't have a horse in the race as I don't use Firefox or Mercury, but continuing to use a bloated beast that labors under its own weight like Ubuntu for benchmarking browsers seems like a strange choice. I don't know if the results are applicable to anything I would use.

doesn't matter really. as long as you use the same environment for each benchmark.

more performant application will still be a bit faster in that setup.

[Anux]

Doesn't Firefox use LTO+PGO as well?

I think so, at least I read about it on phoronix a while ago. Ubuntu also uses the official Firefox build in a snap if I recall it right, so this should already be the best version of each. The only benchmark where mercury could win would be startup time then ...

I just recently did some testing of waterfox, firefox and mercury myself and came to the same conclusions. Mercury is at best as fast as firefox while waterfox is much slower.

Maybe compiling mercury with "native" for your own system could improve things, but that would also hold true for FF.

[hkupty]

I have to say I kind of feel betrayed by mozilla after they silently dropped browser.html and their rust efforts... This fork kind of gave me a glimpse of hope, but that fade away quickly after looking at the numbers...

[user1]

Debian uses Firefox ESR anyway. It's a compatibility thing that's perfectly understandable. I've seen new mainstream versions of Firefox break websites that were working fine the version/day before (and no, Chrome is no better at this). If you're dependent on a browser to do work, school, finances, etc having it maintain a steady compatibility profile long term is far more important than half second differences in Javascript performance.

However, if you're one of the many people that want the rapid release cycle mainstream Firefox, then that's what flatpak is for and it works swimmingly on Debian.

Yes, it's true that Firefox ESR will give you better stability like any other LTS software. But no matter if it's Firefox ESR or the latest stable release, I don't understand why do Debian, Fedora and probably other distros refuse to build Firefox with Clang and insist on continue building it with gcc. I mean that's the very reason official Firefox builds by Mozilla are slightly faster than distro builds. Because with Clang you can have some browser related compiler optimizations that are impossible to achieve with gcc. I heard that Chrome is also built with Clang for this reason.

[avis]

Yes, it's true that Firefox ESR will give you better stability like any other LTS software. But no matter if it's Firefox ESR or the latest stable release, I don't understand why do Debian, Fedora and probably other distros refuse to build Firefox with Clang and insist on continue building it with gcc. I mean that's the very reason official Firefox builds by Mozilla are slightly faster than distro builds. Because with Clang you can have some browser related compiler optimizations that are impossible to achieve with gcc. I heard that Chrome is also built with Clang for this reason.

Octane Score

Debian Firefox 102.14.0esr (64-bit): 35801
Official Firefox 102.14.0esr (64-bit): 38167

Yeah, you're onto something that's why I've always used the official build. Though 8-10% performance difference is not something I'd bother about if I were a hardcore Debian lover.

Meanwhile Google Chrome Version 116.0.5845.110 (Official Build) (64-bit): 84693

Firefox is so slow.

[Anux]

Though 8-10% performance difference is not something I'd bother about if I were a hardcore Debian lover.

You could never notice without measuring it, in a blind test you wouldn't be able to tell which is which.

Meanwhile Google Chrome Version 116.0.5845.110 (Official Build) (64-bit): 84693

That might be noticeable but only if you do compute intensive stuff like benchmarking or FPS in a 3D browsergame. For normal surfing the most time is spend waiting for your input and the internet connection, a few ms more or less are hardly noticeable in this scenario.

[ultimA]

The whole premise of AVX+PGO+LTO in a browser bringing 8-20% sounds unrealistic for me.

Javascript benchmarks cannot take advantage of AVX, and while AVX could in theory be a big benefit for graphics operations, most of those are usually already gpu-accelerated in modern browser versions.
LTO and PGO could help with Javascript and in general at other places where AVX doesn't make sense, but they rarely bring more than just 2-5%, so this up to 20% performance seems very dubious. Even if there is a code piece where they could bring "8-20%" (such as hashing or cryptography), that won't change the overall user experience as these are not the bottleneck. I'm not saying PGO and LTO are useless, every percent matters, but this here smells like snake-oil and over-hype.

But maybe most importantly, Firefox uses JIT to execute Javascript, so the speed of executing JS will not depend on GCC or Clang at all, but instead will solely depend on what bytecode the JS is compiled into by Firefox at runtime. GCC/Clang optimizations matter at most for how fast Firefox can start executing JS, but that is not what benchmarks usually measure.

And Michael's benchmarks in the current article perfectly demonstrate these points.

[user1]

You could never notice without measuring it, in a blind test you wouldn't be able to tell which is which.

It might be anecdotal evidence, but I remember once started using the Fedora Firefox package and immediately noticed that YouTube and some other websites were loading slower than usual. Then I switched to the official Mozilla build from the tarball and noticed that the slow loading issue is gone. It was the same version of Firefox at that time.