Benchmarking Mozilla's Firefox Performance Over The Past Two Years

[microcode]

sync, pocket, "premium", maybe vpn (in testing?) and I guess a party or two, ... :-/

At least they have fuzzy microphones and progressive furniture for their events.

[linuxgeex]

I'm glad to see Firefox ESR (based on 60.8) performance isn't falling behind the leading Firefox version. ESR needs to be supported by web developers because it's an official release, often used by businesses and by users who make use of extensions which may not have been updated to run with the latest release of Firefox, but who want the latest security patches. TorBrowser is also based on it. Anyhow, it's nice to see that Mozilla chose a reasonably performant long-stable release for ESR.

[linuxgeex]

How you came to this conclusion from looking at the graphs ?
They switched to clang in firefox 64, when i compare firefox 64 to 63, i don't see anything that stand.

Another thing, they didn't jump from GCC to MSVC, MSVC doesn't work in mac or linux, they had been using MSVC to build for windows, GCC to build for linux and clang to build for mac before jumping to clang in all platforms.

Fully clang'ing the project is a smart move since WASM uses clang+LLVM. Dumping GCC/MSVC to adopt a single compiler infrastructure lowers the project management complexity, which allows them to focus on more beneficial areas of development than configuration management and 3 compilers' politics...

[cb88]

The other consideration is Firefox probably isn't using WebRender yet in any of these benchmarks... so most of the new development isn't visible or is hidden by the benchmark... benchmarks tend to focus on one small area of the of the software that isn't always relevant to actual performance.

[kozman]

How you came to this conclusion from looking at the graphs ?
They switched to clang in firefox 64, when i compare firefox 64 to 63, i don't see anything that stand.

Another thing, they didn't jump from GCC to MSVC, MSVC doesn't work in mac or linux, they had been using MSVC to build for windows, GCC to build for linux and clang to build for mac before jumping to clang in all platforms.

Well, not so much a conclusion but more a thought in my head about what major thing had changed from the early 60 series of FF to present and perhaps why some of the early 60 series FF's were more performant than the mid to late FF releases based on the graphs and what had change in those later FF releases. Namely Clang. Not all the graphs show 60 series FF beating current FF though so not making a blanket statement or drawing a conclusion.

[kozman]

Fully clang'ing the project is a smart move since WASM uses clang+LLVM. Dumping GCC/MSVC to adopt a single compiler infrastructure lowers the project management complexity, which allows them to focus on more beneficial areas of development than configuration management and 3 compilers' politics...

Is that what it was, to have one compiler to, proverbially speaking, rule them all? I agree, it would make sense to do that to lower project complexity. I'm not a coder or project manager but on it's face, working smarter and not harder seems like a no brainer. I don't know what the trade off was in terms of what GCC or VC brought to the table in terms of perf. For my needs, FF seems fine. As FF/Clang mature and bugs in them are fixed, I'd hope performance increases. Time will tell as will user adoption or defection.

[finalzone]

Firefox of Fedora is built with gcc.

[hotaru]

without WebRender, Firefox is absolutely terrible on Android. even the simplest animations look like a slideshow and will drain a typical smartphone battery from 100% to 0% in less than an hour. turn on webrender, and suddenly performance is very close to Chrome.

most desktops have more powerful CPUs that can muscle through simple animations at 40+ fps, but still benefit a lot from having WebRender turned on.

[Ray_o]

Well, not so much a conclusion but more a thought in my head about what major thing had changed from the early 60 series of FF to present and perhaps why some of the early 60 series FF's were more performant than the mid to late FF releases based on the graphs and what had change in those later FF releases. Namely Clang. Not all the graphs show 60 series FF beating current FF though so not making a blanket statement or drawing a conclusion.

The only releases that matter here is 63 vs 64. so if there is a performance issue because of clang switch you would see it in 64.
let say clang switch led to a 5% reduction in performance in one benchmark, it wouldn't increase to 7% in Firefox 65 then to 10% in Firefox 66 as long as you are using the same version of clang and using the same compiler flags.

So why Firefox is doing worse in some benchmarks, code changes could be the reason.

[linuxgeex]

Is that what it was, to have one compiler to, proverbially speaking, rule them all? I agree, it would make sense to do that to lower project complexity. I'm not a coder or project manager but on it's face, working smarter and not harder seems like a no brainer. I don't know what the trade off was in terms of what GCC or VC brought to the table in terms of perf. For my needs, FF seems fine. As FF/Clang mature and bugs in them are fixed, I'd hope performance increases. Time will tell as will user adoption or defection.

For the most part clang/LLVM and GCC are on par with each other for performance and features these days. Cohort benchmarks of GCC 9 vs Clang/LLVM 9 put them within the margin of error of each other, but with Clang/LLVM compiling significantly faster. There are some specific cases where each leads. Here's Phoronix's results from July pitting GCC against Clang... on Cascade Lake Clang appears to have a temporary advantage:

MSVC has seriously fallen behind GCC and Clang. ICC and AOC lead, but only for scientific workloads.