Some Additional Chrome vs. Firefox Benchmarks With WebRender, 67 Beta / 68 Alpha

[Michael_S]

We need test results for Firefox on Windows with NVIDIA graphics to determine the actualperformance impact of WebRender, since this is the configuration Mozilla develops WebRender for.

There must be other Windows-specific engineering work done by Mozilla for Firefox. These benchmark numbers are a few months out of date, but on my hardware the MotionMark (Animation testing) benchmark on Chrome had a more than 400% advantage on Linux regardless of Webrender. On Windows on the same benchmark, Chrome had only (for some odd definition of 'only') a 70% advantage.

I don't have Nvidia graphics.

On the other hand, the Speedometer benchmark probably comes closer to a real world comparison of speed that a end user would notice. On that one Chrome has a 20% advantage on Windows but Firefox had a 10% lead on Linux.

[kozman]

Man, this is not good. I'm a big fan of Mozilla but, man.....<sigh>. I would like to see results on Windows but I would like to point you guys to http://www.speed-battle.com. Look at the 2019 results for Windows 7 6.1. See all those test results between 64, 65 and 66 in the many thousands? That's me running a simple non-stessing bench test on an old Optiplex 7020. 66 has actually *dropped* in performance compared to 65. I don't know how much Rust is helping FF's cause but WTF were they thinking when they switched to LLVM from VS? It's just a turd. I mean, they *just* switched to VS a few years ago from GCC. Pick a friggin' compiler and stick with it!

FF 66 was build against an early LLVM 8.0 RC (I think) but seeing as how 8.0 final was just recently released after 66 went final, I doubt it would make any more perf improvement. I agree FF doesn't get as much love as in Windows but it has made a lot of strides since 65 but I sometimes feel like they're just chasing shadows or something.

[Michael]

Michael

Honestly I've never heard of this benchmark.

Why haven't you checked the performance in JetStream (ex SunSpider), Octane, and Kraken? What about the Windows versions of these web browsers?

I've also found these tests:

http://www.kevs3d.co.uk/dev/canvasmark/ (for me Chrome is twice as fast as Firefox in this test).
https://testdrive-archive.azurewebsi...ance/fishbowl/ (test with 2000 fish)
https://testdrive-archive.azurewebsi...ce/chalkboard/ (this one is painfully slow in Firefox)

https://web.basemark.com/

http://webvizbench.com/Default.htm?skipdetect (doesn't properly work on Firefox for Linux - no video, still Chrome is thrice as fast)

Basemark is included as are others. Are you referring to 'Selenium'? It's simply a WebDriver for allowing the automation of the browser tests.. As you can see from the graphs, Basemark, Octane, and others are what is being tested.

Jetstream now added, the URL wasn't working a few days ago. https://openbenchmarking.org/test/system/selenium

And for those not familiar with Selenium: https://www.seleniumhq.org/ after learning its interfaces, liking it quite a lot for automating all the browser benchmarks from Python.

[juno]

I haven't checked if it's applicable here but the actual rendering (and not js-execution) bound benchmarks could suffer from the same issue Firefox has with WebGL on Linux: copying back every frame before compositing.

[g3wcm2V8uqwR]

I haven't checked if it's applicable here but the actual rendering (and not js-execution) bound benchmarks could suffer from the same issue Firefox has with WebGL on Linux: copying back every frame before compositing.

Are you referring to the fact that Chromium gets more FPS because it skips (rendering) frames when it detects latency? as per https://bugzilla.mozilla.org/show_bug.cgi?id=611596#c26

[Cape]

Let's wait and see what excuses Google haters will come up with this time around...

From a technical standpoint, chrome is a wonderful piece of engineering. And it's got plenty room for improvement as well.

And you bet it does! It's the most used browser on the planet by far, developed from the most rich and successful company of the modern era, who has 98,000 - well payed employee in CA alone!

I wouldn't expect nothing less from this gigantic pile of financial/political admixture!

[cb88]

Man, this is not good. I'm a big fan of Mozilla but, man.....<sigh>. I would like to see results on Windows but I would like to point you guys to http://www.speed-battle.com. Look at the 2019 results for Windows 7 6.1. See all those test results between 64, 65 and 66 in the many thousands? That's me running a simple non-stessing bench test on an old Optiplex 7020. 66 has actually *dropped* in performance compared to 65. I don't know how much Rust is helping FF's cause but WTF were they thinking when they switched to LLVM from VS? It's just a turd. I mean, they *just* switched to VS a few years ago from GCC. Pick a friggin' compiler and stick with it!

FF 66 was build against an early LLVM 8.0 RC (I think) but seeing as how 8.0 final was just recently released after 66 went final, I doubt it would make any more perf improvement. I agree FF doesn't get as much love as in Windows but it has made a lot of strides since 65 but I sometimes feel like they're just chasing shadows or something.

If you are such a big fan stop being ignorant. LLVM is faster than gcc or VS, in many cases with FireFox as it enables optimizations that the other two can't do yet. LLVM also has superior MSVC compatiblity compared to GCC. See here for explination https://glandium.org/blog/?p=3888

Also WebRender isn't fully utilized under Linux yet, as using the GPU for acceleration isn't enabled by default see here https://bugzilla.mozilla.org/show_bug.cgi?id=1491303 , But the fact is that FireFox has not had acceleration on Linux in eons, so the fact that it is being worked on now is encouraging.

[treba]

Also WebRender isn't fully utilized under Linux yet, as using the GPU for acceleration isn't enabled by default

It is. If you were to run Webrender on software it would be terribly slow. The old 'hardware acceleration' via 'layers.acceleration.force-enabled' is replaced by Webrender, which is why you don't need to enable both to have Webrender running. Overall, we now have 'Basic', 'OpenGL' and 'Webrender' options.

[kozman]

If you are such a big fan stop being ignorant. LLVM is faster than gcc or VS, in many cases with FireFox as it enables optimizations that the other two can't do yet. LLVM also has superior MSVC compatiblity compared to GCC. See here for explination https://glandium.org/blog/?p=3888

Also WebRender isn't fully utilized under Linux yet, as using the GPU for acceleration isn't enabled by default see here https://bugzilla.mozilla.org/show_bug.cgi?id=1491303 , But the fact is that FireFox has not had acceleration on Linux in eons, so the fact that it is being worked on now is encouraging.

I'm not a programmer, but may of the benchmarks I've seen done on Phoronix over the years comparing GCC to LLV/Clang don't bear out what you say. Sometimes GCC trounces Clang in a *specific* bench, sometimes the other way around. They each seem to have their own + and -. The folks in the GCC camp, I am told, are rather stubborn in accepting contributions to the compiler which may greatly increase perf but will slow compilation (or some other related function) as a byproduct. I don't know how LLVM land is in terms of that.

As a regular bug reporter to Mozilla, yes, I have been following the Webrenderer stuff and it is good that Linux is getting work in that area done but it is lacking compared to Windows-land due to limitations in Linux. Just have to give it some time to mature.

[birdie]

Basemark is included as are others. Are you referring to 'Selenium'? It's simply a WebDriver for allowing the automation of the browser tests.. As you can see from the graphs, Basemark, Octane, and others are what is being tested.

Jetstream now added, the URL wasn't working a few days ago. https://openbenchmarking.org/test/system/selenium

And for those not familiar with Selenium: https://www.seleniumhq.org/ after learning its interfaces, liking it quite a lot for automating all the browser benchmarks from Python.

All great but we need specific benchmarks (some of which I've listed) to test rendering/rasterization/presentation performance and your benchmark doesn't quite catch that. Please next time include some of them.