Linux Distributions vs. BSDs With netperf & iperf3 Network Performance

[boxie]

Michael is there any chance of some 10gbit networking tests? 1gbit networking tests would be the equivalent of testing ioquake3 0.8.5

[Michael]

Michael is there any chance of some 10gbit networking tests? 1gbit networking tests would be the equivalent of testing ioquake3 0.8.5

Definitely, soon as someone/company sponsors the hardware for such tests... Until then, I don't have such equipment...

Edit: obviously if I had such hardware would have provided some tests in the first place.

[gugus]

I believe there is a bug with the phoronix iperf3/UDP bench on FreeBSD.
From a small 4 cores ATOM C2558 with Intel i350 gigabit NIC:

[root@netgate]~# iperf3 -c 198.19.0.203 -u -p 9090 -b 1G -P 5
Connecting to host 198.19.0.203, port 9090
[  4] local 198.19.0.209 port 42742 connected to 198.19.0.203 port 9090
[  6] local 198.19.0.209 port 48170 connected to 198.19.0.203 port 9090
[  8] local 198.19.0.209 port 52621 connected to 198.19.0.203 port 9090
[ 10] local 198.19.0.209 port 46259 connected to 198.19.0.203 port 9090
[ 12] local 198.19.0.209 port 12809 connected to 198.19.0.203 port 9090
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  21.6 MBytes   182 Mbits/sec  23909
[  6]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec  2770
[  8]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec  2732
[ 10]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec  2732
[ 12]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec  2732
[SUM]   0.00-1.00   sec   107 MBytes   898 Mbits/sec  34875
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec  23.0 MBytes   193 Mbits/sec  27529
[  6]   1.00-2.00   sec  22.7 MBytes   190 Mbits/sec  2944
[  8]   1.00-2.00   sec  22.7 MBytes   190 Mbits/sec  2901
[ 10]   1.00-2.00   sec  22.7 MBytes   190 Mbits/sec  2901
[ 12]   1.00-2.00   sec  22.7 MBytes   190 Mbits/sec  2901
[SUM]   1.00-2.00   sec   114 MBytes   953 Mbits/sec  39176
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec  23.0 MBytes   193 Mbits/sec  27531
[  6]   2.00-3.00   sec  22.6 MBytes   190 Mbits/sec  2942
[  8]   2.00-3.00   sec  22.6 MBytes   190 Mbits/sec  2899
[ 10]   2.00-3.00   sec  22.6 MBytes   190 Mbits/sec  2899
[ 12]   2.00-3.00   sec  22.6 MBytes   190 Mbits/sec  2899
[SUM]   2.00-3.00   sec   114 MBytes   953 Mbits/sec  39170
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec  23.1 MBytes   194 Mbits/sec  27437
[  6]   3.00-4.00   sec  22.8 MBytes   191 Mbits/sec  2962
[  8]   3.00-4.00   sec  22.8 MBytes   191 Mbits/sec  2919
[ 10]   3.00-4.00   sec  22.8 MBytes   191 Mbits/sec  2919
[ 12]   3.00-4.00   sec  22.8 MBytes   191 Mbits/sec  2919
[SUM]   3.00-4.00   sec   114 MBytes   959 Mbits/sec  39156
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec  23.2 MBytes   194 Mbits/sec  27321
[  6]   4.00-5.00   sec  22.8 MBytes   192 Mbits/sec  2967
[  8]   4.00-5.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 10]   4.00-5.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 12]   4.00-5.00   sec  22.8 MBytes   192 Mbits/sec  2924
[SUM]   4.00-5.00   sec   115 MBytes   961 Mbits/sec  39060
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec  23.2 MBytes   194 Mbits/sec  27557
[  6]   5.00-6.00   sec  22.8 MBytes   192 Mbits/sec  2967
[  8]   5.00-6.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 10]   5.00-6.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 12]   5.00-6.00   sec  22.8 MBytes   192 Mbits/sec  2924
[SUM]   5.00-6.00   sec   115 MBytes   961 Mbits/sec  39296
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec  23.2 MBytes   194 Mbits/sec  27363
[  6]   6.00-7.00   sec  22.8 MBytes   192 Mbits/sec  2967
[  8]   6.00-7.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 10]   6.00-7.00   sec  22.8 MBytes   192 Mbits/sec  2924
[ 12]   6.00-7.00   sec  22.8 MBytes   192 Mbits/sec  2924
[SUM]   6.00-7.00   sec   115 MBytes   961 Mbits/sec  39102
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec  23.1 MBytes   194 Mbits/sec  27480
[  6]   7.00-8.00   sec  22.8 MBytes   191 Mbits/sec  2958
[  8]   7.00-8.00   sec  22.8 MBytes   191 Mbits/sec  2916
[ 10]   7.00-8.00   sec  22.8 MBytes   191 Mbits/sec  2916
[ 12]   7.00-8.00   sec  22.8 MBytes   191 Mbits/sec  2916
[SUM]   7.00-8.00   sec   114 MBytes   958 Mbits/sec  39186
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec  23.0 MBytes   193 Mbits/sec  27535
[  6]   8.00-9.00   sec  22.6 MBytes   190 Mbits/sec  2942
[  8]   8.00-9.00   sec  22.6 MBytes   190 Mbits/sec  2899
[ 10]   8.00-9.00   sec  22.6 MBytes   190 Mbits/sec  2899
[ 12]   8.00-9.00   sec  22.6 MBytes   190 Mbits/sec  2899
[SUM]   8.00-9.00   sec   114 MBytes   953 Mbits/sec  39174
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec  23.0 MBytes   193 Mbits/sec  27568
[  6]   9.00-10.00  sec  22.7 MBytes   190 Mbits/sec  2944
[  8]   9.00-10.00  sec  22.7 MBytes   190 Mbits/sec  2901
[ 10]   9.00-10.00  sec  22.7 MBytes   190 Mbits/sec  2901
[ 12]   9.00-10.00  sec  22.7 MBytes   190 Mbits/sec  2901
[SUM]   9.00-10.00  sec   114 MBytes   953 Mbits/sec  39215
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   229 MBytes   192 Mbits/sec  0.629 ms  241919/271230 (89%)
[  4] Sent 271230 datagrams
[  6]   0.00-10.00  sec   226 MBytes   190 Mbits/sec  199.514 ms  505/29363 (1.7%)
[  6] Sent 29363 datagrams
[  8]   0.00-10.00  sec   226 MBytes   190 Mbits/sec  0.678 ms  64/28939 (0.22%)
[  8] Sent 28939 datagrams
[ 10]   0.00-10.00  sec   226 MBytes   190 Mbits/sec  0.636 ms  63/28939 (0.22%)
[ 10] Sent 28939 datagrams
[ 12]   0.00-10.00  sec   226 MBytes   190 Mbits/sec  0.660 ms  68/28938 (0.23%)
[ 12] Sent 28938 datagrams
[SUM]   0.00-10.00  sec  1.11 GBytes   951 Mbits/sec  40.423 ms  242619/387409 (63%)

iperf Done.

[MadCatX]

One has got to love the "scientific" approach almost everybody in this thread has displayed so far. Michael's benchmark had spitted out some numbers and everyone immediately jumped to conclusions about Linux sucking, BSD being far superior and whatnot. Nobody seemed to realize that results of any benchmark are just numbers which are absolutely meaningless without proper interpretation. For instance, Fedora comes with a firewalld daemon enabled by default and so does CentOS - it is quite likely that it could have interfered with some of the tests. Look at the min-max bars of CentOS and Fedora performances in that one microbenchmark where BSD came out ahead - in some cases it gets close to 100 % error. This clearly suggests there was something going on behind the scenes. But yeah, it's easier to take some empty numbers at face value and go from there...

[doom_Oo7]

People don't understand what bloated/lightweight means. Archlinux for example claims to be lightweight, and yet a minimal installation is several times larger than a Debian minimal installation. Installing some packages just increases this difference further. Just take something like gnuplot. On Arch it will not only pull in qt5 but also mesa, gtk2, gstreamer, avahi, wayland ... and their dependencies plus all the development files ...
We're talking ~10 vs. over 250 MB here.

Ubuntu is not much more bloated, and it doesn't matter in this test anyway. It ships more up-to-date software however, which is probably the main reason why there are differences.

Do you really think that the number and size of packages that are on your hard drive are in any measure relevant to network speed ? It's just 0s and 1s on your hard drive, you could have 8k HDR cat videos on your computer and it would be the same.


I wonder, with such differences, if there may not be a problem at some point with the ethernet packets sent by Linux that forces the router to do more work or something like this... it would be interesting to do some comparison with wireshark between linux and bsd for the same task.

[renox]

I wonder if we're not seeing the throughput vs latency trade-off here: if you use big buffer you have high bandwidth but poor latency and if you use small buffers you have low latency but low bandwidth.
FreeBSD seems tuned more for high throughput.

[indepe]

Considering that Ubuntu is running on the server in all tests including the BSD ones, this looks like the differences in transaction rates may be due to the workstation vs. server settings on the Linux distributions.

It would be interesting to see which rates could be achieved with settings optimally configured for the test, and what these settings would be. (Also using CPU freq "performance" instead of "powersave" on all distros, even though on at least some distros this is unnecessarily cumbersome to change.)

[aht0]

(Also using CPU freq "performance" instead of "powersave" on all distros, even though on at least some distros this is unnecessarily cumbersome to change.)

just lock it on "performance" from BIOS? Even if you'd have to benchmark using consumer-grade board..

[starshipeleven]

Michael would be nice to note basic things like firewalls were enabled or not in the article, even if someone could eventually find that info from the "default settings" of each tested distro.

[talking of firewalls enabled]
Fedora - Yes
Ubuntu - No (apparently doesn't even have the kernel module installed by default)
If he was just testing the default install of FreeBSD the firewall (or rather your choice of PF, IPFW, or IPFilter) would be disabled

Neat, thanks for this info.

So we can see that with firewall the linux net latency is doubled, while FreeBSD without firewall has like half the latency of non-firewalled distros while on pure bandwith tests everyone is more or less on par.

this is an explanation of the workloads where FBSD is going better than Linux (UDP is another network protocol but the same thing is performed)

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.performance/tcp_request_response_workload_tun.htm

"The primary metric of interest in these workloads is the round-trip latency of the network"

So it seems FBSD kernel has a much lower network latency, without firewall. Would be interesting to know its latency with firewall too.

[starshipeleven]

just lock it on "performance" from BIOS? Even if you'd have to benchmark using consumer-grade board..

You mean "disable CPU powersaving" or something like that? Most BIOS have that option and it should lock frequency on max. I never tried it though.