Netflix Optimized FreeBSD's Network Stack More Than Doubled AMD EPYC Performance
Written by Michael Larabel in BSD on 4 November 2019 at 07:35 PM EST. 47 Comments
BSD --
Drew Gallatin of Netflix presented at the recent EuroBSDcon 2019 conference in Norway on the company's network stack optimizations to FreeBSD. Netflix was working on being able to deliver 200Gb/s network performance for video streaming out of Intel Xeon and AMD EPYC servers, to which they are now at 190Gb/s+ and in the process that doubled the potential of EPYC Naples/Rome servers and also very hefty upgrades too for Intel.

Netflix has long been known to be using FreeBSD in their data centers particularly where network performance is concerned. But in wanting to deliver 200Gb/s throughput from individual servers led them to making NUMA optimizations to the FreeBSD network stack. Allocating NUMA local memory for kernel TLS crypto buffers and for backing files sent via sentfile were among their optimizations. Changes to network connection handling and dealing with incoming connections to Nginx were also made.

For those just wanting the end result, Netflix's NUMA optimizations to FreeBSD resulted in their Intel Xeon servers going from 105Gb/s to 191Gb/s while the NUMA fabric utilization dropped from 40% to 13%.


The AMD EPYC performance is even more impressive in going from 68GB/s to 194GB/s. So while EPYC started out much slower than Xeon, the Netflix AMD EPYC servers are now closer than Intel for achieving 200Gb/s performance.


Not only is EPYC faster, but thanks to the 128 PCIe lanes per socket they are able to get by in one socket what they are using two Intel Xeon CPUs for otherwise. One area that AMD was critiqued for is the inability for Netflix to monitor the Infinity Fabric saturation as "AMD's tools are lacking (even on Linux)."


In the end they are now effectively at 200Gb/s encrypted video streaming from FreeBSD per server. More details via this interesting slide deck.
Related News
About The Author
Author picture

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter or contacted via MichaelLarabel.com.

Popular News This Week