Netflix Continues Experiencing Great Performance In Using FreeBSD For Their CDN

[aht0]

I think the hardware of their Open Connect Applicance fascinating.. "embedded" indeed, lol.

https://gizmodo.com/this-box-can-hold-an-entire-netflix-1592590450

Why FreeBSD. Official statement:

https://www.freebsdfoundation.org/testimonial/netflix/

More than 30 million Netflix streaming members around the globe watch more than a billion hours of movies and TV shows each month. In the US, Netflix video streaming accounts for a third of peak downstream Internet traffic. Netflix created Open Connect, a single-purpose content delivery network, to help deliver these petabytes of data.

The main component of Open Connect is the Open Connect Appliance, a small-footprint network streaming device. The Open Connect Appliance is a 4U Intel-based server that is designed to economically maximize storage density in a space and power footprint that is ideal for both ISP data centers and metro-area network interchanges.

The Open Connect Appliance runs on FreeBSD 9. Netflix picked FreeBSD 9 because it is a high performing, low-maintenance and reliable operating system that is supported by major hardware vendors. FreeBSD 9 provides a foundation of reliability, performance, and hands-free manageability. Combined with NGINX, a light-weight and high performance Web server, FreeBSD 9 provides a simple but powerful solution that is capable of serving tens of thousands of simultaneous video streams across multiple 10Gbit fiber optic links.

Beyond its technical strengths, FreeBSD comes with an outstanding ecosystem of developers, vendors, and users who openly share expertise, talent, and technical improvements. Netflix has embraced this community and is committed to giving back its bug fixes and enhancements, thus completing the circle of community collaboration.

– David Fullagar, Director of Content Delivery Architecture, Netflix

I'd say following addresses alot of ranting and rambling in this thread, especially traditional bullshit allegations about the BSD license being the cause.. Blog's owner, Adrian Chadd is one of the FreeBSD's driver developers.

http://adrianchadd.blogspot.com/2012/06/freebsd-netflix-cdn.html

You could say "Netflix chose FreeBSD because they can keep their changes proprietary." Sure, they could. But they're not making appliances that they're selling - they're owning the infrastructure and servers. It's unclear whether they'd have to contribute back any Linux changes if they ran Linux on their open connect platform. They're making a conscious, public decision to distribute their changes back to FreeBSD - even though they don't have to.


You could say "Netflix chose FreeBSD because the people inside the company knew FreeBSD." Sure, they may have. The same thing could be said about why start-ups and tech companies choose Linux. A lot of the time its because they're chasing enterprise support from Redhat. But technology startups using Ubuntu or Debian tend not to be paying support fees - they hire smart people who know the technology. So, yes - "using what they know."


According to the Netflix Openconnect website:


"This was selected for its balance of stability and features, a strong development community and staff expertise. We will contribute changes we make as part of our project to the community through the FreeBSD committers on our team."



Let's pull this apart a little.

1. "Balance of stability and features." FreeBSD has long been derided for how slowly it moves in some areas. The FreeBSD developers tend to be a conservative bunch, trying to find the balance between new feature development and maintaining both stability and backwards compatibility.
2. "Strong community." FreeBSD has a strong technical development community and Netflix finds this very important. They're also willing to join and participate in the community like many other companies do.
3. "Staff expertise." So yes, their staff are familiar with FreeBSD. They're also familiar with Linux. They chose a platform which they have the expertise to develop, use and improve. They didn't just choose an unfamiliar platform because of marketing brochures or sales promises. I don't see any negatives here. I'm sure that Google engineers chose Linux to begin with because they were familiar with Linux.
4. "Contribute changes we make as part of our project to the community." Netflix has committed to push improvements and fixes back to the upstream project They contributed some bug fixes in the 10GE Intel driver and IPv6 stack this week. This is collaborative open source working the way it should.

Why would Netflix push back changes and improvements into a project when they're not required to? That's something you should likely ask them. But the same good practice arguments hold for both Linux AND BSD projects:

• The project is a constantly moving target. If you don't push your changes back upstream, you risk carrying around increasingly larger changes as your project and your BSD upstream project diverge. This will just make things more difficult in the long run.
• By pushing your changes upstream, you make it easier to move with the project - including adopting improvements and new features. If you keep large changes to yourself, you will likely find it increasingly difficult to update your software to the newer upstream versions. And that upstream project is likely adding bug fixes, improvements and new features - which at some point you may wish to leverage. By pushing your changes upstream, you make it a lot easier to move to future versions of the upstream project, allowing you to leverage all those fixes and improvements without too much engineering time.
• By participating, you encourage others to adopt your technology. By pushing your changes and improvements upstream, you decrease the amount of software you have to maintain yourself (and keep patching as the upstream project moves along.) But you also start to foster technology adoption. The FreeBSD jail project started out of the desire by a hosting company to support virtualisation. Since then, the Jail infrastructure has been adopted by many other companies and individuals.
• When others use your technology, they also find and fix bugs in your technology; they may even improve it. The FreeBSD jail support has been extended to include IPv6 support, shared memory support and integrates into the VIMAGE (virtualised networking) stack (which, by the way, came from Ironport/Cisco.) As a company, you may find that the community will do quite a lot of the work that you would normally have to hire engineers to do yourself. This saves time and saves money.
• When companies contribute upstream, it encourages other companies to also contribute upstream. A common issue is "reinventing the wheel", where companies end up having to reinvent the same technology privately because no-one has contributed it upstream. They solve the same problems, they implement the same new features .. and they all spend engineering time and resources to do so.
• And when companies contribute upstream, it encourages (private) developers to contribute. Open source developers love to see their code out there in the wild, in places they never quite thought of. It's encouraging to see companies build products with their code and contribute back bug fixes and improvements. It fosters a sense of community and participation, of "give and take", rather than just "take". This is exactly the kind of thing that keeps developers coming back to contribute more - and it attracts new developers. Honestly, who wouldn't want to say that some popular device is running code that they wrote in their spare time?

So, you could rant and rave about the conspiracy side of Linux versus FreeBSD. You could rant on about GPL versus BSD. Or, you could see the more useful side of things. You could see a large company who didn't have to participate at all, agreeing to contribute back their improvements to an open source operating system. You could see that by doing so, the entire open source ecosystem benefits - not just FreeBSD. There's nothing stopping Linux or other BSD projects from keeping an eye on the improvements made by Netflix and incorporating those improvements into their own project. And it's another case of a company participating and engaging the open source community - and having that community engage them right back.

Good show, Netflix. Good show.

[stiiixy]

Having said that, not sire how he'd go about testing Netflix's workloads specifically. However if the tests are there, NetFlix could provide meaningful results using PTS at some stage. The issue though would be sanitising any corporate secrets they would prefer not reverse engineered to easily.

[Wojcian]

With FreeBSD on commodity server hardware they are able to achieve 90 Gb/s serving on TLS-encrypted connections with not even full CPU utilization.

Is 90 Gb/s upper limit? If it is not it sounds rather like a failure. On the other side if there is hardware offloading it doesn't say much about system performance.

[tuxd3v]

Its very well known in the SysAdmin area that a System that tries to parametrize everything in a static way( like Windows does )..is in the loosing side in the server space( But it wins in the desktop )
Linux was known in the past for its ability to molde in a way the SysAdmins wanted too..

We were able to build Servers for several tasks at which they were really good, and they performed that task only..
Nowadays with so static parameters compiled there, its harder to optimize OS for one area, whereas in the past was amazing what we achieved..
Too much bloatware.. its the downside of GNU/Linux this days..

FreeBSD have been capitalizing on that for the last years, in some aspects...it is the good GNU/Linux we had from the past
FreeBSD takes evolutive approach..

[Wojcian]

We were able to build Servers for several tasks at which they were really good, and they performed that task only..
Nowadays with so static parameters compiled there, its harder to optimize OS for one area, whereas in the past was amazing what we achieved..
Too much bloatware.. its the downside of GNU/Linux this days..

Is this hidden anti systemd propaganda? I thought we were talking about Netflix results.

[tuxd3v]

Is this hidden anti systemd propaganda? I thought we were talking about Netflix results.

Yes we were talking about NetFlix results,
But comparing with lots of GNU/Linux distributions nowadays..

I don´t even touched that name, out-of-nothing, it started to poping out..

[cybertraveler]

Its very well known in the SysAdmin area that a System that tries to parametrize everything in a static way( like Windows does )..is in the loosing side in the server space( But it wins in the desktop )
Linux was known in the past for its ability to molde in a way the SysAdmins wanted too..

We were able to build Servers for several tasks at which they were really good, and they performed that task only..
Nowadays with so static parameters compiled there, its harder to optimize OS for one area, whereas in the past was amazing what we achieved..
Too much bloatware.. its the downside of GNU/Linux this days..

FreeBSD have been capitalizing on that for the last years, in some aspects...it is the good GNU/Linux we had from the past
FreeBSD takes evolutive approach..

I think your post is reasonable, however I think there are still lightweight GNU/Linux options to use in the scenarios you described. For instance:

• https://www.gentoo.org/ - I think it's described as a meta-distro. You can make something very small, optimized and domain specific using it, with convenient tools to cut down work.
• https://www.alpinelinux.org/ - Very small. I think it's mostly focused on running in containers, but I bet it could make for a nice base for a project where you want a stripped down distro.
• https://clearlinux.org/ - If you're using modern Intel hardware, this is probably a decent option to start with. Bleeding edge and performance focused.

[hajj_3]

videos from fosdem are being uploaded here: https://video.fosdem.org/2019/ It usually takes them a few weeks to upload all the talks.

[polarathene]

Yep, look at the post just above yours :-)

Yeah, you kinda beat me to it, it wasn't there when I had began typing :P

[Antartica]

AFAIK, the FreeBSD network stack has higher throughput than Linux's because it manages to eschew some buffer copying (zero-copy).

Some time ago there was some work in the Linux network stack towards also doing that (haven't tracked that development, don't know if it's still a work in progress).