Ultra Ethernet Consortium Started By LF, Intel, AMD, Meta, HPE & Others

[DavidBrown]

apparently Netflix-like traffic is not the main target of that standard.

According to their FAQ: "Deliver a complete architecture that optimizes Ethernet for high performance AI and HPC networking", and Linux is king in that area.

As far as I understand the way Netflix works, they don't use top-shelf networking speeds. Netflix prefers larger numbers of servers with cheaper hardware - they optimise for power usage, equipment cost, and convenient availability of parts. Big AI and HPC optimise for absolute speed, which is very different. (There are also systems which use expensive per-system or per-socket software licenses, and these also optimise for speed because horizontal scaling costs extra licenses. Again, this is not an issue for Netflix.)

Also, Netflix scales their networking by co-hosting with ISPs. No ISP wants to pay for the upstream links to support all their Netflix viewers - but they want to charge their customers for their downstream links. So they have Netflix caching servers at their hubs. That way, end-users get their Netflix feeds without needing the traffic on the trunk lines. (Google, Facebook, etc., have similar solutions, but it is particularly effective with Netflix.)

[DavidBrown]

Cisco carrier grade routes are running IOS-XR which is Linux based since 2013 when they switched from using QNX. Their non-Linux IOS is only used on lower end hardware these days, all their big stuff ruins either IOS-XR or IOS-XE both being Linux based.

Lots of routers and switches use Linux as their OS. But your aim when designing such hardware is that the packets do not get to the processor and the software. If you have a fast switch with 24 ports at 100 Gb, the network link to the processor running Linux might be only 1 or 10 Gbe. All the switching (and routing, for routers) is in the switch fabric on the switch chip. (The processor cores might be in the same chip, but they are independent sections of the chip.) Packets start coming in one port and are forwarded towards a new port before the packet has fully arrived at the switch - you certainly do not want packets moving into and out of the processor cores! You only pass in packets that are relevant to the core, such as RSTP or BGP packets to handle routing and switching paths, and of course management packets.

So the OS running on these things does not need support for 100 Gb networking, or anything in that line. It only needs support for 1 Gb or 10 Gb networking. And the user-level software running on the system needs to know how to control the high speed network switch fabric.

This is a very different situation from a server with a high speed network interface, where the traffic on the network port goes to and comes from the OS.

If you are not sure of the difference, a clue is in the core counts. On a switch, an 8 core ARM processor might be controlling 48 ports at 400 Gb. On a server, two 100 Gb ports might be serving 64 x86 cores.

[DavidBrown]

I remember, in 2007, being laughed at by all the experts for suggestion network companies like Cisco and Juiper run their cheaper offerings on linux to get it field tested and possibly bring their proces down somewhat (or make more profit!)

So Cisco changed in 2013. Glad to see I was right!

Anyone know if Ubiquiti run their gear on a linux stack?

Ubiquiti's EdgeOS was a fork of the Vyatta Linux firewall-oriented distribution. It was quite a while ago, and they've done a lot since, but that's the original. It's Debian at the core. Most firewalls and routers these days have Linux under the hood.

[DavidBrown]

I also find it irritating that all of this gold rush for fundamentally broken ML models is going to end up costing the environment far more damage than cryptocurrencies ever did or will. I can just hope that all this waste on AI bubbles will end up benefiting other more useful markets even as data centers drive water shortages in places they never should have been built in the first place.

Say what you will about the success or potential for ML models, but you are wrong about the environmental damage being worse. The key difference between cryptocurrency and big ML models is that the big ML runs in data centres paid for by companies. Running costs matter to them - the cost of electricity dominates, and thus they have an incentive to aim for efficiency. Cryptocurrency is mined by amateurs with poor quality and inefficient hardware, spread around the world. The cost of the electricity is often hidden (paid for by your parents, stolen from neighbours, acquired by dodgy deals with bureaucrats in high corruption countries, etc.). Sure, there are some professional groups that have serious setups, and they are often the ones actually making money, but it is all the rest that costs the environment.

I am not suggesting that big data ML is not an environmental cost - or that it is not a waste of electricity - but it wastes a small fraction of the power used in cryptocurrency, and cryptocurrency really is entirely wastage.

[stiiixy]

Bring their prices down. LOL. I'm not surprised people were laughing at you and still would be.

Fark mate, did you not read the part where I said "make more money"?

Bottom lines.

And try read between them, sometimes.

[stiiixy]

Ubiquiti's EdgeOS was a fork of the Vyatta Linux firewall-oriented distribution. It was quite a while ago, and they've done a lot since, but that's the original. It's Debian at the core. Most firewalls and routers these days have Linux under the hood.

I'm actually talking to a mate who ised to deploy Uniquit hear for regional networks, as we possibly need a network covering several hundred square kilomtres of country, and Ubiquit kept coming up as a decent product ans price for our use case.

I'm curious to know if you do, doe s Ubiquiti still use a closed off top end software manafement suite on top of a linux base?

[DavidBrown]

I'm actually talking to a mate who ised to deploy Uniquit hear for regional networks, as we possibly need a network covering several hundred square kilomtres of country, and Ubiquit kept coming up as a decent product ans price for our use case.

I'm curious to know if you do, doe s Ubiquiti still use a closed off top end software manafement suite on top of a linux base?

I must admit that I've never bothered looking at the licensing or source availability for their web interface or their command-line configuration tools. It's all easily accessible Linux underneath, at least on the EdgeRouter-X devices we use regularly - I use ssh into them all the time for setup and configuration. These are 5 port devices, and give you a lot for the money (which is not much more than some 5 port switches). We use a fair number of them on different sites, controlling small local networks and with OpenVPN connections back to the main site. We've had a few failures out at the small sites, but those are relatively harsh environment that can be daunting for a lot of hardware. We simply replace the EdgeRouters as needed - they are cheap enough to keep extra on stock.

I'm not too keen on Ubiquiti's Unifi stuff. I think its fine if you have a unified network using just their stuff, but it's a pain if you want a mixed environment. But their wireless equipment seems good quality, and we've used a point-to-point link between buildings that has worked flawlessly.

[cynic]

I read that, too. But it won't change that it will be used for other interconnects, not just AI and HPC. My point is that the assertion that Linux is the only high bandwidth OS is provably false.

indeed nobody said that. Please rearead with care.
High bandwidth is a thing, HPC is another thing.

I also find it irritating that all of this gold rush for fundamentally broken ML models is going to end up costing the environment ar more damage than cryptocurrencies ever did or will. I can just hope that all this waste on AI bubbles will end up benefiting other more useful markets even as data centers drive water shortages in places they never should have been built in the first place.

LOL!

https://endthefud.org/

[cynyr]

10Gbe is getting more "affordable" for loose definitions. A lot of mid/high end motherboards now come with 10Gb options. The catch is the CAT-6/CAT-6A or fiber you need for it. Very few (if any) houses will have that pre-wired, so the effort to get the cable pulled is going to be pretty brutal. Fiber based hardware (ex. older mellanox connectx-3 hardware) is really quite affordable on E-bay these days.

A year or so ago a friend of mine built a full out 10Gb network in his house. He used mellanox equipment and pulled new fiber cable throughout. Fiber cable can be easier to pull than CAT (its smaller) but you do have to be more careful with it.

At best this Ultra-Ethernet will likely compete with infiniband in data center applications.

The cards are pretty cheap, but a 12+ port 10gbe switch (10gbaseT or SFP+) for a reasonable price is basically unobtainium. for a small cluster, a set of 2 port connectx'3's would work great, for the nas, server, router, and 2 desktops you need a switch and it would be nice to have a bit of expand-ability for the future.

[zexelon]

The cards are pretty cheap, but a 12+ port 10gbe switch (10gbaseT or SFP+) for a reasonable price is basically unobtainium. for a small cluster, a set of 2 port connectx'3's would work great, for the nas, server, router, and 2 desktops you need a switch and it would be nice to have a bit of expand-ability for the future.

https://www.amazon.com/Mikrotik-CRS310-1G-5S-4S-Ports-Gigabit-Ethernet/dp/B09W7WBQ77/ref=sr_1_1?crid=3MYYMAO0T6U8A&keywords=CRS310-1G-5S-4S%2Bin&qid=1689903581&sprefix=crs310-1g-5s-4s%2Bin%2Caps%2C101&sr=8-1

Not sure what country you might be in, but the above link is to a Mikrotik solution. Its not "super great" but for a basic setup it could work well and its $200 USD. That's in the ball park of a decent wifi router these days... granted this is just a switch but hey it supports 10Gb

I have used some of the Mikrotik equipment in some very basic setups and it fully supports 10Gbe over fiber without any issues on the four port unit I used. It was only for in rack communications to support a GlusterFS setup.