Ultra Ethernet Consortium Started By LF, Intel, AMD, Meta, HPE & Others

[F.Ultra]

Ya, Cisco is the only game in town. smh.

and what does that have to do with the comment that I quoted and corrected?

[F.Ultra]

Lots of routers and switches use Linux as their OS. But your aim when designing such hardware is that the packets do not get to the processor and the software. If you have a fast switch with 24 ports at 100 Gb, the network link to the processor running Linux might be only 1 or 10 Gbe. All the switching (and routing, for routers) is in the switch fabric on the switch chip. (The processor cores might be in the same chip, but they are independent sections of the chip.) Packets start coming in one port and are forwarded towards a new port before the packet has fully arrived at the switch - you certainly do not want packets moving into and out of the processor cores! You only pass in packets that are relevant to the core, such as RSTP or BGP packets to handle routing and switching paths, and of course management packets.

So the OS running on these things does not need support for 100 Gb networking, or anything in that line. It only needs support for 1 Gb or 10 Gb networking. And the user-level software running on the system needs to know how to control the high speed network switch fabric.

This is a very different situation from a server with a high speed network interface, where the traffic on the network port goes to and comes from the OS.

If you are not sure of the difference, a clue is in the core counts. On a switch, an 8 core ARM processor might be controlling 48 ports at 400 Gb. On a server, two 100 Gb ports might be serving 64 x86 cores.

Ofc they use switching fabrics, I thought that was common knowledge? There is no way that they would every be able to sustain line level performance at those speeds with the weak cpu:s that they have on those things. The context was that Cisco used IOS on those routers when they in fact use IOS-XE or IOS-XR, not if they used Linux to switch packages.

[DavidBrown]

Ofc they use switching fabrics, I thought that was common knowledge? There is no way that they would every be able to sustain line level performance at those speeds with the weak cpu:s that they have on those things. The context was that Cisco used IOS on those routers when they in fact use IOS-XE or IOS-XR, not if they used Linux to switch packages.

It makes any discussion about OS support for new high-speed networking pretty much irrelevant for these kinds of systems, that's all.

[DavidBrown]

https://www.amazon.com/Mikrotik-CRS310-1G-5S-4S-Ports-Gigabit-Ethernet/dp/B09W7WBQ77/ref=sr_1_1?crid=3MYYMAO0T6U8A&keywords=CRS310-1G-5S-4S%2Bin&qid=1689903581&sprefix=crs310-1g-5s-4s%2Bin%2Caps%2C101&sr=8-1

Not sure what country you might be in, but the above link is to a Mikrotik solution. Its not "super great" but for a basic setup it could work well and its $200 USD. That's in the ball park of a decent wifi router these days... granted this is just a switch but hey it supports 10Gb

I have used some of the Mikrotik equipment in some very basic setups and it fully supports 10Gbe over fiber without any issues on the four port unit I used. It was only for in rack communications to support a GlusterFS setup.

Mikrotik make fine switches, with excellent value for money. We use a lot of those particular devices - simple, reliable, low-power and easy to work with. But they are often unobtanium - the global challenges in semiconductor and electronic part supplies, combined with the popularity of Mikrotik devices, has made supply of them unreliable. If you can get them, they are great devices.

[zexelon]

Mikrotik make fine switches, with excellent value for money. We use a lot of those particular devices - simple, reliable, low-power and easy to work with. But they are often unobtanium - the global challenges in semiconductor and electronic part supplies, combined with the popularity of Mikrotik devices, has made supply of them unreliable. If you can get them, they are great devices.

Thanks for the feedback! Ironically for our needs it was the only thing we could get our hands on. Personally I had no prior experience so I was definitely hesitant about using them as a critical piece so we built fail over in just in case. So far though no issues and yes the performance is awesome!

[DavidBrown]

Thanks for the feedback! Ironically for our needs it was the only thing we could get our hands on. Personally I had no prior experience so I was definitely hesitant about using them as a critical piece so we built fail over in just in case. So far though no issues and yes the performance is awesome!

I also first used Mikrotik as it was pretty much the only available 10 G trunk switch at a sane price, in stock at our suppliers. After we got a few of these, and liked them, we bought about a dozen of them - most of the stock in the country at the time. I keep an eye out, and buy them when the distributor gets them in, so that I have some extras on stock, as the distributors can be out for months at a time.

[F.Ultra]

It makes any discussion about OS support for new high-speed networking pretty much irrelevant for these kinds of systems, that's all.

true, was just surprised that it was written as a response to my post and not the original one . As for why Linux is involved this clearly have to do with simple the Linux Foundation being a driving force behind this _and_ the prevalence of Linux on the HPC market.

[cb88]

The Ubiquiti ER4 runs on a Linux stack. I have set up several of them now and they are an odd stopgap of really low end "commercial grade" or quite high end "pro-sumer" grade. Their not to bad but not to great either.

Yeah not a single Ubiquiti product supports any form of RDMA or any hardware acceleration at to even be able to utilize 10GB and faster switches.

There is really no point in paying extra for Ubiquiti when it is in the same class of hardware is Mikrotik.

[cb88]

I also first used Mikrotik as it was pretty much the only available 10 G trunk switch at a sane price, in stock at our suppliers. After we got a few of these, and liked them, we bought about a dozen of them - most of the stock in the country at the time. I keep an eye out, and buy them when the distributor gets them in, so that I have some extras on stock, as the distributors can be out for months at a time.

The problem with thier swtich and Ubiquiti's is no RMDA... so even though it advertises 10GB good luck ever getting it to transfer anything fast.

[stiiixy]

The problem with thier swtich and Ubiquiti's is no RMDA... so even though it advertises 10GB good luck ever getting it to transfer anything fast.

What we're after, 1GB would be sufficient. It's a reliable signal in all but the worst of weather we need. Mikrotik would be an option but it might be on that 'if so cheap, cant be good' manager stigma wagon.

Definitely going to look in to Mikrotik for personal/SMB use though.