800Gb/sec Ethernet Spec Announced

[The1Dickens]

Now you can download RDR in ~1.5 seconds (tons of caveats and conditions apply) with the newly minted specification.

Quote

The specification announced Monday will be used by enterprises way before you're able to use it with your best desktop PC. In fact, 10 GbE is still a notable cost undertaking for consumers. Instead, 800 GbE will target high-performance computing (HPC) and mainstream applications, such as remote AI processing, that will benefit from lower latency.

 

Formally called 800GBASE-R, the standard is primarily based on the existing 400 GbE logic, as this would ensure fewer changes are necessary to get the new standard up and running. But it also brings a new media access control (MAC) and Physical Coding Sublayer (PCS).

There is a link to the pdf in the article, but some folks don't like link-hopping, so I'll link it here as well --> 800Gb/s Specification

 

Posted on Tom's Hardware.

So, I went into this article (with the opening paragraph) thinking it was going to be used for Netflix-like services for better serving entertainment, but further in the article, it says that it's going to be used for "high performance computing" and "AI Processing" to take advantage of the lower latency. Which sounds like, best case, it will be a decade before Netflix (and similar services) would implement this. I've yet to read the spec, but it may be well above my understanding at this point. And I am doubtful, but I wonder if ISPs could utilize this to start upgrading infrastructure and more home users could have access to 1Gb/s+ service.

[TempestCatto]

I wish that'd be right up to the house, but having it go between data centers is probably much more beneficial to us. Maybe Netflix in HD will finally look decent.

[Techstorm970]

21 minutes ago, The1Dickens said:

Now you can download RDR in ~1.5 seconds (tons of caveats and conditions apply) with the newly minted specification.

There is a link to the pdf in the article, but some folks don't like link-hopping, so I'll link it here as well --> 800Gb/s Specification

 

Posted on Tom's Hardware.

So, I went into this article (with the opening paragraph) thinking it was going to be used for Netflix-like services for better serving entertainment, but further in the article, it says that it's going to be used for "high performance computing" and "AI Processing" to take advantage of the lower latency. Which sounds like, best case, it will be a decade before Netflix (and similar services) would implement this. I've yet to read the spec, but it may be well above my understanding at this point. And I am doubtful, but I wonder if ISPs could utilize this to start upgrading infrastructure and more home users could have access to 1Gb/s+ service.

800Gb/sec = 100GB/sec

 

Holy fucking shit!!!!

[Fnige]

Just now, Techstorm970 said:

800Gb/sec = 100GB/sec

 

Holy fucking shit!!!!

finally, i can download the world of Flight simulator 2020 in 23.15 days

[ARikozuM]

Great... Now they have even more reason to ship us an unfinished game with 100GB of broken-ass updates. 

[WereCatf]

In the meanwhile, even just the 2.5Gbps Ethernet-hardware is still missing in the affordable, consumer-targeted market.

[williamcll]

That's nearly 3 times as fast as a PCIe4 x16, do you plug the network card into the DIMM slot or what?

[Dylanc1500]

34 minutes ago, williamcll said:

That's nearly 3 times as fast as a PCIe4 x16, do you plug the network card into the DIMM slot or what?

PCIe 32x or a custom slot that passes through as many lanes as your want. There are tons of servers that use proprietary slots for PCIe devices that have more than 16 lanes. Super micro has a board with a four 24 lane slots.

 

1 hour ago, SlimyPython said:

finally, i can download the world of Flight simulator 2020 in 23.15 days

Yeah, at 1:25 scale.

[Orangeator]

Can a CPU even handle 100 Gigabytes per second? PCIe Gen 4 does 32 Gigabytes per second each way correct? So both directions would be 64 Gigabytes per second. Nevertheless, even if you had PCie Gen 4 x32 slot (somehow), where would that data go to be stored? To my knowledge there is nothing that could send that amount of data that fast off, nevertheless receive it... 

[BuckGup]

4 hours ago, TempestCatto said:

I wish that'd be right up to the house, but having it go between data centers is probably much more beneficial to us. Maybe Netflix in HD will finally look decent.

They use other technologies then standard RJ45. Everything is fiber optic and uses Mellanox or Slingshot technology. But that's mainly used for nodes talking to one another not data between datacenters 

[leadeater]

1 hour ago, Dylanc1500 said:

PCIe 32x or a custom slot that passes through as many lanes as your want. There are tons of servers that use proprietary slots for PCIe devices that have more than 16 lanes. Super micro has a board with a four 24 lane slots.

Actual cards in the wild will probably end up being PCIe 5.0 since that's actually not that far off.

[CarlBar]

37 minutes ago, Orangeator said:

Can a CPU even handle 100 Gigabytes per second? PCIe Gen 4 does 32 Gigabytes per second each way correct? So both directions would be 64 Gigabytes per second. Nevertheless, even if you had PCie Gen 4 x32 slot (somehow), where would that data go to be stored? To my knowledge there is nothing that could send that amount of data that fast off, nevertheless receive it... 

 

As noted further up custom solutions exist  or can be made easily enough. Ditto on the transfer rate of data from thing. A single SATA drive could never provide that much data, but a larger server grade storage array absolutely could. Of the top of my head, (going from memory so apply a little salt), a dual EPYC system has somthing like 10 times this bandwidth in PCIE lanes.

[CarlBar]

1 minute ago, leadeater said:

Actual cards in the wild will probably end up being PCIe 5.0 since that's actually not that far off.

 

This also.

[leadeater]

20 minutes ago, CarlBar said:

 

As noted further up custom solutions exist  or can be made easily enough. Ditto on the transfer rate of data from thing. A single SATA drive could never provide that much data, but a larger server grade storage array absolutely could. Of the top of my head, (going from memory so apply a little salt), a dual EPYC system has somthing like 10 times this bandwidth in PCIE lanes.

In reality the first place that will get deployment of 800Gb interfaces will be top of rack switches to distribution switches, between switches first. Servers with 800Gb interfaces are very much ages away as even in the highest performance realms 100Gb is still the go to not 400Gb. Right now Mellanox only has 200Gb cards per port max.

 

Mellanox does have 16 port 400Gb switch ICs (ASICs), Spectrum-2, but those are designed to go in to switches not on NICs.

 

The important thing to note however is this is not an upgrade to the signaling technology, 800Gb is actually a combined 400Gb which is also actually a combined 100Gb (2 x 4 x 100 lanes). So far signaling technology has gone from 1Gb to 10Gb to 25Gb to 100Gb and you can have situations where 100Gb can be either 4x 25Gb or 1x 100Gb based.

 

For all the different standards of QSFP the Q stands for Quad, none of it makes much sense if you don't know what the Q means.

[CarlBar]

1 hour ago, leadeater said:

In reality the first place that will get deployment of 800Gb interfaces will be top of rack switches to distribution switches, between switches first. Servers with 800Gb interfaces are very much ages away as even in the highest performance realms 100Gb is still the go to not 400Gb. Right now Mellanox only has 200Gb cards per port max.

 

Mellanox does have 16 port 400Gb switch ICs (ASICs), Spectrum-2, but those are designed to go in to switches not on NICs.

 

The important thing to note however is this is not an upgrade to the signaling technology, 800Gb is actually a combined 400Gb which is also actually a combined 100Gb (2 x 4 x 100 lanes). So far signaling technology has gone from 1Gb to 10Gb to 25Gb to 100Gb and you can have situations where 100Gb can be either 4x 25Gb or 1x 100Gb based.

 

For all the different standards of QSFP the Q stands for Quad, none of it makes much sense if you don't know what the Q means.

 

Oh absolutely. I was more commenting on weather it was even theoretically possibble for a single server to utilize that kind of bandwidth. Like any extreme end tech though actual good use cases for it will tend to be specialised or a logistics scale thing.

[mbntr]

but how? PCIe x16 4.0 has less bandwidth

[cj09beira]

thats more than 2 times as fast as my system's ram, nice

wonder what they will use to connect this to a server, might even be gen-z,

btw its been years and almost no talk of gen-z that i heard of, hope it comes soon

[Franck]

7 hours ago, williamcll said:

That's nearly 3 times as fast as a PCIe4 x16, do you plug the network card into the DIMM slot or what?

nope but gone are the days of all these standard. Your motherboard will now only have 1 giant set of Ethernet port that you plug your ram, video card, storage into it.

One standard to rule them all.

[jasonvp]

All I can say is: bleah.  Another useless stepping stone to where we want the actual next Ethernet spec to be: Tbit/sec.  Ethernet should be staying on its "order of magnitude" growth, but past 10Gbit/sec that became... hard.  So Cisco "tricked" us all into going for 40Gbit/sec because 100Gbit/sec was too far away and/or too difficult to do at the moment.  100Gbit/sec gets here and what's next?  Can we stretch it till the "order of magnitude" jump?  Tbit/sec?  .... nope.  400Gbit/sec.

 

And now this.  Ugh, what a waste.

 

Fatter pipes are almost always a good thing in the data centers and backbones, and that's where this spec will be utilized; not in your home or on end servers for quite some time.  As lead pointed out, it'll require PCI-E 5.0 before a server could handle that kind of speed through a single card.

 

The issue is timing and cost.  This is another thing for manufacturers to waste their time, money, and engineering on (same as they did with 40Gbit/sec), until we finally get to Tbit/sec.  And it's coming.  Eventually.  Then all of the switches that were purchased w/800Gbit/sec interfaces will be insta-obsolete.

[rcmaehl]

7 hours ago, williamcll said:

That's nearly 3 times as fast as a PCIe4 x16, do you plug the network card into the DIMM slot or what?

Network interface is directly connected to the CPU die to prevent latency.

[descendency]

Standardizing things like this are important, because companies outside of Google, Facebook, Netflix, and others need fast data-centers. 

[LAwLz]

6 hours ago, jasonvp said:

All I can say is: bleah.  Another useless stepping stone to where we want the actual next Ethernet spec to be: Tbit/sec.  Ethernet should be staying on its "order of magnitude" growth, but past 10Gbit/sec that became... hard.  So Cisco "tricked" us all into going for 40Gbit/sec because 100Gbit/sec was too far away and/or too difficult to do at the moment.  100Gbit/sec gets here and what's next?  Can we stretch it till the "order of magnitude" jump?  Tbit/sec?  .... nope.  400Gbit/sec.

 

And now this.  Ugh, what a waste.

 

Fatter pipes are almost always a good thing in the data centers and backbones, and that's where this spec will be utilized; not in your home or on end servers for quite some time.  As lead pointed out, it'll require PCI-E 5.0 before a server could handle that kind of speed through a single card.

 

The issue is timing and cost.  This is another thing for manufacturers to waste their time, money, and engineering on (same as they did with 40Gbit/sec), until we finally get to Tbit/sec.  And it's coming.  Eventually.  Then all of the switches that were purchased w/800Gbit/sec interfaces will be insta-obsolete.

You're missing the point. This will most likely not be used for servers or the likes, just like 400Gbps isn't used in servers or the likes. It will be used for large backbone connections in data centers. Things like interconnects for spine and leaf switches, where you have tens, hundreds or tousands of servers sending data over the same physical cables.

 

 

And of course we want "1Tbps" connections. Do you really think companies would spend time and effort not just making this spec, but also products that uses this new 800Gbps standard if they could just go to 1Tbps? It's like saying "oh the new 5000mAh battery in the Galaxy Ultra is such a waste because we should just make 50,000 mAh batteries. Battery tech should just be that good".

I don't think any company like Cisco would say no to a 1Tbps spec, but clearly that's not possible in the foreseeable future. So they have to make due with what we can do with our current technology.

 

And do you really think equipment with 800Gbps ports will be "insta-obsolete" just because a new standard comes out? Data center equipment isn't like consumer electronics where you just throw it out as soon as a new thing comes out.

The Nexus 5596 we have in our datacenter are around 8 years old. We're still using them and they still work wonderfully. We're just now starting to phase them out in favor of equipment that supports ACI (C9364C and C93180YC). I am not sure how long we'll stick with these new switches, but I wouldn't be surprised if the plan is to use them for close to 10 years, regardless of what other stuff comes out during that time.

[Lurick]

16 hours ago, LAwLz said:

-snip-

Exactly, I believe the next step after 800Gbps is 1.6Tbps anyways for the Ethernet Alliance Roadmap and then from there 3.2Tbps and so on.

40G also isn't obsolete either even though 100G and 400G gear is out there although it's definitely not sold as much now than 100G gear is out which can also negotiate to 40G with a QSFP swap. Sure 40G will eventually migrate away to 25G and 50G but it still has it's place

[Doobeedoo]

Good stuff. I'm still dreaming of fiber connection one day and get away from measly 10Mbps/0.5 up/down though. 

[StDragon]

So a use case for this? Think NVMeOf (NVMe over Fabrics). Basically a much lower latency protocol compared to iSCSI primarily (if only) used for NVMe flash storage arrays and to to supplement tiered storage.