100 Gbps Network Speeds

[EricTrujillo]

So I was at work today and an interesting thought came into my head. Obviously 10 Gbps is possible at the moment and its not out of reach, but how about 100 Gbps?

 

I've seen networking equipment around the web that supports it, and although its very expensive and purely fiber optic at this point, what else would bottle neck the system? Are there storage devices or protocols out there written to read/write at those speeds? Setting price aside, how difficult would it be to achieve such a thing?

[Lurick]

400Gbps is just around the corner and will be on the market this year. GPU based systems can easily saturate 100Gbps links for number crunching and certain tasks that are oriented to take advantage of GPUs. I'm working on a small project now for a customer, as part of a larger engagement, where they need lossless 100Gbps links between all their GPU clusters.

 

It's not going to be cheap but I would say that at current prices it's not crazy expensive, depending on what you need. For a switching platform with some decent L3 support they are generally around $1000 or so per 100Gbps port which is pretty nice. Scale up the feature sets needed and you can go much higher but for a lot of fast scale data centers that just need switching and some L3 capabilities, $1000 or less per port is pretty nice.

[suchamoneypit]

Using the internet at these speeds is probably unreasonable at this time, but datacenters and niche scenarios like NASA can have some pretty insane speeds, but primarily these speeds are used locally in something like transferring massive amounts of data to and from servers.

[JCBiggs]

100gb is possible with fiber.  you just have to have the memory speed to support it.  Hard drive more than anything. 

 

 i went too google to do the conversion, and this is what i found... thats a new one on me google.  (yes apparently thats a real thing, but i was confused....dyslexai)

[Lurick]

3 minutes ago, suchamoneypit said:

Using the internet at these speeds is probably unreasonable at this time, but datacenters and niche scenarios like NASA can have some pretty insane speeds, but primarily these speeds are used locally in something like transferring massive amounts of data to and from servers.

Datacenter interconnects are often times going to use bundles of 100GbE as well,  ISP backbone networks have a lot of 100GbE and I know of several companies that do it at their larger headend sites and locations too. They don't do it from each branch but they've got a couple 100Gb or a couple 40Gb connections to the ISP on their headend aggregation stuff.

[orbitalbuzzsaw]

10Gb/S is common in the enterprise space

[EricTrujillo]

13 minutes ago, Lurick said:

400Gbps is just around the corner and will be on the market this year. GPU based systems can easily saturate 100Gbps links for number crunching and certain tasks that are oriented to take advantage of GPUs. I'm working on a small project now for a customer, as part of a larger engagement, where they need lossless 100Gbps links between all their GPU clusters.

 

It's not going to be cheap but I would say that at current prices it's not crazy expensive, depending on what you need. For a switching platform with some decent L3 support they are generally around $1000 or so per 100Gbps port which is pretty nice. Scale up the feature sets needed and you can go much higher but for a lot of fast scale data centers that just need switching and some L3 capabilities, $1000 or less per port is pretty nice.

 

It’s theoretical at this point. I would be using it for large file transfers. I was thinking more of how easy it could be done. My main concern is finding a storage device that could read and write that fast. I know it can be done with a couple of cards, some fiber optic cable and a capable switch.

[Lurick]

6 minutes ago, EricTrujillo said:

 

It’s theoretical at this point. I would be using it for large file transfers. I was thinking more of how easy it could be done. My main concern is finding a storage device that could read and write that fast. I know it can be done with a couple of cards, some fiber optic cable and a capable switch.

For file transfers you're going to be hard pressed to hit anywhere near that unless you've got one or two dozen really solid SSDs in like a RAID 10 or RAID 0 probably unless you want to pay a TON of money.

[EricTrujillo]

7 minutes ago, Lurick said:

For file transfers you're going to be hard pressed to hit anywhere near that unless you've got one or two dozen really solid SSDs in like a RAID 10 or RAID 0 probably unless you want to pay a TON of money.

Something like a PCI Express SSD? I know SATA 3.2 has a 16 Gbps limit but I haven’t seen any actual devices with that spec in it.

[Lurick]

1 minute ago, EricTrujillo said:

Something like a PCI Express SSD? I know SATA 3.2 has a 16 Gbps limit but I haven’t seen any actual devices with that spec in it.

Even most PCIe SSDs are going to be limited to, in most cases, PCIe 3.0 x4 speeds, so you'll need several of them as well. If you can get a solution that uses several and puts them into a single PCIe 3.0 x16 slot then you'll be able to utilize a 100GbE link for transfers since that will give you over 100Gb of bandwidth.

[Mornincupofhate]

This can easily be done with Linux, Using Intel's DPDK. It's an extremely fast networking stack.

[beersykins]

3 hours ago, Mornincupofhate said:

easily be done

I assume you don't factor expense into that?

 

But otherwise you can get some off the shelf stuff for 100 GbE.  There's some 40 gig equipment like Arista switches you could roll used for under $1k.  Main bottlenecks are generally storage or routing capacity, depends on what you're trying to do.

[Lurick]

7 hours ago, beersykins said:

I assume you don't factor expense into that?

 

But otherwise you can get some off the shelf stuff for 100 GbE.  There's some 40 gig equipment like _Arista switches you could roll used for under $1k.  Main bottlenecks are generally storage or routing capacity, depends on what you're trying to do.

Let's not bring that evil "A" word into the mix now

[EricTrujillo]

6 hours ago, Lurick said:

Let's not bring that evil "A" word into the mix now

I'm not very familiar with them, do they just have garbage products or something?

[dalekphalm]

20 hours ago, JCBiggs said:

100gb is possible with fiber.  you just have to have the memory speed to support it.  Hard drive more than anything. 

 

 i went too google to do the conversion, and this is what i found... thats a new one on me google.  (yes apparently thats a real thing, but i was confused....dyslexai)

I just wanted to make a clarification.

 

100Gbps = 100 Gigabit per second, not GigaByte (or GibiByte, for that matter).

 

Still a huge number, but much smaller than what you've listed.

 

100 Gbps = 11,920.9 MiB/s (12,500 MB/s)

[Mornincupofhate]

16 hours ago, beersykins said:

I assume you don't factor expense into that?

 

But otherwise you can get some off the shelf stuff for 100 GbE.  There's some 40 gig equipment like Arista switches you could roll used for under $1k.  Main bottlenecks are generally storage or routing capacity, depends on what you're trying to do.

Expenses for what? Installing a new open source networking stack?

I wasn't talking at all about hardware in my previous post.

[Mikensan]

Outside of data centers and normal ISPs, you have deep sea fiber where the combined speeds are measured in terabit/s. Granted it is all aggregated so not a single connection, but damn impressive still.

[beersykins]

1 hour ago, Mornincupofhate said:

Expenses for what? Installing a new open source networking stack?

I wasn't talking at all about hardware in my previous post.

It doesn't happen without hardware my man.

[Franck]

On 8/13/2018 at 6:36 PM, Lurick said:

Datacenter interconnects are often times going to use bundles of 100GbE as well,  ISP backbone networks have a lot of 100GbE and I know of several companies that do it at their larger headend sites and locations too. They don't do it from each branch but they've got a couple 100Gb or a couple 40Gb connections to the ISP on their headend aggregation stuff.

Indeed they have large backbone. Distributor of fiber TV uses 1.45 gbps for 1 channel at 1080p and a 4 1080p signal for 4k which total's up to 5.8 gbps that is input from the tv station to their server. Then they compress and redistribute that signal over land lines and satellite. At over 230 channels (from one specific provider i know the guy that actually work there) that is a huge bandwidth coming in their system. That become even crazier especially when TV shows are live then the data cannot be process prior to diffusion, it has to be done live so their equipment definitively support very high bandwidth.

[schizznick]

We are installing a 100gbps Ethernet ring using Nokia... it's dang fast but the cost is high. Our 100gbps CSFP's are around 14k a piece for 10km optics. The 40km optics are roughly double that. 

 

 

 

This is the 10km CSFP optic we use for the shorter links. You can see they are quite a bit larger than SFP+ optics for 10gbps. 

 

This is the interface info showing the 100gbps. 

 

We are using this as our backbone for all services. Internet will be the largest consumer of bandwidth. But even that in most cases won't be close to saturating this connection anytime soon. Even our data center is mostly 10gbps.