40 Gb/s Upgrade?

[Drjack6]

My network...

 

10Gb/s switch

Custom built FreeNAS file server

Couple of custom built workstations

A Mac with thunderbolt 3 

 

ive been looking at 40Gb/s networking because my server and workstation storage are both capable of over 4 GB/s. (so 40 Gb/s) Ive found switches, pci-e cards, and cables. But im very unfamiliar with it. 

 

My question...

Is it as easy as it looks? 

Do i just buy the equipment, plug it in and enjoy the speed or am i missing something?

Any advice or educationis appreciated!

[Oshino Shinobu]

You need to check driver support for the NICs. FreeNAS is pretty lacking for NIC driver support outside of Intel NICs. From what I've seen of technologies like Infiniband, getting it to work with FreeNAS is hit and miss at best. 

 

EDIT: It also really bugs me how the FreeNAS dev team seems to have said they have no intention of adding support for Asus's XG C100C 10G NIC. The lack of well documented support for high bandwidth NICs is one of the few things that bugs me with FreeNAS. 

[CyberFern0]

21 minutes ago, Drjack6 said:

My network...

 

10Gb/s switch

Custom built FreeNAS file server

Couple of custom built workstations

A Mac with thunderbolt 3 

 

ive been looking at 40Gb/s networking because my server and workstation storage are both capable of over 4 GB/s. (so 40 Gb/s) Ive found switches, pci-e cards, and cables. But im very unfamiliar with it. 

 

My question...

Is it as easy as it looks? 

Do i just buy the equipment, plug it in and enjoy the speed or am i missing something?

Any advice or educationis appreciated!

40000 megabytes per second?!?!?

 

I didn't even know that you can get over 1 GB/s!

[Oshino Shinobu]

1 minute ago, CyberFern0 said:

40000 megabytes per second?!?!?

 

I didn't even know that you can get over 1 GB/s!

No, 40,000 Megabits (or 40Gb/s) per second. It's 5 Gigabytes per second. 

[unkn0wn1]

I'm assuming you are planing to use infiniband for this upgrade, which apparently fre-bsd has support for. according to this http://acadix.biz/infiniband.php

[Drjack6]

 

10 hours ago, unkn0wn1 said:

I'm assuming you are planing to use infiniband for this upgrade, which apparently fre-bsd has support for. according to this http://acadix.biz/infiniband.php

Ok well ive looked up the difference between 40Gbe and Infiniband. I could do regular 40Gbe. Would i run into the same possible comptability issues? I want to stick with FreeNAS but thanks for the tip on FreeBSD. 

[leadeater]

20 hours ago, Drjack6 said:

 

Ok well ive looked up the difference between 40Gbe and Infiniband. I could do regular 40Gbe. Would i run into the same possible comptability issues? I want to stick with FreeNAS but thanks for the tip on FreeBSD. 

FreeNAS under the hood is FreeBSD. Stick with Ethernet if you can since it's much easier to add in a switch and not have to worry about needing to have a gateway device that can bridge Ethernet and InfiniBand.

[CyberFern0]

On 1/13/2018 at 9:35 PM, Oshino Shinobu said:

No, 40,000 Megabits (or 40Gb/s) per second. It's 5 Gigabytes per second. 

Which company!?!?!?

[Lurick]

3 minutes ago, CyberFern0 said:

Which company!?!?!?

?

Just about every networking company out there has 40Gb switches/routers. Most also have 100Gb gear, and soon 400Gb gear.

The price you'll pay for this stuff varies greatly depending on the port density, software, features, quality, etc.

[CyberFern0]

Just now, Lurick said:

?

Just about every networking company out there has 40Gb switches/routers. Most also have 100Gb gear, and soon 400Gb gear.

I always thought that 1gb/s was the highest.

[Lurick]

5 minutes ago, CyberFern0 said:

I always thought that 1gb/s was the highest.

For consumer grade internet and most consumer products, that's generally about the fastest you'll find for now, Comcast does offer 2Gb/s right now as well.

There are small pockets in the US that sell 10Gb to consumers and most ISPs sell up to 100Gb to large companies who pay at least tens of thousands of dollars for it.

 

For switching and routing speeds beyond typical consumer grade you have FAR faster speeds than 1Gb/s. Heck, I have 10 fiber QSFP optics sitting on my desk that do 100Gb/s each.

 

 

 

Edit:

I do want to mention that most of what's being talked about is for local networking, not internet connectivity, although you can get 100Gb internet speeds, it's not consumer grade stuff, and is nowhere close to being cheap

[CyberFern0]

19 minutes ago, Lurick said:

For consumer grade internet and most consumer products, that's generally about the fastest you'll find for now, Comcast does offer 2Gb/s right now as well.

There are small pockets in the US that sell 10Gb to consumers and most ISPs sell up to 100Gb to large companies who pay at least tens of thousands of dollars for it.

 

For switching and routing speeds beyond typical consumer grade you have FAR faster speeds than 1Gb/s. Heck, I have 10 fiber QSFP optics sitting on my desk that do 100Gb/s each.

 

 

 

Edit:

I do want to mention that most of what's being talked about is for local networking, not internet connectivity, although you can get 100Gb internet speeds, it's not consumer grade stuff, and is nowhere close to being cheap

what companies sell 1gbps?

[JoostinOnline]

I pay $65 a month for 45mbps.

 

On 1/13/2018 at 9:09 PM, Drjack6 said:

4 GB/s. (so 40 Gb/s)

That's 32Gbps, not 40.

[Lurick]

Just now, CyberFern0 said:

what companies sell 1gbps?

Spectrum, Comcast, AT&T and Verizon are the four big ones I know about that sell gigabit internet.

There are also a bunch of municipal broadband providers along with smaller companies that sell gigabit as well.

[Oshino Shinobu]

2 hours ago, CyberFern0 said:

I always thought that 1gb/s was the highest.

Bear in mind this is local networking, not internet connections. 

 

Even for internet connections, 1Gbps isn't the fastest, but it's generally the limit for consumer connections. Consumers can easily get over 1Gbps for local networking

[Tomasz Wachla]

Why not adding a IB switch? The used ones should be available, so saturating 40gbit should be relatively easy. 

 

Another thing, why not use full potential of IB and not test using bunch of e.g. graphic cards connected to one device on a network on a NUC or something laptop like. I would be happy to help out design that.