Networkng 10Gb vs 40Gb -- Why is 40Gb cheaper???

[anakin61]

I am looking to upgrade my home network to 10Gb. I have fiber internet and am going to 2.5Gb + have a NAS that will be upgraded to 10Gb.

 

My question is why is it that some 40Gb gear is way cheaper than 10Gb gear?

 

I found a Mellanox IS5030 InfiniBand for $300 CAD including shipping which is a 36 port 40Gb/s each port...

 

Where if I go look for a 10Gb networking switch, like the TP-Link TL-SX1008, which is $500 CAD and only has 8 ports 10Gb/s each port.

 

What am I missing? I understand that you would have to use QSFP+ instead of Cat 6, but you can get converters that are less than $100 to run Cat 6.

 

Why wouldn't someone get the Mellanox IS5030 InfiniBand for $300 and get a couple converters to 10Gb? Then it would be future proof because in the future you can just revert it back to 10Gb?

[Lurick]

There are no fiber to copper 40G converters out there, the only thing you can do is get QSFP to SFP+ adapters and SFP+ copper is limited to 30 or 80m depending on how much you spend and that of course is only 10G not 40G

[Electronics Wizardy]

Infiniband isn't based of the IP protocol that the internet uses, it has its own way of sending data. You can use IPoIB to send normal ip packets, but that requires more setup, and performance might not be great often.

 

Im pretty sure you can't just plug and play ethernet into that switch, so you can't just use cat6 cables or qsfp to sfp adapters. Its also a managed switch, so I think you have to connect via serial to get it setup so you can use the management port.

 

The other thing is these switches are loud and hot. The power will make up for the cheaper switch likely, and you won't want to be in the same room as the switch.

 

 

 

 

[anakin61]

@Electronics Wizardy Thank you! I wasn't too sure what InfiniBand was but looked it up after you send that reply. That makes a lot more sense!

 

@Lurick You're right, I did some research and found that I would have to get a breakout cable 1x QSFP+ (optical) to  4x SFP+ (optical) and just run OM3 right to my computer.

 

However, this wouldn't fix the inherent problem of the managed network switch and the possibility that Infiniband would inhibit me going to WAN.

Here is a great article on the problem: https://blog.patshead.com/2016/04/infiniband-an-inexpensive-performance-boost-for-your-home-network.html

 

Looks like you can get it to work with a bit of work. However, I don't think I have the experience/knowledge to make it work.

 

 

 

[Lurick]

4 minutes ago, anakin61 said:

@Electronics Wizardy Thank you! I wasn't too sure what InfiniBand was but looked it up after you send that reply. That makes a lot more sense!

 

@Lurick You're right, I did some research and found that I would have to get a breakout cable 1x QSFP+ (optical) to  4x SFP+ (optical) and just run OM3 right to my computer.

 

However, this wouldn't fix the inherent problem of the managed network switch and the possibility that Infiniband would inhibit me going to WAN.

Here is a great article on the problem: https://blog.patshead.com/2016/04/infiniband-an-inexpensive-performance-boost-for-your-home-network.html

 

Looks like you can get it to work with a bit of work. However, I don't think I have the experience/knowledge to make it work.

Yah, doing IP over IB is possible but I don't believe the other way is possible which shouldn't be too much of an issue but still you do have to take that into account. To answer your original question though, even 40G IP switches are pretty dirt cheap as it was mostly just a stopgap solution on the road to 100G+ switching gear so it died out pretty quick once 100Gb gear started rolling out.

[mariushm]

You can get a 32 x 40g QSFP+ switch for $419 US plus shipping : https://unixsurplus.com/quanta-t5032-ly6-switch/

 

They have a 40 x RJ45 10g + 8 x SFP+ 10g  for $1500 : https://unixsurplus.com/quanta-t3040-ly3-40-port-10gb-rj45-8-port-1-10gbe-sfp-1ly3bzz0st3-l3-switch/

if you account for not having to deal with transceivers, dac cables, various fiber lengths, it can be an attractive price. 

But with a bit of search, you should be able to find older Arista or Brocade 10g switches on ebay 

 

For example, 48 SFP+ 10g ports + 4 x 40g QSFP ports :

$850 plus shipping  for  Arista DCS-7050S-64-F 48 Port 10G SFP 4 QSFP 40G Front to Rear Airflow :  https://www.ebay.com/itm/284906558358

Or $1150 plus shipping for the higher model Arista DCS-7280SE-64-R 48 Port SFP 10G 4 QSFP 40G R-F Airflow Network Switch : https://www.ebay.com/itm/284868012970

 

Brocade VDX series may be another option, but I heard they're very noisy and it's hard to change fans on them.

 

BUT ... keep in mind these are NOISY and somewhat power hungry, maybe 100w+ with just a few ports used. In contrast a small 2.5g/5g consumer switch is maybe 5-10w per port.

 

A passive copper 3 meters 40g QSFP+ runs you $50 : https://unixsurplus.com/new-mellanox-3m-passive-copper-cable/

You can get 40g QSFP+ to 4 x SFP+ 10g  DAC cable (2 meters) for 90$:  https://unixsurplus.com/unixplus-breakout-cable-2-meter/

or 5 meter 40g QSFP+ to 4 x 10g LC duplex (to add your own transceivers) for $100 : https://unixsurplus.com/arista-qsfp-40g-sr4-40g-network-cable/

 

40g cards are 50-2000$ depending on how old card is, how many hops you're willing to go through ... dual 10g cards can be bought for under $50

 

[anakin61]

On 8/9/2022 at 11:34 PM, mariushm said:

You can get a 32 x 40g QSFP+ switch for $419 US plus shipping : https://unixsurplus.com/quanta-t5032-ly6-switch/

 

They have a 40 x RJ45 10g + 8 x SFP+ 10g  for $1500 : https://unixsurplus.com/quanta-t3040-ly3-40-port-10gb-rj45-8-port-1-10gbe-sfp-1ly3bzz0st3-l3-switch/

if you account for not having to deal with transceivers, dac cables, various fiber lengths, it can be an attractive price. 

But with a bit of search, you should be able to find older Arista or Brocade 10g switches on ebay 

 

For example, 48 SFP+ 10g ports + 4 x 40g QSFP ports :

$850 plus shipping  for  Arista DCS-7050S-64-F 48 Port 10G SFP 4 QSFP 40G Front to Rear Airflow :  https://www.ebay.com/itm/284906558358

Or $1150 plus shipping for the higher model Arista DCS-7280SE-64-R 48 Port SFP 10G 4 QSFP 40G R-F Airflow Network Switch : https://www.ebay.com/itm/284868012970

 

Brocade VDX series may be another option, but I heard they're very noisy and it's hard to change fans on them.

 

BUT ... keep in mind these are NOISY and somewhat power hungry, maybe 100w+ with just a few ports used. In contrast a small 2.5g/5g consumer switch is maybe 5-10w per port.

 

A passive copper 3 meters 40g QSFP+ runs you $50 : https://unixsurplus.com/new-mellanox-3m-passive-copper-cable/

You can get 40g QSFP+ to 4 x SFP+ 10g  DAC cable (2 meters) for 90$:  https://unixsurplus.com/unixplus-breakout-cable-2-meter/

or 5 meter 40g QSFP+ to 4 x 10g LC duplex (to add your own transceivers) for $100 : https://unixsurplus.com/arista-qsfp-40g-sr4-40g-network-cable/

 

40g cards are 50-2000$ depending on how old card is, how many hops you're willing to go through ... dual 10g cards can be bought for under $50

 

 

Awesome information! Yea I can see how 10g is a much more attractive solution right now. Even 2.5/5g would be pretty good.