Home Rack Network/Interconnects

NorthernWing · January 9, 2018

Evening, everyone -

So I'm in the process of building up one server and a storage/archive box for myself, both of which are going to be stuck in a 36U Middle Atlantic rack cabinet. More will prolly follow... It sits beside my desk, and I'm probably gonna rebuild the desktop into a 4U case too, eventually. I host a few things for twenty or so people on a gigabit fiber internet connection, and decided I need an actual machine to do so.

I can't decide how I want to network these together. I'll regularly be making large transfers between all three+ systems and was considering setting up a network of surplus QDR Infiniband cards, but I don't have a clue how I'd be able to set up the HDA on my desktop environment. Have never dealt with IB before. Fibre wouldn't be worth it over just plain ethernet unless there's a way to gang FC connections I'm not aware of similar to IB. Ten gig ethernet is still prohibitively expensive.

Any thoughts on what the fastest interconnect is that I could fire up in this cabinet out of any new or surplus enterprise hardware? All I really need is the ability to map a network drive/build an SAN/whatever the current buzzword is. What would ~$200/box be able to drive, excluding cables/SFP modules?

Just looking to brainstorm here... Let me know what you think!

leadeater · January 9, 2018

7 hours ago, NorthernWing said:

I can't decide how I want to network these together. I'll regularly be making large transfers between all three+ systems and was considering setting up a network of surplus QDR Infiniband cards, but I don't have a clue how I'd be able to set up the HDA on my desktop environment. Have never dealt with IB before. Fibre wouldn't be worth it over just plain ethernet unless there's a way to gang FC connections I'm not aware of similar to IB. Ten gig ethernet is still prohibitively expensive.

Go on to ebay and have a look at Mellanox Connect-X2 10Gb NICs, they are really cheap, with them get SFP+ Direct Attached Copper cables and direct connect the required systems. This is the same as you were planning to do with IB but the cost is the same, it's even the same cards with different firmware, but easier to use as it's standard Ethernet.

Later on if you need to you can buy a used 10Gb switch off ebay and migrate from direct connect mesh to a switched network.

NorthernWing · January 10, 2018

I was just reading through Mellanox's support archives at work today - I see that ConnectX-3 MCX354A-QCBT cards can be had for $60 NOS on eBay and evidently support 40Gb/s IB and 10GBaud Ethernet. Sounds like I'll need to find a pair of QSFP+ xceivers/passive copper. I think I'm going to keep pulling for IB just for experience... if I wanted something simple I'd just slap some Intel X520s in there and call it a day.

I've been looking at 40G Mellanox switches (MTS3600/Infiniscale IV), but anything in decent condition still costs as much as the rest of this server.

Two questions. First, is auto bitrate negotiation an IB spec or vendor specific? I know I'd read that ConnectX adapters do so with other Mellanox endpoints, but I don't know whether that's their marketing fluff or if it's actually something specific to Mellanox cards.

Secondly, I understand IB lanes are ganged for increased throughput, with QDR running at 10G/lane, and so 40G/QSFP+. How exactly is this handled in the software side? If I've got one HBA with two QSFP+ sockets, does that mean I have eight lanes (2x4) that can be ganged? I assume this isn't the case, and especially not across multiple cards, but I can't find anything on actually setting up IB past the 'plug it in and install drivers' explanations online while I wait for hardware. I suspect what I'm describing (SMB Multichannel for ethernet) doesn't apply here, but any thoughts?

leadeater · January 10, 2018

39 minutes ago, NorthernWing said:

Secondly, I understand IB lanes are ganged for increased throughput, with QDR running at 10G/lane, and so 40G/QSFP+. How exactly is this handled in the software side? If I've got one HBA with two QSFP+ sockets, does that mean I have eight lanes (2x4) that can be ganged? I assume this isn't the case, and especially not across multiple cards, but I can't find anything on actually setting up IB past the 'plug it in and install drivers' explanations online while I wait for hardware. I suspect what I'm describing (SMB Multichannel for ethernet) doesn't apply here, but any thoughts?

Basically all of that is handled in hardware/firmware, you can actually get QSFP+ breakout cables that have 4 SFP+ connectors on the other end though I have only seen that used to split out a 40Gb switch port to 4 10Gb connections.