10Gig network for HPCs but with a twist

[Sreejit]

So I have to build a HPC cluster with 10 gigabit network but the roadblock is that all the motherboard of the PCs that will be in the network support 1gigabit. Now I can install 10g NICs but all the pcie slots are filled with GPUs (since the GPUs are chonky bois, there is no space on the mobo to place a nic). Can anyone suggest how to solve this problem without sacrificing the GPU performances because the nodes are being used in machine learning training?

Details on the nodes :

node1 -> Gigabyte x299 wu8 mobo with 4 2080ti's

node2 -> Asrock x399 Taichi mobo with 4 2080s's

node3 -> Aorus trx40 mobo with 3 3090's

 

[Kilrah]

Risers?

[Sreejit]

Just now, Kilrah said:

Risers?

I have got a set of 1to4 pcie riser kit, but I guess they might hamper my GPU performance cause they are 1x to 16x and usb 3.0 (3.1 gen1).
I ought to try that out yet though.

[Guest]

As @Kilrah suggested, riser is the best course of action. I have a dual 10 gig Intel NIC and USB 3 Gen 2 card running off of a bifurcated x16 to dual x8 riser cable and it's been fine. Your x1 riser will def. hurt performance though.

[KnotRolls]

Not usually recommended but with no more PCIe capacity and only 1 Gig onboard it's an option to go a pair of USB-C > 5 Gig nic adapters and teaming them.

Don't know of any 10 Gig USB adapters but there's enough bandwidth in the spec to do it.

 

Look at something like an Aquantia (Marvell) AQC112U. Marvell AQtion AQC111U & AQC112U

 

That said USB NICs can cause issues of their own but it's worth a shot in this situation.

[Kilrah]

I was suggesting leaving the GPUs in as normal, but using a riser to "move" one of the slots you can't access somewhere else for the NIC, you might have enough room to plug the riser into a slot that's under a GPU cooler and plug the NIC at the other end.

 

Obviously that's supposing you do have enough slots, lane configuration works out well etc. 

Could be a riser that comes from an m.2 slot too.

 

If none of these work then there's not exactly a magical solution other than giving up on a GPU or replacing hardware with something more suitable to the task.

[Sreejit]

19 minutes ago, Den-Fi said:

As @Kilrah suggested, riser is the best course of action. I have a dual 10 gig Intel NIC and USB 3 Gen 2 card running off of a bifurcated x16 to dual x8 riser cable and it's been fine. Your x1 riser will def. hurt performance though.

Can you suggest some good x16 risers?

[Sreejit]

4 minutes ago, KnotRolls said:

Not usually recommended but with no more PCIe capacity and only 1 Gig onboard it's an option to go a pair of USB-C > 5 Gig nic adapters and teaming them.

Don't know of any 10 Gig USB adapters but there's enough bandwidth in the spec to do it.

 

Look at something like an Aquantia (Marvell) AQC112U. Marvell AQtion AQC111U & AQC112U

 

That said USB NICs can cause issues of their own but it's worth a shot in this situation.

well, I need 10g and thunderbolt supports 10g but i dont think usb-c supports 10g

[Sreejit]

7 minutes ago, Kilrah said:

I was suggesting leaving the GPUs in as normal, but using a riser to "move" one of the slots you can't access somewhere else for the NIC, you might have enough room to plug the riser into a slot that's under a GPU cooler and plug the NIC at the other end.

 

Obviously that's supposing you do have enough slots, lane configuration works out well etc. 

Could be a riser that comes from an m.2 slot too.

 

If none of these work then there's not exactly a magical solution other than gigving up on a GPU or replacing hardware with something more suitable to the task.

I have thought about that too, but since the risers are usb 3.0, I dont think I can get 10g from them

[Kilrah]

The risers aren't USB3, some just repurpose USB3 connectors to pass PCIe lanes directly. What matters is to have enough lanes in them. 

[KnotRolls]

10 minutes ago, Sreejit said:

well, I need 10g and thunderbolt supports 10g but i dont think usb-c supports 10g

Yeah for sure with thunderbolt but I'm fairly sure on your X299 the thunderbolt add on requires using a PCIe slot that you don't have spare.

[Sreejit]

7 minutes ago, KnotRolls said:

Yeah for sure with thunderbolt but I'm fairly sure on your X299 the thunderbolt add on requires using a PCIe slot that you don't have spare.

Exactly

[Sreejit]

12 minutes ago, Kilrah said:

The risers aren't USB3, some just repurpose USB3 connectors to pass PCIe lanes directly. What matters is to have enough lanes in them. 

Oukay, understood.