Poor man's infiniband

[SpaceGhostC2C]

I need to connect at least 2 computers together at home. In fact, they are already connected through my home router (a.k.a. consumer router+switch), but my question is: can I do anything to achieve a faster communication between computers?

 

Why do I ask? I'm running distributed computational workloads on the PCs, and roughly speaking, jumping from 1 to 2 PCs is like going from RAM to pagefile: you get more, but slower  I've built a cluster for work in the past, using Dell servers/workstations with Infiniband connectivity between them, but that's not the kind of hardware I have at home. However, I can get my hands on cheap, used Ethernet server cards with multiple ports (I already have one of these to play with), and somewhat more expensive optic fiber cards (which I never used). So, my more detailed question is:

 

- Is there anything I can do based on multiple Ethernet ports to achieve faster PC to PC communication?

- Would it be better to use fiber instead? If so, how do I set up a fiber-based network between the two PCs? Is it like Ethernet with different components or do I need some additional conversion somewhere?

 

If it matters, I don't care too much about the outbound connection (to the router, and then the internet), or if only one of the PCs has direct access to the router, etc. As long as there is some way to connect to the internet with at least one machine, all that matters to me is PC to PC speed without huge investments (I will probably buy second hand any additional hardware I may need).

[tt2468]

If they both have extra pcie slots, you could get a few 10Gb nics off of ebay and a DAC. That would give you 10-gigabit communication between both pcs.

[SpaceGhostC2C]

1 minute ago, tt2468 said:

If they both have extra pcie slots, you could get a few 10Gb nics off of ebay and a DAC. That would give you 10-gigabit communication between both pcs.

Thanks for your reply!

What is a DAC in this context? I'm only familiar with audio DACs. How would the setup be like?

 

Regarding 10 Gbit, that seems a natural solution, although it gets pricey. Would something like this work for a PC-to-PC connection, luike one would do with ethernet? I see they call the cable a "DAC" cable, so maybe that answer my previous question...  

[tt2468]

Direct Attach Cable

[leadeater]

2 hours ago, SpaceGhostC2C said:

Thanks for your reply!

What is a DAC in this context? I'm only familiar with audio DACs. How would the setup be like?

 

Regarding 10 Gbit, that seems a natural solution, although it gets pricey. Would something like this work for a PC-to-PC connection, luike one would do with ethernet? I see they call the cable a "DAC" cable, so maybe that answer my previous question...  

Yep what you linked is exactly what @tt2468 was talking about 

[thething55]

Direct attached is one way to get 10 gigabit speeds using SFP+ (small form factor pluggable plus). which I will now call SFP+-DA. If you use Direct attached then you will need SFP+-DA on both devices and a direct attach cable to link the machines together.

It is however important to remember that SFP (without the plus) is only 1 gigabit while SFP+ is 10 gigabit. with SFP+-DA you are looking to get a maximum range of about 15 meters cable length.

If you want a longer distance then you can get modules for SFP+-DA which convert the signal into fibre which depending on the module (these are also called transceiver) can go from 25 meters to 300 meters for a short range module and many thousands of meters for a long range module. Typically theses will be called SFP+-SR for short range and SFP+-LR for long range. Some PCIe cards actually come with SR/LR built in instead of being SFP+DA but personally I think this limits your options in case you want to move things around or install the card somewhere else so I would always go for the DA versions personally.

 

You other option is using 10Gbase-T which is like your traditional twisted pair gigabit network but faster. Like before both sides will need a card but 10gbase-t can be easier to install. standard cat 6 can be around 37 meters and 55 meters for 10gbase-t depending on several factors including the quality of the cable and the environment around it etc. Cat 6A cable however, should be able to run up to around 100 meters for 10 gigabit so if you wanted to go down that route I would go for cat 6a and play it safe.

 

I would like to comment on  infiniband but I don't know no where near enough about it to comment on the use of it. If anyone else knows/you want to do research then you can also look into fibre channel but apart from seeing the link on a server and a SAN I have never actually used it.

[mariushm]

You could go with 2 10 gbps and copper attached cable, Linus has a video about this, some older cards designed to work with optical fiber but you can use some copper cable instead.

 

Alternatively, you could buy older gigabit server network cards, which have up to 4 gigabit ports.  With a suitable switch that supports port trunking / lacp, you can tie all four ports to act as a single 4 gbps port.  eBay has cards with the i350am4 chipset from intel like this one, or cards based on RTL8111 which also support LACP and are a bit cheaper, like this one or this one

It's only 4gbps in total, but you can use cheap cat5e/cat6 patch cables, long ones if you wish. Copper attached cables can be expensive for long length (something like 50$ for 3m) and when you add the price of two cards it gets expensive.

You may not even need a switch between the two computers, if you enable port trunking on both cards they may be smart enough to detect each cable and set it as crossover automatically and work just fine though i can't tell you for sure, i didn't have a need to do this myself yet.

 

[SpaceGhostC2C]

1 minute ago, mariushm said:

You can either go with 2 10 gbps and copper attached cable.

 

Alternatively, you could buy older gigabit server network cards, which have up to 4 gigabit ports.  With a suitable switch that supports port trunking / lacp, you can tie all four ports to act as a single 4 gbps port.  eBay has cards with the i350am4 chipset from intel like this one, or cards based on RTL8111 which also support LACP and are a bit cheaper, like this one or this one

It's only 4gbps in total, but you can use cheap cat5e/cat6 patch cables, long ones if you wish. Copper attached cables can be expensive for long length (something like 50$ for 3m) and when you add the price of two cards it gets expensive.

You may not even need a switch between the two computers, though i'm not sure, i didn't have a need to do this myself yet.

 

Interesting, I was reading about "link aggregation" and wondering if something like this was possible  If I understand your post correctly, though, it won't be enough to have multiple ports, but also a particular technology in the switch - or investigating whether direct connection would work.

 

You have all been very helpful, this is getting interesting. I have to compare the cost of 2x quad port cards and possibly a capable switch with the cost of each 10Gbit alternative explained by @thething55, and the cost difference to the speed difference. Since I already have a dual port card, I may get me another one (~ €8) or a quad port (more expensive) to experiment and see if I can get a direct 2Gbit connection (i.e., whether I effectively get twice the speed / half the communication overhead) before making a final decision.

[leadeater]

17 minutes ago, mariushm said:

You could go with 2 10 gbps and copper attached cable, Linus has a video about this, some older cards designed to work with optical fiber but you can use some copper cable instead.

 

Alternatively, you could buy older gigabit server network cards, which have up to 4 gigabit ports.  With a suitable switch that supports port trunking / lacp, you can tie all four ports to act as a single 4 gbps port.  eBay has cards with the i350am4 chipset from intel like this one, or cards based on RTL8111 which also support LACP and are a bit cheaper, like this one or this one

It's only 4gbps in total, but you can use cheap cat5e/cat6 patch cables, long ones if you wish. Copper attached cables can be expensive for long length (something like 50$ for 3m) and when you add the price of two cards it gets expensive.

You may not even need a switch between the two computers, if you enable port trunking on both cards they may be smart enough to detect each cable and set it as crossover automatically and work just fine though i can't tell you for sure, i didn't have a need to do this myself yet.

 

 

2 minutes ago, SpaceGhostC2C said:

Interesting, I was reading about "link aggregation" and wondering if something like this was possible  If I understand your post correctly, though, it won't be enough to have multiple ports, but also a particular technology in the switch - or investigating whether direct connection would work.

 

You have all been very helpful, this is getting interesting. I have to compare the cost of 2x quad port cards and possibly a capable switch with the cost of each 10Gbit alternative explained by @thething55, and the cost difference to the speed difference. Since I already have a dual port card, I may get me another one (~ €8) or a quad port (more expensive) to experiment and see if I can get a direct 2Gbit connection (i.e., whether I effectively get twice the speed / half the communication overhead) before making a final decision.

 

The main problem Link Aggregation is it is designed for multiple connection/endpoint balancing not one-to-one so only one cable will be used between any two computers no matter how many are used. If the application itself can make use of multiple paths, SMB3 multichannel for example, then this isn't an issue and Link Aggregation isn't even needed.

 

If the two computers are Linux then there is one Link Aggregation method that would actually work, balance-alb, but there is no Windows equivalent anymore. There used to be before in Server 2008 R2 and below with Intel/HP NIC teaming software but that is no longer supported in newer Windows operating systems.

 

Networking is one of those areas where brute for bigger is better applies very strongly.

[SpaceGhostC2C]

41 minutes ago, leadeater said:

 

 

The main problem Link Aggregation is it is designed for multiple connection/endpoint balancing not one-to-one so only one cable will be used between any two computers no matter how many are used. If the application itself can make use of multiple paths, SMB3 multichannel for example, then this isn't an issue and Link Aggregation isn't even needed.

What a buzzkiller  

42 minutes ago, leadeater said:

If the two computers are Linux then there is one Link Aggregation method that would actually work, balance-alb, but there is no Windows equivalent anymore. There used to be before in Server 2008 R2 and below with Intel/HP NIC teaming software but that is no longer supported in newer Windows operating systems.

 

Networking is one of those areas where brute for bigger is better applies very strongly.

They could be. I run Fortran code, so I could compile it for any OS. The cluster I mentioned in the OP ran Debian; installing it taught me everything I know about Linux. There were some rough edges, but I can repeat it for this computers. Or go with older Windows, at least for testing purposes (I guess I could try a VM, but I'm afraid passing the hardware to the VM and then getting it to work there can be more cumbersome than just going Debian - and it's all teh same to me, I can dual-boot if I need Windows for something else).

 

I'm not giving up teaming 2 ports yet, because it's 8 bucks and the fun of it, but maybe it's time to start budgeting those 10Gbits solutions.

[tt2468]

Holy sh*t people seem to really love writing essays now. Just an observation.

[Mr_Flynn]

17 hours ago, SpaceGhostC2C said:

Dell servers/workstations with Infiniband connectivity between them, but that's not the kind of hardware I have at home.

Hell, most companies cant afford that kind of gear (not like they would need it in the first place). You've got to wonder how much of the cost of that switching platform is Mellanox driving up the price because they virtually control the market.

 

Just to put this in perspective for everyone else, a 36 port infiniband switch from Mellanox costs ~$15,000.

[leadeater]

7 hours ago, tt2468 said:

Holy sh*t people seem to really love writing essays now. Just an observation.

Guilty has charged .

 

I just like to explain everything fully, sometimes at the expense of people actually reading it due to length.

[Lurick]

5 hours ago, Mr_Flynn said:

Hell, most companies cant afford that kind of gear (not like they would need it in the first place). You've got to wonder how much of the cost of that switching platform is Mellanox driving up the price because they virtually control the market.

 

Just to put this in perspective for everyone else, a 36 port infiniband switch from Mellanox costs ~$15,000.

Psh, who cares about Infiniband. Just go with a Nexus 9508 with 8x N9K-X9732C-EX linecards to give you 256x 100Gb ports, of course it would cost at least an extra 0 on the end of that number but still, you get all the bragging rights  

[leadeater]

5 hours ago, Mr_Flynn said:

Hell, most companies cant afford that kind of gear (not like they would need it in the first place). You've got to wonder how much of the cost of that switching platform is Mellanox driving up the price because they virtually control the market.

 

Just to put this in perspective for everyone else, a 36 port infiniband switch from Mellanox costs ~$15,000.

True but once the warranty expires they show up on ebay for extremely low prices, old equipment is just cheap no matter how good it actually is .

[SpaceGhostC2C]

17 minutes ago, leadeater said:

Guilty has charged .

 

I just like to explain everything fully, sometimes at the expense of people actually reading it due to length.

Personally, I'm glad to get this kind of replies, even if (or maybe because) I'll have to spend some time chewing them and doing additional research on the things you are telling me  

7 minutes ago, Lurick said:

Psh, who cares about Infiniband. Just go with a Nexus 9508 with 8x N9K-X9732C-EX linecards to give you 256x 100Gb ports, of course it would cost at least an extra 0 on the end of that number but still, you get all the bragging rights  

That's it, forget networking, now I'm going to take out my apartment floor and buld an LHC underneath! 

[porina]

22 hours ago, SpaceGhostC2C said:

Why do I ask? I'm running distributed computational workloads on the PCs, and roughly speaking, jumping from 1 to 2 PCs is like going from RAM to pagefile: you get more, but slower  I've built a cluster for work in the past, using Dell servers/workstations with Infiniband connectivity between them, but that's not the kind of hardware I have at home. 

Sounds like entering the realms of HPC here. Is the application bandwidth limited? Reason for asking is in HPC terms, latency is usually the killer and things like infiniband are better optimised there.

8 hours ago, tt2468 said:

Holy sh*t people seem to really love writing essays now. Just an observation.

Sometimes you need the essay to get enough info over.

[SpaceGhostC2C]

5 minutes ago, porina said:

Sounds like entering the realms of HPC here. Is the application bandwidth limited? Reason for asking is in HPC terms, latency is usually the killer and things like infiniband are better optimised there.

Good question. I assumed so, as I have to transfer large arrays across computers.

 

Since you are into HPC, I'll digress a bit to explain the use case, and maybe that will help you help me by giving you a better idea of bandwidth requirements:

 

I run a Fortran program using MPI to simultaneously run on several cores / PCs. The program solves a dynamic programming problem with continuous state variables by "discretizing" the state space, and finding the solution for a grid of points in the state space in each iteration. each iteration must be done sequentially in a loop, but each point in the state space grid can be solved independently within each iteration. The number of points is several orders of magnitude higher than the number of cores, so it scales pretty well (yes, I should be doing this in GPGPU, but until I learn to do it this thing has to run somewhere ). Hence, at each iteration, each core is solving for a subset of points in the grid. Once they're done, they have to gather all the pieces from all cores and store them in a unified array containing the full solution for that iteration. They need the full solution since the solution for any point in iteration i depends on the full solution in iteration i-1.

 

Correct me if I'm wrong, but my understanding is that bandwidth will be the limiting factor if the chunks being "traded" are large enough? Or there is no way my puny vectors can saturate normal ethernet, and it's all about latency anyway?

 

In case it matters, here's a plot of a toy experiment in the cluster with infiniband mentioned in my first post. Each node had 8 cores, so you can see the jump in times when going from 8 (1 PC) to 9 cores (2 PCs), but then continues to scale (I didn't plot the scaling, but up to 8 it was practically 1/n).

Spoiler

 In that example, it was optimal to stop at 8, but the bigger boys continued to take advantage of additional cores despite the additional overhead up until we ran out of nodes  What I want to avoid in my home version is to have a larger 8-to-9 jump that limits my options to 8.

[porina]

4 hours ago, SpaceGhostC2C said:

Since you are into HPC

I'm actually not seriously into HPC. Well, I looked into it in the past, but my compute needs don't need the faster interconnect so I stopped looking once I saw the pricing. It was just a thought you might want to make use of whatever tools you have available to monitor network usage and decide how to optimise from there.

[Mr_Flynn]

5 hours ago, leadeater said:

True but once the warranty expires they show up on ebay for extremely low prices, old equipment is just cheap no matter how good it actually is .

A lot of the EDR+ infiniband gear is too new to find used versions yet. You can definitely find older gear for very good deals, though.