Setting up 40gbE link between 2 PCs

Nemesis158 · September 13, 2019

So I recently discovered some 40gb dual port QSFP cards on ebay for $13 and decided i wanted to try out getting a link set up between my main pc and my server box. 2 cards and a cable cost $40 total so why not?

The cards I ordered were Sun Oracle OEM Connectx-3's. i had them both on a test bench and flashed them both to the latest CX354a-FCBT firmware to get 40gbE.

I currently have one card in my main PC and the other one on the test bench so i can figure out how to configure them before i take down my server to install the card.

I have the latest available WinOF drivers from mellanox on both machines, Both cards are set to ETH for both ports, both are showing 40gbE links and showing sent/recieved traffic in status windows, and I have link and activity lights on both cards.

But i cannot get the Two PCs to talk to each other and i am not sure what i am doing wrong. I can't get either machine to ping the other.

I have tried leaving them on autoconfig IPs, and setting manual IPs. both machines are running windows 10.

I am not at home right now but I can try to provide additional information if any is needed.

Any help with this would be appreciated.

Slayerking92 · September 13, 2019

You are most likely missing the network setup of this guide. It should be the same for ethernet as a 40gbe connection, except the cable is different. as long as you have links on both computers.

How to Connect Two Windows PCs with a LAN Cable

1. Go to “Control Panel -> Network and Internet -> Network and Sharing Center -> Change Adapter Settings.”

2. Click on “Change Adapter Settings.” This will reveal different connections. Select the appropriate connection for your LAN. Usually, but not necessarily, the connection will be called Ethernet and have the description network cable unplugged.

Right-click on the connection and select “Properties.” The local area’s connection properties window will appear.

3. Under the network tab select “Internet protocol version 4 (TCP/IPv4),” then click on “Properties.”

In the Properties windows set the IP address and subnet masks of the first computer to:

IP – 192.168.0.1
Subnet Mask – 225.225.225.0

Repeat all the above steps for the second computer, and set the IP address and subnet Mask as follows:

IP – 192.168.0.2
Subnet Mask – 225.225.225.0

Note: it is important to ensure that the last values of the IP address for both computers are different.

4. The next step is to connect the crossover cable to the network ports of the two computers. Both ends of the network cable look the same, so it doesn’t matter which end you use first. The network ports look something like the image below. (Do note that most modern ~~computers~~ laptops don’t come with a LAN port.)

5. Right-click on “This PC,” and choose “Properties.” Click on “Change settings -> Change.” This reveals the window with the name of the work group. The value for the workgroup name should be the same for both PCs. By default, the workgroup name will be WORKGROUP, but you can change it to any name you like.

6. Right-click on the drive you want to share. Scroll to the “Give access to” option and click “Advanced Sharing.” Under the sharing tab, click the “Advanced Sharing” button.

This reveals the advanced sharing window. Check the “Share this folder” checkbox, and click “Apply -> OK.”

At this stage, you will have successfully connected the two Windows computers to share your drives between them.

Nemesis158 · September 26, 2019

I ended up getting help elsewhere but the issue was being caused by Windows Firewall. I have set the main network connections to private and leave the windows firewall on for private networks. the direct connection is set as a public network and windows firewall is disabled for public networks. I then had to set jumbo frame MTU to get 10Gb/s over the link