DragonQuiz

Install the two cards in your system, ours sadly came with no PCI slot covers, however we don’t recommend using the cards without a bracket, so you might have to get creative.
For the purposes of this video, our setup will be a “Gaming-PC” and a “NAS-PC”
Boot up systems and head to windows device manager, and under “Network adapters” you should see “Mellanox ConnectX-2 IPoIB Adapter”
If you don’t see them now, you may after installing the drivers, if you still don’t, you may have a faulty card
Head to the Mellanox OFED for Windows page (http://www.mellanox.com/page/products_dyn?product_family=32&menu_section=34)
For the ConnectX-2 series the drivers are hidden under “Archive Versions”, so we didn’t immediately find anything, and the ConnectX-2 cards aren’t listed in the recommended versions, scaring me a bit.
Were going to install version 4.95 and then update to 5.10 which seemed to be the most reliable install
WinOF Download>Archive Versions>5.10>Windows Client>10
WinOF Download>Archive Versions>4.95>Windows Client>8.1
Install the 4.95 drivers and once done, immediately install with the 5.10 drivers
Now, technically our cards should have had their firmware updated with the driver install, but this didn’t seem to happen
We will need Mellanox Firmware Tools and the firmware file for our specific cards to update them
Download and Install MFT for Windows
Download the firmware file
In our case was for the MHQH19B-XTR
Open an elevated command prompt
Get your device name by running mst status
Format: mt<id>_pci_cr0
CD to the directory containing the firmware file
In our case was cd C:\User\6950X\Desktop\firmware
Run flint -d <Device Name> -i <Firmware File>.bin burn
In our case was flint -d mt26428_pci_cr0 -i fw-ConnectX2-rel-2_9_1000-MHQH19B-XTR_A1-A3.bin burn
Re run the version 5.10 driver installer choosing the “repair” option
Follow these steps on both machines
We now have functioning Infiniband NICs, but no subnet manager (which assigns an ID to the NICs, and creates a routing table based off of that)
I’m going to setup the subnet manager on our “NAS-PC” to keep any extra load off the “Gaming-PC”
Now, if you want, you can open up command prompt and run the OpenSM executable every time you turn on your PC, but that's way too much work.
Elevated Cmd>C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe
So, with the magic of Windows services, we can make it start on login in the background!
Start by opening cmd as administrator (Right Click Windows Icon in Bottom Left>Command Prompt [Admin])
Enter sc create <SERVICE NAME> binPath= “<FILE PATH TO OPENSM> --service” start=auto
Replacing <SERVICE NAME> with your desired name, we chose OpenSM
And inputting the file path which in our case is
C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe
Which equated to
sc create OpenSM binPath= “C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe --service” start=auto
MAKE SURE, you have a space after the “binPath=”, and that you use quotation marks around your file path
Then just run sc start <SERVICE NAME>, which in our case is sc start OpenSM
To ensure the service is indeed working as it should
open Task Manager, go to Services
Find OpenSM and check that it’s status is Running
Now that we have a subnet manager, we need to assign each NIC an IP
We want our Infiniband cards to operate on a separate subnet from the rest of the devices on our LAN
Open cmd, run ipconfig
Look for your regular Ethernet adapter which in our case is Ethernet adapter Ethernet 3
Under that look for IPv4 Address which is listed as 192.168.0.20 for our host PC
Which means our regular network is running on 192.168.0.xxx
With that knowledge, I decided to assign our Infiniband NICs IP under 192.168.1.xxx which should prevent any conflicts
On each PC, search View Network Connections in the start menu
Right click on your Infiniband adapter which for us is Mellanox ConnectX-2 IPoIB Adapter
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties
Click Use the following IP address
For our “Gaming-PC”, we are going to assign 192.168.1.21 as the IP
The subnet mask should autofill as 255.255.255.0
Last, set the Default gateway to the IP of the other PC, which we are going to set as 192.168.1.20
For our “NAS-PC”, the ip is set to 192.168.1.20
With a subnet mask of 255.255.255.0
And a default gateway set to the IP of our host PC which in this case is 192.168.1.21
Should receive a “new network popup” asking if you’d like to enable network discovery, say yes on both machines.
To test that you do indeed have apc nam connection
Open cmd, and ping the opposite PC, which from our NAS-PC (192.168.1.20) would be ping 192.168.1.21
You’re probably thinking that you’ll need to unplug your ethernet cable every time you want to transfer files to your other PC over Infiniband right? WRONG.
To ensure your PC’s utilize Infiniband when doing transfers, we have to set the priority of the Network adapters, otherwise known as Metric
On each PC, search View Network Connections in the start menu
Right click on your Infiniband adapter, which for us is Mellanox ConnectX-2 IPoIB Adapter
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced
Deselect Automatic metric
Enter a higher number than your regular ethernet adapter, we are going to use 10
Right click on your regular ethernet adapter, which for us is Ethernet 3
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced
Deselect Automatic metric
Enter a higher number than your Infiniband adapter, we are going to use 20
You now have a 40Gbit link between your two PCs.
Click on your Infiniband Adapter and then click View status of this connection
Should show a speed of 32.0 Gbps
To test the full 40Gbit (which after 8/10 encoding shows as 32Gbit in our connection status), we needed an extremely fast storage medium
So we reached out to the folks over SoftPerfect who hooked us up with a copy of their RAM Disk software for our testing, if you a ton of extra RAM and want to use it for insanely fast storage, definitely check them out!
Using their software we were able to obtain around 7 gigabytes per second sequential read & write on each PC
From our NAS-PC, we shared the formated RAM disk over the network, and it showed up immediately on the other PC in the Network tab (if your PC has a password, you will need to enter it at this step, unless you are in a homegroup)
If you were planning on using this as a long term solution, I’d recommend mapping a network drive to this shared folder/disk