Jump to content

40Gbit Tutorial

jakkuh_t

 

  1. Install the two cards in your system, ours sadly came with no PCI slot covers, however we don’t recommend using the cards without a bracket, so you might have to get creative.

    1. For the purposes of this video, our setup will be a “Gaming-PC” and a “NAS-PC”

  2. Boot up systems and head to windows device manager, and under “Network adapters” you should see “Mellanox ConnectX-2 IPoIB Adapter”

    1. If you don’t see them now, you may after installing the drivers, if you still don’t, you may have a faulty card

  3. Head to the Mellanox OFED for Windows page (http://www.mellanox.com/page/products_dyn?product_family=32&menu_section=34)

  1. For the ConnectX-2 series the drivers are hidden under “Archive Versions”, so we didn’t immediately find anything, and the ConnectX-2 cards aren’t listed in the recommended versions, scaring me a bit.

  2. Were going to install version 4.95 and then update to 5.10 which seemed to be the most reliable install

    1. WinOF Download>Archive Versions>5.10>Windows Client>10

    2. WinOF Download>Archive Versions>4.95>Windows Client>8.1

  1. Install the 4.95 drivers and once done, immediately install with the 5.10 drivers

  2. Now, technically our cards should have had their firmware updated with the driver install, but this didn’t seem to happen

    1. We will need Mellanox Firmware Tools and the firmware file for our specific cards to update them

      1. Download and Install MFT for Windows

      2. Download the firmware file

        1. In our case was for the MHQH19B-XTR

      3. Open an elevated command prompt

      4. Get your device name by running mst status

        1. Format: mt<id>_pci_cr0

      5. CD to the directory containing the firmware file

        1. In our case was cd C:\User\6950X\Desktop\firmware

      6. Run flint -d <Device Name> -i <Firmware File>.bin burn

        1. In our case was flint -d mt26428_pci_cr0 -i fw-ConnectX2-rel-2_9_1000-MHQH19B-XTR_A1-A3.bin burn

      7. Re run the version 5.10 driver installer choosing the “repair” option

    2. Follow these steps on both machines

  3. We now have functioning Infiniband NICs, but no subnet manager (which assigns an ID to the NICs, and creates a routing table based off of that)

    1. I’m going to setup the subnet manager on our “NAS-PC” to keep any extra load off the “Gaming-PC”

      1. Now, if you want, you can open up command prompt and run the OpenSM executable every time you turn on your PC, but that's way too much work.

        1. Elevated Cmd>C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe

      2. So, with the magic of Windows services, we can make it start on login in the background!

        1. Start by opening cmd as administrator (Right Click Windows Icon in Bottom Left>Command Prompt [Admin])

        2. Enter sc create <SERVICE NAME> binPath= “<FILE PATH TO OPENSM> --service” start=auto

          1. Replacing <SERVICE NAME> with your desired name, we chose OpenSM

          2. And inputting the file path which in our case is

            1. C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe

          3. Which equated to

            1. sc create OpenSM binPath= “C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe --service” start=auto

          4. MAKE SURE, you have a space after the “binPath=”, and that you use quotation marks around your file path

        3. Then just run sc start <SERVICE NAME>, which in our case is sc start OpenSM

        4. To ensure the service is indeed working as it should

          1. open Task Manager, go to Services

          2. Find OpenSM and check that it’s status is Running

  4. Now that we have a subnet manager, we need to assign each NIC an IP

    1. We want our Infiniband cards to operate on a separate subnet from the rest of the devices on our LAN

      1. Open cmd, run ipconfig

      2. Look for your regular Ethernet adapter which in our case is Ethernet adapter Ethernet 3

      3. Under that look for IPv4 Address which is listed as 192.168.0.20 for our host PC

        1. Which means our regular network is running on 192.168.0.xxx

      4. With that knowledge, I decided to assign our Infiniband NICs IP under 192.168.1.xxx which should prevent any conflicts

    2. On each PC, search View Network Connections in the start menu

      1. Right click on your Infiniband adapter which for us is Mellanox ConnectX-2 IPoIB Adapter

      2. Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties

      3. Click Use the following IP address

      4. For our “Gaming-PC”, we are going to assign 192.168.1.21 as the IP

        1. The subnet mask should autofill as 255.255.255.0

        2. Last, set the Default gateway to the IP of the other PC, which we are going to set as 192.168.1.20

      5. For our “NAS-PC”, the ip is set to 192.168.1.20

        1. With a subnet mask of 255.255.255.0

        2. And a default gateway set to the IP of our host PC which in this case is 192.168.1.21

  5. Should receive a “new network popup” asking if you’d like to enable network discovery, say yes on both machines.

  6. To test that you do indeed have apc nam connection

    1. Open cmd, and ping the opposite PC, which from our NAS-PC (192.168.1.20) would be ping 192.168.1.21

  7. You’re probably thinking that you’ll need to unplug your ethernet cable every time you want to transfer files to your other PC over Infiniband right? WRONG.

    1. To ensure your PC’s utilize Infiniband when doing transfers, we have to set the priority of the Network adapters, otherwise known as Metric

      1. On each PC, search View Network Connections in the start menu

        1. Right click on your Infiniband adapter, which for us is Mellanox ConnectX-2 IPoIB Adapter

          1. Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced

          2. Deselect Automatic metric

          3. Enter a higher number than your regular ethernet adapter, we are going to use 10

        2. Right click on your regular ethernet adapter, which for us is Ethernet 3

          1. Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced

          2. Deselect Automatic metric

          3. Enter a higher number than your Infiniband adapter, we are going to use 20

  8. You now have a 40Gbit link between your two PCs. :D

    1. Click on your Infiniband Adapter and then click View status of this connection

    2. Should show a speed of 32.0 Gbps

  9. To test the full 40Gbit (which after 8/10 encoding shows as 32Gbit in our connection status), we needed an extremely fast storage medium

    1. So we reached out to the folks over SoftPerfect who hooked us up with a copy of their RAM Disk software for our testing, if you a ton of extra RAM and want to use it for insanely fast storage, definitely check them out!

      1. Using their software we were able to obtain around 7 gigabytes per second sequential read & write on each PC

      2. From our NAS-PC, we shared the formated RAM disk over the network, and it showed up immediately on the other PC in the Network tab (if your PC has a password, you will need to enter it at this step, unless you are in a homegroup)

      3. If you were planning on using this as a long term solution, I’d recommend mapping a network drive to this shared folder/disk

widget.png?style=banner2

PC: 13900K, 32GB Trident Z5, AORUS 7900 XTX, 2TB SN850X, 1TB MP600, Win 11

NAS: Xeon W-2195, 64GB ECC, 180TB Storage, 1660 Ti, TrueNAS Scale

Link to comment
Share on other sites

Link to post
Share on other sites

Please adjust the format for Dark Theme users ... :| 

Thanks :) 

 

*edit

its been a few days now ... still difficult to read ... hint, hint ... wink, wink ...

Edited by SansVarnic

COMMUNITY STANDARDS   |   TECH NEWS POSTING GUIDELINES   |   FORUM STAFF

LTT Folding Users Tips, Tricks and FAQ   |   F@H & BOINC Badge Request   |   F@H Contribution    My Rig   |   Project Steamroller

I am a Moderator, but I am fallible. Discuss or debate with me as you will but please do not argue with me as that will get us nowhere.

 

Spoiler

  

 

Character is like a Tree and Reputation like its Shadow. The Shadow is what we think of it; The Tree is the Real thing.  ~ Abraham Lincoln

Reputation is a Lifetime to create but seconds to destroy.

You have enemies? Good. That means you've stood up for something, sometime in your life.  ~ Winston Churchill

Docendo discimus - "to teach is to learn"

 

 CHRISTIAN MEMBER 

 

 
 
 
 
 
 

 

Link to comment
Share on other sites

Link to post
Share on other sites

Nice post! Though, I would agree with some of the comments stating the problems with the Dark Theme. It's not the first time I've encountered it, specially on Floatplane (which is completely understandable, mind you...), but this is the first time where it actually makes it harder to read due to the font being too dark for the background. If you guys could look it up and add to the fix-list, would be awesome! Thanks a lot! And keep up with the great work, you guys!  @jakkuh_t @Slick

Link to comment
Share on other sites

Link to post
Share on other sites

When did you rack up 479 posts....

I am good at computer

Spoiler

Motherboard: Gigabyte G1 sniper 3 | CPU: Intel 3770k @5.1Ghz | RAM: 32Gb G.Skill Ripjaws X @1600Mhz | Graphics card: EVGA 980 Ti SC | HDD: Seagate barracuda 3298534883327.74B + Samsung OEM 5400rpm drive + Seatgate barracude 2TB | PSU: Cougar CMX 1200w | CPU cooler: Custom loop

Link to comment
Share on other sites

Link to post
Share on other sites

On 2/10/2017 at 8:32 PM, SansVarnic said:

Please adjust the format for Dark Theme users ... :| 

Thanks :) 

 

*edit

its been a few days now ... still difficult to read ... hint, hint ... wink, wink ...

fine, gosh.

widget.png?style=banner2

PC: 13900K, 32GB Trident Z5, AORUS 7900 XTX, 2TB SN850X, 1TB MP600, Win 11

NAS: Xeon W-2195, 64GB ECC, 180TB Storage, 1660 Ti, TrueNAS Scale

Link to comment
Share on other sites

Link to post
Share on other sites

2 minutes ago, CatBoiler said:

K, thx.

"trust me, im an engineer"

 

come play ark tonight, kthx.

widget.png?style=banner2

PC: 13900K, 32GB Trident Z5, AORUS 7900 XTX, 2TB SN850X, 1TB MP600, Win 11

NAS: Xeon W-2195, 64GB ECC, 180TB Storage, 1660 Ti, TrueNAS Scale

Link to comment
Share on other sites

Link to post
Share on other sites

When you are configuring the IP settings for the two InfiniBand NICs, you set the default gateway even though you were already connected to your primary network with another NIC.

 

Windows gave a warning, and its an important warning! This could cause weird and unusual network connectivity for that computer, as it will have two "defaults" to choose from.

 

The way to fix this is just leave default gateway blank on the InfiniBand NICs. The two computers should still communicate fine without a gateway, just as long as they're on the same subnet.

 

This is at least the case with Ethernet based networking. I doubt its much different for InfiniBand (its still TCP/IP).

Link to comment
Share on other sites

Link to post
Share on other sites

huh... I'm not a floatplane pilot and somehow I just found this on the regular forum... it was right there, the red circle. weird glitch? I never was a float plane pilot... @jakkuh_t

ltt glitch.PNG

linus sex tips

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, wii8cookies said:

-snip-ltt glitch.PNG

I'm here too I don't know why or how the linked it on the YouTube

Link to comment
Share on other sites

Link to post
Share on other sites

How do I get this to work between windows 10 and freenas (for iscsi, but I already got that part figured out)

Link to comment
Share on other sites

Link to post
Share on other sites

Also came here from the link on main page as above.

 

Was going to jokingly add:

Step #1: Use QSFP+

Step #2: Cry about the cost

Link to comment
Share on other sites

Link to post
Share on other sites

If anyone is planning to try this with the intent of getting storage performance, you should spend a bit more and get connectx-3 cards.  Connectx-3 and newer support Network Direct which enables SMBDirect.   You should come close to saturating the 40 gig link when using SMBDirect with lower cpu utilization than when using IPoIB.   

 

Using connectx 2 as nothing more than an IPoIB adapter kind of defeats the purpose of using infiniband in the first place.  

Link to comment
Share on other sites

Link to post
Share on other sites

Nice guide

 

I only want to mention, it isn't best practice to add a gateway when there is none. A gateway is a way to communicate with another network. In this specific case, as there is no network, there is no need to put one in the config. 

 

Also, was there any option for jumbo frame / MTU ajustment with that kind of adapter to see greater speed?

 

Thanks!

Link to comment
Share on other sites

Link to post
Share on other sites

if you make this floatplane only again

je suis triggered

shiver me triggers

Ryzen 5 3600 stock | 2x16GB C13 3200MHz (AFR) | GTX 760 (Sold the VII)| ASUS Prime X570-P | 6TB WD Gold (128MB Cache, 2017)

Samsung 850 EVO 240 GB 

138 is a good number.

 

Link to comment
Share on other sites

Link to post
Share on other sites

Does anyone know if this method will work for freenas? If this gear will play nice with my nas, I know where my tax return money is going. 

Link to comment
Share on other sites

Link to post
Share on other sites

Can you provide the links to all the things mentioned as well. I cannot find the cheap Mellanox cards ...

Link to comment
Share on other sites

Link to post
Share on other sites

DaveBG, look for any of the product codes in these 2 PDFs (Linus/Jake used MHQH19B-XTR from the first file), you might stumble onto something cheap.

 

Here's are two PDFs from Mellanox with more info on various cards:

https://www.mellanox.com/related-docs/user_manuals/ConnectX 2_VPI_UserManual.pdf

https://www.mellanox.com/related-docs/user_manuals/ConnectX-2_EN_user_manual.pdf

 

I think Mellanox drivers (correctly) report 32Gbps (and not 40Gbps) because they're chocked by only having 8 lanes of PCIe 2.0 (totalling 32Gbps) -- you can still interface them with other 40Gbps equipment (as along as it's the same 40Gbps standard). The cheap ConnectX-2 come from old servers that are comparatively expensive to run these days and are therefore getting decommissioned.

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

Hi!

Can i make a super fast Pf Sense router with these NIC-s ?

I'm really interested about it cause im planning to upgrade my home network.:)

 

Link to comment
Share on other sites

Link to post
Share on other sites

For anyone looking for parts. @DaveBG These are the ones I found
(I make no claim to the correctness or working order of these parts. Please do your own research to verify) 

Cable: http://ebay.to/2lfpzxl

Network Cards: http://ebay.to/2lHoaBn



Link to download the firmware for the card mentioned in the OP :

http://www.mellanox.com/page/firmware_table_ConnectX2IB

If God exists, you'll want to know Him

If God is Christ, He died so that you could. (Romans 5:6-11)

Link to comment
Share on other sites

Link to post
Share on other sites

As I wrote in a YouTube comment and as others have mentioned you should not be setting the default gateway on the IB network.

 

On 16/02/2017 at 11:54 PM, adam8797 said:

When you are configuring the IP settings for the two InfiniBand NICs, you set the default gateway even though you were already connected to your primary network with another NIC.

 

Windows gave a warning, and its an important warning! This could cause weird and unusual network connectivity for that computer, as it will have two "defaults" to choose from.

 

The way to fix this is just leave default gateway blank on the InfiniBand NICs. The two computers should still communicate fine without a gateway, just as long as they're on the same subnet.

 

This is at least the case with Ethernet based networking. I doubt its much different for InfiniBand (its still TCP/IP).

5

 

On 17/02/2017 at 2:37 AM, sangokudbz79 said:

Nice guide

 

I only want to mention, it isn't best practice to add a gateway when there is none. A gateway is a way to communicate with another network. In this specific case, as there is no network, there is no need to put one in the config. 

 

Also, was there any option for jumbo frame / MTU ajustment with that kind of adapter to see greater speed?

 

Thanks!

2

It's not only not "best-practice" but will cause networking problems. I am not actually sure how you can still access the rest of the network with both PCs having the lowest metric route being via each other, but I am sure its some RFC breaking Windows magic if it does work.

 

If you communicate between the two machines using their IPoIB addresses then you just don't set the default gateway or touch any metrics and it will work.

 

If you want to be able use their LAN IP addresses and have it routed over IB (or just not worry about what IP is discovered using Windows's automatic discovery), then that's also possible. Again, don't set the default gateway or touch any metrics. Give the two machines static IPs (either manually or via DHCP). Add persistent routes between the machines for their respective LAN IP addresses.

 

Based on your example configuration and a Google search, using an Administrator Command Prompt, if we assume that the LAN IP for the NAS and Gaming PC are 192.168.0.20 and 192.168.0.21 respectively:

  • On the NAS-PC (192.168.0.20/192.168.1.20) run "route -p add 192.168.0.21 255.255.255.255 192.168.1.21"
  • On the Gaming-PC (192.168.0.21/192.168.1.21) run "route -p add 192.168.0.20 255.255.255.255 192.168.1.20"

This may be possible through the Windows GUI but again, I do not use Windows so cannot confirm this.

 

In my opinion, the configuration you have given is fundamentally flawed, does not achieve what it sets out to do and is misinforming people who do not understand how networking, routing and IP work. I hope you can take this feedback, test it and make the appropriate corrections.

Link to comment
Share on other sites

Link to post
Share on other sites

On 17/02/2017 at 9:33 AM, risk said:

I think Mellanox drivers (correctly) report 32Gbps (and not 40Gbps) because they're chocked by only having 8 lanes of PCIe 2.0 (totalling 32Gbps) -- you can still interface them with other 40Gbps equipment (as along as it's the same 40Gbps standard). The cheap ConnectX-2 come from old servers that are comparatively expensive to run these days and are therefore getting decommissioned.

 

 

 

The 8/10 encoding is actually the correct reason for it showing as 32Gbps. The 8 lanes of PCIe 2.0 were chosen based on the fact that 32Gbps is the actual data rate rather than 40Gbps which is the transfer rate. Infiniband is advertised using its "signalling rate" rather than its throughput. What is going on is that IB uses 10 bits for each 8 bits of data, for the reasons and details about this, see 8b/10b encoding on Wikipedia. PCIe and ethernet also use similar encodings but the speeds you typically see advertised are the throughput and not the signalling rate.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×