Jump to content

40Gbit Tutorial

jakkuh_t

I also used ConnectX-2 Mellanox cards and called them why I never get more then 24 Gbit speed while copying files:

Here is their answer:

PCI_LANES(8)*PCI_SPEED(5)*PCI_ENCODING(0.8)*PCI_HEADERS(128/152)*PCI_FLOW_CONT(0.95) = 25.6 Gbit that is the usable speed

 

So ConnectX-2 goes to 25 GBit. If someone here really need 40+ Gbits you can go for ConnectX-3 which uses PCIe 3.0 and can do usable 56 Gbit

Some even use Infiniband AND Ethernet so you can use a QSFP to RJ45 transreciever to use them in your current 10Gbit or even 40 GBit Ethernet network.

http://www.ebay.com/itm/MELLANOX-MCX353A-FCBT-FDR-FDR14-INFINIBAND-40GbE-ETHERNET-VPI-CONNECTX-3-HCA-/252761374676?hash=item3ad9c07fd4:g:0uoAAOSwB09YDL7K

(THIS THING DOES 56Gbit IB and 40Gb Ethernet)

 

Mellanox Product Page:

http://www.mellanox.com/page/products_dyn?product_family=119&mtag=connectx_3_vpi

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 17/02/2017 at 2:38 AM, themctipers said:

if you make this floatplane only again

What? We got this video like 7 days earlier than you guys...

 

On 17/02/2017 at 0:56 AM, wii8cookies said:

huh... I'm not a floatplane pilot and somehow I just found this on the regular forum... it was right there, the red circle. weird glitch? I never was a float plane pilot... @jakkuh_t

 

 

On 17/02/2017 at 2:09 AM, iPolymer said:

I'm here too I don't know why or how the linked it on the YouTube

Have you heard of this thing called moving threads? It used to be in FPC and not anymore as the plebs on youtube now needs to see it as well :P 

Looking at my signature are we now? Well too bad there's nothing here...

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

What? As I said, there seriously is nothing here :) 

Link to comment
Share on other sites

Link to post
Share on other sites

Anyone have any luck getting these 'MHQH19B-XTR' cards to work in unraid? I've got one showing up as a pci device but not an ethernet device. It looks like you can switch modes between infiniband and ethernet but the software side of things is beyond me when it comes to linux.

Link to comment
Share on other sites

Link to post
Share on other sites

On 2/18/2017 at 11:24 AM, flungo said:

What is going on is that IB uses 10 bits for each 8 bits of data

Having a pair of ConnectX-2 EN (as opposed to VPI) would also yield 32Gbps then (due to PCIe this time, not due to ethernet)?

Link to comment
Share on other sites

Link to post
Share on other sites

Update 2: Just did a disk test on my RMA instead of just looking a the files speeds and it's saying 8gb/s 


Update:
After removing the gateway as per @flungo post and rebooting I'm getting 700mb/s across but that's still 10x less than what @jakkuh_t got in his original test.

What's also Odd. I did a Ram to Ram test on the same system and got around 2gb/s which would translate into a theoretical 4gb/s as it's using the ram disk for both read and right. But if I'm pushing 4gb/s on the ram how could we ever get 7 down the pipe? 



So I've got everything set up but I'm only getting 224mb/s 

Ram Disk is 13.1gb 
Transfer 12.6gb video file from Comp1 to comp2 with all other networks disabled (for simplicity sake) 
Results in ~224 MB/s Any idea what the bottleneck would be?

Comp 1 is running windows 7 

Comp 2 is running windows 10

Both have i7s

If God exists, you'll want to know Him

If God is Christ, He died so that you could. (Romans 5:6-11)

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

I am using ConnectX-2 cards on 3 Windows 2012 R2 servers, and 1 Windows 10 workstation. I am using a Voltaire 4036 switch.

 

At the moment I am just utilizing Infiniband for file transfers. Here are some of the nuggets from my experience.

 

If you are using the MHQH19B-XTR or any ConnectX-2 gen card, firmware is very important. The "latest" firmware offered by Mellanox is something like 2.9.1000, however RDMA is NOT functional on this firmware version. There is newer firmware that is no longer available that you need to flash in order to support RDMA. I am using 2.10.720 on my MHQH19B-XTR adapters and RDMA is functioning just fine. I found this firmware EXTREMELY hard to find so for your convenience it is attached, and the flashing instructions in the OP still apply for this firmware.

 

In the OP, an older version of WinOF was recommended. I am not sure why, but WinOF 5.35 for ConnectX-2 works just fine for me, and actually displays the connection as 40Gbps instead of 32Gbps even though there is no difference.

 

RDMA is not available on Windows 10 professional so you will not be able to utilize the maximum line speed unless you are using a Windows Server operating system (version 2012 or newer).

 

Using a Voltaire 4036 switch? You will need to be on firmware version 3.9.1 for RDMA to function properly. Google yourself a guide to do so, but in my quest I will say that most guides are written around using the serial management port and PuTTy, but the path of least resistance (and eBaying serial adapters) is SSH-ing into the eth port using PuTTy. My eth port wanted to request a new IP address from DHCP every 10 minutes or so, so I would also recommend giving it a DHCP reservation so you can keep a stable PuTTy connection.

 

EDIT: I would also add that cable length is important in achieving stable and maximum speeds. If using copper cable, I wouldn't go over 3 meters if you intend on running your link at QDR. If using fiber length isn't an issue, but prices go way up. If you are having cable issues you aren't going to get a notification that the cable is bad, if you manage to get a 40G link your transfers will stall out and you'll see protocol errors in the SMBClient event logs.

 

MHQH19B-XTR Firmware 2.10.720.zip

Link to comment
Share on other sites

Link to post
Share on other sites

Hi,

Thanks a lot for this guide, i bought two MNPA19-XTR whit 10GBit/s, followed all the steps and now i have a stable connection to my backup server(Windows 10 Pro).

At the moment i use a 3m DAC cable, but i found a 30m glasfiber cable on ebay, so my backup server will move to the basement.

My standart speed test with the Starwars collection reached 739MB/s betwen a Samsung 950 m.2 and a Intel 600p.

 

Thanks for speeding up my network

Link to comment
Share on other sites

Link to post
Share on other sites

  • 3 months later...

found this from youtube Our CHEAPEST & FASTEST Network Speed Yet!

Got 2x MHQH19b-xtr and Mellanox MC2210128-003 lead. http://www.mellanox.com/related-docs/prod_cables/DS_40GbE_Passive Copper Cables.pdf

Hooked everything up all good cards showing green light when lead pluged in.

But showing Unplugged in connections on both systems.

 

Windows 10

Driver Version : 5.35.12978.0
Firmware Version : 2.7.626
Port Number : 1
Bus Type : PCI-E 5.0 Gbps x8
Link Speed : ----
Part Number : MHQH19B-XTR
Device Id : 26428
Revision Id : B0
Current MAC Address : 02-02-C9-4A-FC-5F
Permanent MAC Address : 02-02-C9-4A-FC-5F
Network Status : Disconnected
Adapter Friendly Name : Ethernet
IPv4 Address : 169.254.246.1
Adapter User Name : 0xffff-IPoIB
Adapter PKey : 0xffff (Available)

 

Windows 2016 server

 

Driver Version : 5.35.12978.0
Firmware Version : 2.7.626
Port Number : 1
Bus Type : PCI-E 5.0 Gbps x4
Link Speed : ----
Part Number : MHQH19B-XTR
Device Id : 26428
Revision Id : B0
Current MAC Address : 02-02-C9-4B-09-FB
Permanent MAC Address : 02-02-C9-4B-09-FB
Network Status : Disconnected
Adapter Friendly Name : Ethernet 2
IPv4 Address : 192.168.1.1
Adapter User Name : 0xffff-IPoIB
Adapter PKey : 0xffff (Available)

 

Tryed updating firmware But having no luck

 

C:\WINDOWS\system32>mst status -v
MST devices:
------------
  mt26428_pci_cr0        bus:dev.fn=02:00.0
  mt26428_pciconf0       bus:dev.fn=02:00.0

C:\WINDOWS\system32>Format: mt26428_pci_cr0
'Format:' is not recognized as an internal or external command,
operable program or batch file.

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

You're trying to run IP over Infiniband.

 

Are you running in datagram or connected mode.

 

Have you tried to boot into Linux and look/poke around, see what kernel logs when you plug in the cable, and so on.

Link to comment
Share on other sites

Link to post
Share on other sites

25 minutes ago, risk said:

You're trying to run IP over Infiniband.

 

Are you running in datagram or connected mode.

 

Have you tried to boot into Linux and look/poke around, see what kernel logs when you plug in the cable, and so on.

Thanks for reply .

But resolved it.

Managed to update fireware on both cards to 2.9.1000 just skipped the Format part and all is good and connected .

 

Driver Version : 5.35.12978.0
Firmware Version : 2.9.1000
Port Number : 1
Bus Type : PCI-E 5.0 Gbps x4
Link Speed : 40.0 Gbps/Full Duplex
Part Number : MHQH19B-XTR
Device Id : 26428
Revision Id : B0
Current MAC Address : 02-02-C9-4B-09-FB
Permanent MAC Address : 02-02-C9-4B-09-FB
Network Status : Connected
Adapter Friendly Name : Ethernet 2
IPv4 Address : 192.168.1.20
Adapter User Name : 0xffff-IPoIB
Adapter PKey : 0xffff (Available)

 

Driver Version : 5.35.12978.0
Firmware Version : 2.9.1000
Port Number : 1
Bus Type : PCI-E 5.0 Gbps x8
Link Speed : 40.0 Gbps/Full Duplex
Part Number : MHQH19B-XTR
Device Id : 26428
Revision Id : B0
Current MAC Address : 02-02-C9-4A-FC-5F
Permanent MAC Address : 02-02-C9-4A-FC-5F
Network Status : Connected
Adapter Friendly Name : Ethernet
IPv4 Address : 192.168.1.21
Adapter User Name : 0xffff-IPoIB
Adapter PKey : 0xffff (Available)

 

Only getting around 3Gbps From nvme to nvme But its faster .

and only cost me £60 for cards and leads so happy.

Thanks for the post.

 

Link to comment
Share on other sites

Link to post
Share on other sites

  • 5 months later...

The new mellanox Connectx-6 is 200gbit/sec ... and we should try this cards on a Threadripper. May be there is a limitation of intel or the pci-lines???

Link to comment
Share on other sites

Link to post
Share on other sites

  • 7 months later...
  • 10 months later...

Not to beat a dead thread, but I had to revisit this subject yesterday when I added a MHQH29C-XTR to one of my servers. It's one of the dual port QDR adapters.

 

Below are the resources needed to flash the MHQH29C-XTR and the popular MHQH19B-XTR adapters to firmware 2.10.720 (RDMA supported)

 

Things have gotten a bit more tricky than I last remember.

 

You need to install WinMFT 4.3.0.25 for burning firmware to the ConnectX-2 cards. Later versions dropped support, namely the mic.exe binary and it's associated libraries.

 

  • Install the above WinMFT version
  • Open elevated command prompt, run command mst status
  • Copy the device ID, for example mt26428_pci_cr0
  • Run the command flint -d mt26428_pci_cr0 query (substitute your device id if different)
  • Note your Board ID or PSID. My cards only showed a PSID. If your ID is different than MT_0D90110009 or MT_0FC0110009 than I would refer to this guide for information about attempting to modify the firmware build for your specific card.
  • Extra the firmware for the card from the appropriate attached ZIP to a easy to locate folder. Navigate to that folder in your command window.
  • Run the following command, substituting your device ID if neccessary mlxburn.exe -dev mt26428_pci_cr0 -fw fw-ConnectX2-rel.mlx
  • Once the process completes you can reboot the system and check the driver properties or run ibstat from a command window to check that the new firmware is running.

Hope this saves someone some time, as I had to blindly try software versions and make modifications to the firmware until I got my adapter to take v2.10.720.

MHQH19B-XTR Firmware 2.10.720.zip

MHQH29C-XTR Firmware 2.10.720.zip

Link to comment
Share on other sites

Link to post
Share on other sites

Also worth noting that RDMA now works great on these cards when using Windows 10 for Workstations.

Link to comment
Share on other sites

Link to post
Share on other sites

  • 1 year later...

Sorry for raising this zombie thread. 

 

In the process of setting up a direct link and may look in a breakout cable down the road to try and connect into my ethernet switch.

 

But for the folks who have used these cards, do I need to worry about adding a fan like you do on some HBA's or RAID cards or are these things okay being cooled by a typical system case fan setup?

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×