DaveBG

DaveBG, look for any of the product codes in these 2 PDFs (Linus/Jake used MHQH19B-XTR from the first file), you might stumble onto something cheap.
 
Here's are two PDFs from Mellanox with more info on various cards:
https://www.mellanox.com/related-docs/user_manuals/ConnectX 2_VPI_UserManual.pdf
https://www.mellanox.com/related-docs/user_manuals/ConnectX-2_EN_user_manual.pdf
 
I think Mellanox drivers (correctly) report 32Gbps (and not 40Gbps) because they're chocked by only having 8 lanes of PCIe 2.0 (totalling 32Gbps) -- you can still interface them with other 40Gbps equipment (as along as it's the same 40Gbps standard). The cheap ConnectX-2 come from old servers that are comparatively expensive to run these days and are therefore getting decommissioned.
 
 

For anyone looking for parts. @DaveBG These are the ones I found
(I make no claim to the correctness or working order of these parts. Please do your own research to verify) 
Cable: http://ebay.to/2lfpzxl
Network Cards: http://ebay.to/2lHoaBn


Link to download the firmware for the card mentioned in the OP :
http://www.mellanox.com/page/firmware_table_ConnectX2IB

Buy network cards on Amazon: http://geni.us/KG3N
 
Get a lightning fast local network speed for less than $100!
 
 

As I regulary watch LinusTechtips on youtube I stumbled across a video called: 10x Your Network Speed.. On a Budget!
 
I thought that is still too slow. So I googled about alternate technology and found that infiniband can also do normal IP networking.
So I bought two Infiniband adapters with 40Gbit network speed each and a 7 Meter QSFP cable. All in all less than 100 Dollars.
something like this:
http://www.ebay.at/itm/Mellanox-ConnectX-2-VPI-Network-Adaptor-PCIe-Server-Card-/331765696568
 
I really find the technology fascinating as normal 40GbE Adapters are VERY expensive and I never heard of infiniband before. I bought two cards and they show up with 32Gbit. File sharing speed is at about 5 Gbytes/s (not Gigabit)! I cant find anything wrong with this solution. It works, is cheap and behaves like any other NIC (because of IP over Infiniband)
So normal SMB and other network stuff works as usual.
 
So my question is if any of you has also tried this and if someone on this channel could take a look it this to point out the possible drawbacks. (Maybe Youtube? :D)

Considering this for a video, but need the drivers as our units didn't come with any for Windows. Anyone got them? PM ME!  
 
EDIT: Model number is QLE7340

@jakkuh_t You should add the model numbers of the cards that worked

very cool

How do I get this to work between windows 10 and freenas (for iscsi, but I already got that part figured out)

If anyone is planning to try this with the intent of getting storage performance, you should spend a bit more and get connectx-3 cards.  Connectx-3 and newer support Network Direct which enables SMBDirect.   You should come close to saturating the 40 gig link when using SMBDirect with lower cpu utilization than when using IPoIB.   
 
Using connectx 2 as nothing more than an IPoIB adapter kind of defeats the purpose of using infiniband in the first place.  

Nice guide
 
I only want to mention, it isn't best practice to add a gateway when there is none. A gateway is a way to communicate with another network. In this specific case, as there is no network, there is no need to put one in the config. 
 
Also, was there any option for jumbo frame / MTU ajustment with that kind of adapter to see greater speed?
 
Thanks!

The newest 960 Pro SSDs read at about 3.2 Gigabytes a second. This way I can copy data as fast as it would be locally. Another point would be RAMDISK sharing over network.

Wow guys think outside of the box for a moment. TODAY: if you have a single SSD network transfer is limited by a 10GbE card. Same thing goes for a raid with more than 1,2Gigbytes/s read (about 8 drives raid 0). This is a solution that costs as much as 10GbE but is significantly faster. Please dont spread your 640k mentality.

I thought this forum was for enthusiasts and not for those who ask who needs this. But those who ask whats the limit and how do i break it!
Defending older standards wont bring anyone further. We have been stuck with 1GbE for more than a decade and 10GbE is still very expensive.

Just because YOU dont have a use for your home does not mean nobody else has it. Take the newest movies in 4k for example. The filesize is about 30Gbyte even with HEVC. Quantum Break or Gears of Wars need 80 Gbyte+. Now transfering that much data from A to B is way to slow with 1Gbit. 10 Gbit is too expensive with RJ45 connectors and moving an external drive from A to B is impractical.
 
Seems the cheap way to go here is Thunderbolt or Infiniband. Both support 40Gbit file transfer from A to B for under 100 Dollar.
Infiniband switches are cheaper than 10GBase-T too.

While they didn't see performance above their 10Gbps network video, they did achieve the same performance for less money.
 
Plus, they didn't see the performance because consumer copies of Windows don't support file transers above 10Gbps, apparently. If one uses Linux or Windows Enterprise, they could potentially see the faster speeds, provided they're transferring large or a multitude of files.

You said you might be able to user Windows Server or Linux to make this EVEN FASTER.
 
Yes, please.
 
Additionally, I would argue that you should do this on Linux since most consumers and enthusiasts could get their hands on a Linux distro pretty easily for free, whereas Windows Server costs some extra cash (or a lot of extra cash), and part of the point was to save money.

If you want a really cheap ethernet solution, don't forget you can turn these cards into Ethernet devices so you don't need the Infiniband fabric manager.
 
For Windows, download and install the latest Mellanox WinOF VPI for Windows software package available via the Mellanox Web site at: http://www.mellanox.com => Follow the installation instructions included in the download package. After installing Mellanox WinOF VPI for Windows on your machine, you can change a port's protocol configuration.
 
That means you just bought a 10Gb Ethernet card for $25, which is a much more useful thing!
 
Plus, because it's a Mellanox card with a Mellanox driver stack, they're typically significantly better at being low latency and doing a lot of hardware offloading. Great ethernet solution, we use them all the time.
 
And you don't need to specify a gateway.

Really liked this video, keep it up 

Install the two cards in your system, ours sadly came with no PCI slot covers, however we don’t recommend using the cards without a bracket, so you might have to get creative.
For the purposes of this video, our setup will be a “Gaming-PC” and a “NAS-PC”
Boot up systems and head to windows device manager, and under “Network adapters” you should see “Mellanox ConnectX-2 IPoIB Adapter”
If you don’t see them now, you may after installing the drivers, if you still don’t, you may have a faulty card
Head to the Mellanox OFED for Windows page (http://www.mellanox.com/page/products_dyn?product_family=32&menu_section=34)
For the ConnectX-2 series the drivers are hidden under “Archive Versions”, so we didn’t immediately find anything, and the ConnectX-2 cards aren’t listed in the recommended versions, scaring me a bit.
Were going to install version 4.95 and then update to 5.10 which seemed to be the most reliable install
WinOF Download>Archive Versions>5.10>Windows Client>10
WinOF Download>Archive Versions>4.95>Windows Client>8.1
Install the 4.95 drivers and once done, immediately install with the 5.10 drivers
Now, technically our cards should have had their firmware updated with the driver install, but this didn’t seem to happen
We will need Mellanox Firmware Tools and the firmware file for our specific cards to update them
Download and Install MFT for Windows
Download the firmware file
In our case was for the MHQH19B-XTR
Open an elevated command prompt
Get your device name by running mst status
Format: mt<id>_pci_cr0
CD to the directory containing the firmware file
In our case was cd C:\User\6950X\Desktop\firmware
Run flint -d <Device Name> -i <Firmware File>.bin burn
In our case was flint -d mt26428_pci_cr0 -i fw-ConnectX2-rel-2_9_1000-MHQH19B-XTR_A1-A3.bin burn
Re run the version 5.10 driver installer choosing the “repair” option
Follow these steps on both machines
We now have functioning Infiniband NICs, but no subnet manager (which assigns an ID to the NICs, and creates a routing table based off of that)
I’m going to setup the subnet manager on our “NAS-PC” to keep any extra load off the “Gaming-PC”
Now, if you want, you can open up command prompt and run the OpenSM executable every time you turn on your PC, but that's way too much work.
Elevated Cmd>C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe
So, with the magic of Windows services, we can make it start on login in the background!
Start by opening cmd as administrator (Right Click Windows Icon in Bottom Left>Command Prompt [Admin])
Enter sc create <SERVICE NAME> binPath= “<FILE PATH TO OPENSM> --service” start=auto
Replacing <SERVICE NAME> with your desired name, we chose OpenSM
And inputting the file path which in our case is
C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe
Which equated to
sc create OpenSM binPath= “C:\Program Files\Mellanox\MLNX_VPI\IB\Tools\opensm.exe --service” start=auto
MAKE SURE, you have a space after the “binPath=”, and that you use quotation marks around your file path
Then just run sc start <SERVICE NAME>, which in our case is sc start OpenSM
To ensure the service is indeed working as it should
open Task Manager, go to Services
Find OpenSM and check that it’s status is Running
Now that we have a subnet manager, we need to assign each NIC an IP
We want our Infiniband cards to operate on a separate subnet from the rest of the devices on our LAN
Open cmd, run ipconfig
Look for your regular Ethernet adapter which in our case is Ethernet adapter Ethernet 3
Under that look for IPv4 Address which is listed as 192.168.0.20 for our host PC
Which means our regular network is running on 192.168.0.xxx
With that knowledge, I decided to assign our Infiniband NICs IP under 192.168.1.xxx which should prevent any conflicts
On each PC, search View Network Connections in the start menu
Right click on your Infiniband adapter which for us is Mellanox ConnectX-2 IPoIB Adapter
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties
Click Use the following IP address
For our “Gaming-PC”, we are going to assign 192.168.1.21 as the IP
The subnet mask should autofill as 255.255.255.0
Last, set the Default gateway to the IP of the other PC, which we are going to set as 192.168.1.20
For our “NAS-PC”, the ip is set to 192.168.1.20
With a subnet mask of 255.255.255.0
And a default gateway set to the IP of our host PC which in this case is 192.168.1.21
Should receive a “new network popup” asking if you’d like to enable network discovery, say yes on both machines.
To test that you do indeed have apc nam connection
Open cmd, and ping the opposite PC, which from our NAS-PC (192.168.1.20) would be ping 192.168.1.21
You’re probably thinking that you’ll need to unplug your ethernet cable every time you want to transfer files to your other PC over Infiniband right? WRONG.
To ensure your PC’s utilize Infiniband when doing transfers, we have to set the priority of the Network adapters, otherwise known as Metric
On each PC, search View Network Connections in the start menu
Right click on your Infiniband adapter, which for us is Mellanox ConnectX-2 IPoIB Adapter
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced
Deselect Automatic metric
Enter a higher number than your regular ethernet adapter, we are going to use 10
Right click on your regular ethernet adapter, which for us is Ethernet 3
Properties>Internet Protocol Version 4 (TCP/IPv4)>Properties>Advanced
Deselect Automatic metric
Enter a higher number than your Infiniband adapter, we are going to use 20
You now have a 40Gbit link between your two PCs.
Click on your Infiniband Adapter and then click View status of this connection
Should show a speed of 32.0 Gbps
To test the full 40Gbit (which after 8/10 encoding shows as 32Gbit in our connection status), we needed an extremely fast storage medium
So we reached out to the folks over SoftPerfect who hooked us up with a copy of their RAM Disk software for our testing, if you a ton of extra RAM and want to use it for insanely fast storage, definitely check them out!
Using their software we were able to obtain around 7 gigabytes per second sequential read & write on each PC
From our NAS-PC, we shared the formated RAM disk over the network, and it showed up immediately on the other PC in the Network tab (if your PC has a password, you will need to enter it at this step, unless you are in a homegroup)
If you were planning on using this as a long term solution, I’d recommend mapping a network drive to this shared folder/disk

Its awesome firewall. I still search for free full license and was hoping there will be a giveaway or something here

Its awesome firewall. I still search for free full license and was hoping there will be a giveaway or something here

Given: 12cents/kWh, 700 Watts, 8 hours/day... your looking at $250/year 
Being 2,000 Watts of cooling does not mean it draws 2,000 Watts of power.  It is a heat exchange, in this case while its running it draws ~700W which is half that of a hair dryer, and about the same as an enthusiast PC.  In addition a thermostat would mean it would only run when necessary not continuous. 
Most Home circuits are 20 Amp which is ~2400 Watts...  700W Linus PC + 300W NAS + 700 Wife PC + 700 AC Unit = 2400 W ...   So yes even under full load a single circuit could handle the described loading even if everything was being stress tested!   That being said tying the AC unit into the adjacent room circuit so its not loading the same BKR is Child Play.
Cost???   Sure its not free, $250 for the unit, and then some operating cost equivalent to having another PC in the house.  And the homes AC will have to cool the room that this DUMPS heat into...
This is LTT...  Pink Car, Custom Rack mount PC, Whole Room water cooling.... 

Hey guys,
Just want some community feedback on this. Most of these things are already in the queue to some degree. Just trying to prioritize...

absolutely epic!!!!!!!    now   I smell a super fun video where you all get schooled in csgo

Ever dreamed of what you could do with a $30,000 computer? Turns out the answer is "a lot"
 
Vessel link: tbd
YouTube: 
 

 
Parts List
 
Kingston 32GB DDR4 ECC RAM (x8)
Amazon: http://geni.us/3N2B
 
Kingston KC400 1TB Business SSD (x8)
Amazon: http://geni.us/3c6w
 
Lime Technology unRAID Server Pro Software
Manufacturer Link: http://lime-technology.com/
 
Caselabs Mercury S8
Amazon: n/a
Manufacturer Direct: http://bit.ly/1P8UvW3
 
Acer Predator X34 21:9 Gaming Monitor (x7)
Amazon: http://geni.us/32xr
 
Intel Xeon E5 2697 V3 (x2)
Amazon: http://geni.us/1nya
 
ASUS Z10PE-D8 WS
Amazon: http://geni.us/wov
 
AMD R9 Nano (x7)
Amazon: http://geni.us/Ubr
 
EVGA T2 1600W
Amazon: http://geni.us/3ad1
 
Cablemod Custom Cables for PSU
Manufacturer Link: http://bit.ly/1NYnrCz
 
Water Cooling Parts
CPU blocks - EK Supremacy Evo
Radiators - EK Coolstream XE 360
GPU Blocks - EK R9 Nano
GPU Connector - Custom 7-way version of FC Terminal
Pump/Reservoir - EK Xres 140
Fittings - EK Advanced Compression Fittings
Tubing - Primochill Primoflex Black/Orange