Samishere

I am using ConnectX-2 cards on 3 Windows 2012 R2 servers, and 1 Windows 10 workstation. I am using a Voltaire 4036 switch.
 
At the moment I am just utilizing Infiniband for file transfers. Here are some of the nuggets from my experience.
 
If you are using the MHQH19B-XTR or any ConnectX-2 gen card, firmware is very important. The "latest" firmware offered by Mellanox is something like 2.9.1000, however RDMA is NOT functional on this firmware version. There is newer firmware that is no longer available that you need to flash in order to support RDMA. I am using 2.10.720 on my MHQH19B-XTR adapters and RDMA is functioning just fine. I found this firmware EXTREMELY hard to find so for your convenience it is attached, and the flashing instructions in the OP still apply for this firmware.
 
In the OP, an older version of WinOF was recommended. I am not sure why, but WinOF 5.35 for ConnectX-2 works just fine for me, and actually displays the connection as 40Gbps instead of 32Gbps even though there is no difference.
 
RDMA is not available on Windows 10 professional so you will not be able to utilize the maximum line speed unless you are using a Windows Server operating system (version 2012 or newer).
 
Using a Voltaire 4036 switch? You will need to be on firmware version 3.9.1 for RDMA to function properly. Google yourself a guide to do so, but in my quest I will say that most guides are written around using the serial management port and PuTTy, but the path of least resistance (and eBaying serial adapters) is SSH-ing into the eth port using PuTTy. My eth port wanted to request a new IP address from DHCP every 10 minutes or so, so I would also recommend giving it a DHCP reservation so you can keep a stable PuTTy connection.
 
EDIT: I would also add that cable length is important in achieving stable and maximum speeds. If using copper cable, I wouldn't go over 3 meters if you intend on running your link at QDR. If using fiber length isn't an issue, but prices go way up. If you are having cable issues you aren't going to get a notification that the cable is bad, if you manage to get a 40G link your transfers will stall out and you'll see protocol errors in the SMBClient event logs.
 
MHQH19B-XTR Firmware 2.10.720.zip

Greens are "low power", they run at 5400RPM and assume a might lighter workload.
Blues are 7200RPM, but still assume a "desktop" workload.
Reds are either 5400RPM or 7200RPM depending on the model, but assume 24/7 operation.
Blacks are 7200RPM and have a few optimizations to improve performance, but are still engineered towards the desktop user.
SE and RE drives are Enterprise and come with a better warranty, and are engineered for 24/7 heavy usage.

Well, you could get two of these refurb RE3 drives for $38 each, and then if your server is Windows based you could use software RAID1 to have the redundancy and increased read speeds. http://www.newegg.com/Product/Product.aspx?Item=9SIA7XW3R66557
Two refurbed enterprise drives in RAID1 would be a safer bet than a single desktop class drive if you cared about the data.
That would be my suggestion because I've lost a drive before that contained important stuff.

I don't think that chip will get hot at all.

Wish you the best on the build dude. You should document it and post about it in the forums 

Ivy Bridge and Haswell use TIM, that's the problem. They used to Solder the IHS on.
That's why it's so desirable to delid and apply better TIM: 
 

+1 for normal. The TIM under the built-in spreader on those CPUs isn't so great, so unless you are water cooling you are always going to be several degrees above ambient.

Dude, SMB multichannel can't leverage multiple 1Gb ports on the client to a singe 10Gb port on a server. All the info is here: http://blogs.technet.com/b/josebda/archive/2012/05/13/the-basics-of-smb-multichannel-a-feature-of-windows-server-2012-and-smb-3-0.aspx
It works by finding multiple IP paths and then leveraging RSS or RDMA to initiate multiple channels over those IP paths. So each 10Gb port on a server would only establish one link at 1Gb for the client. So we are back to needing a boat load of ports on the server to support a somewhat decent user experience. Yes I know not every user would be utilizing it all the time at the same time, but if you didn't plan to support (and please pay attention) the best experience for as many users as is possible, then any investment you made to make this whole complicated setup work in the first place would be a waste the moment your users experienced the choke out of resources. That's all the user would care about.
Also, 10Gb NICs are becoming standard on servers at near commodity prices, but switches and cabling are still crazy expensive.
You keep bringing up where you work and such and I don't know what bearing that has here because this tech clearly wouldn't work with what your company does, unless it was for accelerating very isolated workloads of a particular unit in your organization. Hey, I'm the Network Administrator for a small company and there is no way I would commit the resources to build this tech out in our single site. If you tried to deploy something like this on an enterprise scale even though it's clearly not viable you would git wrecked.
Again, the initial point was that it's cool tech, but it can't possibly scale to be useful enough for anything beyond a speedy setup for a personal network where you only need quick SMB transfers. I'm pretty sure you agree, but you keep insisting that you could make it work, but you really can't.

I opened this from USA, but maybe it's valid for you too http://www.amazon.co.uk/Seasonic-80PLUS-Bronze-Power-Supply/dp/B00CBJ2D9O/ref=sr_1_1?ie=UTF8&qid=1454434186

The simple answer would be yes they are more likely to fail, but software RAID1 on linux is fairly simple as well. I'm sure you could find a clear guide on how to do so.

There was a time when they were considered the same drive as the RE models but with disable firmware features to keep them in the desktop market, but I don't know if that's still accurate. They are still marketed toward desktop, so if you were seeking out reliability I would still look for a Red, SE or RE because they are engineered for that purpose and have a better warranty to back it up.

There was a time when they were considered the same drive as the RE models but with disable firmware features to keep them in the desktop market, but I don't know if that's still accurate. They are still marketed toward desktop, so if you were seeking out reliability I would still look for a Red, SE or RE because they are engineered for that purpose and have a better warranty to back it up.

Greens are "low power", they run at 5400RPM and assume a might lighter workload.
Blues are 7200RPM, but still assume a "desktop" workload.
Reds are either 5400RPM or 7200RPM depending on the model, but assume 24/7 operation.
Blacks are 7200RPM and have a few optimizations to improve performance, but are still engineered towards the desktop user.
SE and RE drives are Enterprise and come with a better warranty, and are engineered for 24/7 heavy usage.

If the data you are storing is of value to you at all you might want to look at at least RAID1. Nothing hurts more than valued data that you care about being lost in to the ether.

That was my original point that I've been saying. Building your network around SMB 3.0 Multichannel for increased SMB file transfer doesn't scale, not even to the scale of LMG.

SMB 3.0 Multi-Channel is a SMB file transfer optimization only. Client and Server need to be 8 or above or server 2012 or above, I.E. Windows to Windows only. Your NICs on both ends need to support RDMA or RSS. If your NICs are teamed then you can't use SMB 3.0 Multichannel. You basically need all your NICs on both systems to be basic NICs where each connected port has it's own IP address. SMB 3.0 Multichannel can then find the addition paths between client and server using RDMA or RSS.
Here is the consideration:
SMB 3.0 - Faster SMB File Transfers, but all other network communication will max at the bandwidth of a single NIC port. Only SMB file transfers will be dynamically redundant.
LACP or other Teaming Solution - Any one client can only use the bandwidth of a single NIC port, but multiple clients can be served up the the total aggregated bandwidth on any protocol. All network communication is dynamically redundant.
So if your goal is to get faster SMB file transfers only then config your systems for SMB 3.0 Multichannel, but if you have multiple clients connecting to the same server or need higher aggregated bandwidth for other protocols or need full redundancy then use LACP or another teaming solution.
Honestly, to make SMB 3.0 work for a multi user environment the server needs crazy CPU horsepower, crazy fast storage, and as many NIC ports as there are NIC ports on all your clients combined. And again, only your SMB transfers will be accelerated and router management will be a nightmare because each system will have multiple IPs and your server will have a TON of IPs, and physically you will have so many cables to run.

I use Furmark. It's small, it's free, it does what you need. If you are worried about bench cheating just change the EXE name.
 
I have to comment on the 390X though because I had one. I dumped it within a week because it chugged wattage, and heated my case up like a jerk. I changed the thermal compound to the Gelid GC Extreme TIM, and it still thermal throttled. I had a Gigabyte Windforce variant and the card still idled in the high 60C range.
 
I would call the R9 390X broken. It was a cost saving move from AMD to refresh their lineup and make some sales. They just pumped more power in the R9 290X for higher clocks and called it a new card, and the PCB can barely handle it.

That's an amazing deal as long as it all works.

Your cooler has a fan connector that needs to be plugged into a specific fan header on your motherboard for RPM to be reported.
 
It's usually marked 'CPU_FAN' on the motherboard, so you need to locate that and plug the connector in.