Dual Xeon, SLI Titan X, NVME, Water Cooled Workstation Build

[sysfailur]

First off, hello!  I've become a fan of Linus and figured I'd hop on here to document my build log.  Over the years I've been on HardOCP, Xtremesystems, and Hexus forums.  In 2001 my computer was the first to be featured in a print magazine with a water cooled setup in December 2001 in Maximum PC as a Rig of the Year.  I'm pretty proud of that and obviously I was a bit of pioneer back then if you will.  I did what I could with a limited budget.  

 

The idea behind this build is to fulfill the dream of my 16 year old self--a build which that 16 year old could not afford but my 32 year old self can.  I've always wanted a dual Xeon SLI setup and that is the game plan.  I want it water cooled but it's more for the purpose of being quiet.  It isn't going to be flashy as I don't even think the case has a window--I just want function, stability, reliability, and power while staying quiet and cool.  Overclocking will only be done on the graphics cards and only a little.  My years of peltiers, liquid nitrogren, dry ice, and super chilled water are over.  The last time I built a system was 2008 and consisted of:

 

Old build (some parts to be transfered):

Asus Rampage III Extreme X58, 24GB DDR3 g.skill, Core i7 930 @ 4ghz water cooled, Titan X 12gb (and various cards before that), Corsair 1200w, Lian-Li PCV1200, 1TB Mushkin Reactor which is what I'm currently using and maxed out on.

 

It used to have multiple Raid 5 arrays but I have migrated all that to an HP DL380p server that I got for a steal on ebay.  Specs of that server are Dual 6 core Xeon v4, 128GB, many hard drives.  Running multiple OS'es and servers under vmware esxi.

 

So then if the server is handling all the storage needs what is the goal of the new build?  It will be used primarily for photography work as that is my profession and as my old chipset was maxed at 24GB I have many years of growth on this motherboard as it maxes at 2TB.  My motherboard requirements were: Dual Xeon, SLI, Quad Channel DDR4.  The board I picked has all that.

 

 

Here's the new build specs:

Supermicro X10DAX

(2) x Xeon 2630 V4 ES w/ 3.1ghz max turbo (10 core each)

(4) x Black Diamond 32GB DDR4 2400 ECC Registered (128gb)

(2) x Titan X 12GB

Plextor 1TB NVME PCI-E 4x w/ heat spreader (similar to the Intel 750 but n$400 less)

1TB Mushkin being transfered

1200w Corsair being transfered

Lian Li PC-V1000LB (basically the same case as what I had but the updated version)

Soundblaster Z (been ages since I got a sound card so I figured I'd give it a go)

 

Cooling Setup:

(2) x EK Supremacy Nickel Acetal

Alphacool Eisbrecher Pro XT45 360mm Rad

Alphacool Eisbrecher Pro XT45 240mm Rad

(2) x Alphacool NxXxos GPX Titan X Block

Heatkiller Tube 150 Basic

Alphacool Black Compression fittings in straight and 90

(2) x Alphacool VPP655 PWM D5 Pump for redundancy

 

 

Pictures to come.  I took some but we're having severe wind storms in PA and my cable is having upload issues.  Figured I'd type this up at least first.

 

EDIT:  Pictures so far: http://clients.zacharyemerick.biz/Misc-ShotsTesting/2016-Workstation-Build/n-4R8k7L/

[shadowbyte]

wowowowowowowowowow

can't wait to see some pics

[NTDaws]

This looks sick. Have fun building this!

[sysfailur]

Thanks!  As long as nothing comes DOA I'll have fun

 

Processors are still coming in from Hong Kong and also waiting on the memory and the case but as soon as the case is in I'll start assembling.  I'm sure I'll have to order a few additional fittings and other goodies to make it all work.

 

Also forgot to mention my input/output:

LG 34" Curved 3440x1440 arm mounted to desk.

Sony 43" 4K TV ceiling mounted floating above the LG.

Behringer Studio Monitors

LG G810

Logitech G9X

 

Desk/Chair:

Herman Miller Desk

Herman Miller Embody chair

[shadowbyte]

2 minutes ago, sysfailur said:

Thanks!  As long as nothing comes DOA I'll have fun

 

Processors are still coming in from Hong Kong and also waiting on the memory and the case but as soon as the case is in I'll start assembling.  I'm sure I'll have to order a few additional fittings and other goodies to make it all work.

 

Also forgot to mention my input/output:

LG 34" Curved 3440x1440 arm mounted to desk.

Sony 43" 4K TV ceiling mounted floating above the LG.

Behringer Studio Monitors

LG G810

Logitech G9X

 

Desk/Chair:

Herman Miller Desk

Herman Miller Embody chair

holy shit I thought my new PC was going to be high end

rip in panini

[Shiv78]

holy hadamn 

[stationvsbox]

this build is crazy af

[AdamP]

I don't usually recommend SLI'ing anymore, but go ahead I guess, if you have the money. Beast of a build you got there, though! Have fun building it and torture the hell outta it when you're done.

[sysfailur]

I was borderline on SLI but I've always wanted to try it. I gave it a quick whirl last night but it wasn't working at all in Doom. It'll be a learning curve I am sure to configure things properly. I have time to sort out my needs before the actual build begins.

 

I am at work now but I think my internet is working again so I'll get pics up and do more testing in a few hours.

 

I currently play Doom, WoW, Swtor, and only occasionally really as I am too busy with work.

 

If anyone has tips or a link to a good SLI guide please krt me know. I'm also taking 120mm fan recommends for quiet low rpm speeds and longevity. I don't know what the best brand is these days.  My current setup has Delta fans on a rheobus and they're stupid loud. I am over it.

[sysfailur]

@AdamP It is a continuing theme that we car guys also like computers. Here is my STi and the reason why I haven't upgraded my PC in years.

 

Build gallery:

 

http://clients.zacharyemerick.biz/Cars/Zacks-2008-STi-V-Mount/

 

[sysfailur]

Pictures so far: http://clients.zacharyemerick.biz/Misc-ShotsTesting/2016-Workstation-Build/n-4R8k7L/

 

Cleaned em up real quick.  Obviously I'm not getting fancy with the shots just trying to document the goodies.

 

 

 

 

[Shiv78]

What camera are you using??

[sysfailur]

Those pics are a Sony A7 with Zeiss Batis 25mm.  I'm all Sony.  A6000, A7, A77, many lenses.  I'm watching/waiting and eyeing the Hasselblad X1D for next year.

[sysfailur]

Gallery link added to previous photo post.

[AdamP]

Ah, I see! Didn't catch that you were a car guy too! Although, I don't really think it's that uncommon either. However, actively seeking out such hobbies can be expensive, which is why I think most people don't do it, obviously. But when one enthuses about something so captivating as building computers, I find it rare that such an individual wouldn't try to apply his/her mind to as many things as he/she could get their hands on.... It's what keeps us happy, getting our hands dirty. One of the few things on this little rock in an indifferent universe that allow us to feel connected to something greater, the raw reality of power and speed. Ha, sorry to digress so much on a random forum on the internet; it's just nice to be reminded of that fact and be grounded by that solace by another random enthusiast.

 

Live long and prosper, random enthusiast... 

 

EDIT: Thought I should just add that I just recently traded in my Prius after having saved up enough to cover the difference for a 2017 WRX STI, so I'm stoked and don't even know where to begin. I have so much planned for it. I've always been a bike guy, since they're always so reliable, cheap and quick. But I just didn't use my Prius enough, since I daily my bike.... So, I just decided to pull the trigger and get my hands dirty with another project and pick up a sedan with a reliable 4-stroker that packs a punch... and that obviously led me to the WRX. I wanted 4 doors and cylinders; otherwise, I would have gone for a Nizzy. Ha, hope I made the right decision. Lol.

[AdamP]

Also, so happy to see another SuperMicro fan! Loved them for years, just have been a little skeptical of where they've been going lately trying to tap into more markets, like the mainstream-consumer market, for example. Their gaming boards are so gaudy, less gaudy than most gaming boards, but still. One of the biggest reasons I've been a fan of SuperMicro, is their dedication to performance and utility, and nothing else. I miss that these days.... Their boards have always been about just getting the job done. Even today, I'm starting to wonder if they've forgotten about that quality-driven mindset that got them to where they are now. Now, I understand that the market they are trying to tap into isn't so much concerned about that; and they're wise to follow market trends to make profit.... *sigh* I just wish someone would make proper enthusiast boards that didn't care about looks, hell just leave it all black. To me, that's what is truly beautiful and quality. Hell, I guess my mind is stuck in the nineties and its bulky, boxy design. But everything was built like a tank, and it all just worked, for like forever. Still got my Model M lying around... still feels amazing to type on. Mostly everything these days feels so cheap. You gotta go looking far and wide for the real quality now. I don't know, maybe I've just been left behind, and I've lost touch with what's more important about where we're headed.

 

EDIT: Wow, I talk a lot. Sorry!

[Urishima]

21 hours ago, sysfailur said:

@AdamP It is a continuing theme that we car guys also like computers. Here is my STi and the reason why I haven't upgraded my PC in years.

 

Build gallery:

 

http://clients.zacharyemerick.biz/Cars/Zacks-2008-STi-V-Mount/

 

Nice ride. Tho I personally prefer the sedans from the early 2000. Can't beat a bugeye, imho.

[sysfailur]

I really hope I like this Supermicro board.  I looked at the Asus, Gigabyte and Supermicro platforms which all had similar specs and went with Supermicro.  I know that the Gigabyte can supposedly run the memory at a higher speed than the other two but again I was going for utility and reliability and that's what Supermicro does.  Not to say I have had bad luck with Asus because every board I've bought from them has been a champ--their WS16 C612 chipset board however is a bit hard to get a hold of as well and this X10DAX is much more available which was another deciding factor.

 

I played around with SLI yesterday quite a bit and found that it likes certain games and doesn't like others.  Now I want to order Battlefield 1 haha.  I have to try SW:Battlefront today which I forgot to try yesterday.

 

Looks like if the fans that come with the Lian Li are rubbish I will go with Noctua's Industrial 3 phase fans.  On the note of fans does anyone have a suggestion for a software integrated rheobus?  I don't want to use my 5.25" bay for anything other than installing a spare mesh in it.  I am going optical disk less on this system.  I'm thinking perhaps using a Raspberry Pi as a base for a fan controller and temperature monitor all in one unit.  Haven't really looked into it much but I'm sure that exists.

 

 

Re: my STi.  I blew it up racing auto-x.  I did a 23 month long build on it with many custom components, fully built engine, etc. I planned initially to use it for auto-x but this car is too much car for that so I'm now looking at hill climbs and track days/events.  Really all that is left for it is a slightly larger turbo, E85 flex fuel, full 12 point cage, 6 pot brake kit for the front, and aerodynamics.  I've pretty much taken care of all other aspects of the car.  It's a monster! Currently ~400whp with very fast spool GTX3071 .63 turbo

[sysfailur]

A little update.  Just put in an additional order for the following after deciding how I'm going about this watercooling setup.

 

I'm going to run two loops.

Graphics loop: SLI w/ parallel joiner -> 360mm Rad -> Pump/Res combo.

CPU Loop: 240mm Rad -> Y splitter -> parallel xeon cooling -> Y Splitter -> Pump/Res combo.

 

I'm going to return the 150mm resivoir which I screwed up on.  Should've ordered the integrated D5 pump version so I instead ordered two 200 D5 Heatkiller Tubes and I'll be integrating my Alphacool D5's into it.

 

I ordered the SLI joiner, Y's, second Titan X block, the two new Res's, and 5 x Noctua 120mm fans.  I also picked up an Alphacool Heatmaster II which I will be putting some heatsinks on that I have laying around which will control/monitor the pumps and 5 fans. I also picked up 4 temperature probes and have many laying around to scatter around the build.

 

Ordered 25' of Tygon R-1000 which is my personal tubing of choice over R3603. I actually had some but it is really old and some is discolored so I decided not to use it.

 

With all that said I think nothing else needs to be ordered and I'm gtg on this build.  I contacted FrozenCPU about my Lian Li case and they have a contact in Taiwan who is shipping it out as their American supplier is out so that will most certainly be delaying this build.   The particular case I got is a special edition so perhaps that is part of the issue being that it is harder to get.

[sysfailur]

Apologies for the long wait for an update.  The system is up and running but I am now realizing that there are a few things that need attention.

 

First, I screwed up with the quad channel aspect.  I need four more dimms for that.  Right now I'm just on two channels.  So I'll probably just throw another 128gb in.

 

Second, this system is VERY tight!  What I'm talking about is clearances.  I ran into more clearance and fitment issues that I can shake a stick at.  Both radiators, the pump/res placement, fan placement, memory placement, ATX power connectors, power headers, SLI and PCI-e fitment issues, pci-e to power supply fitment.  It goes on and on.

 

Third, the Heatmaster II that controls my fans works, and the software works but.... it's all in German.  Anyone know how to change it to English before I google translate every page in the program?

 

Fourth, the Intel X540 T2 dual 10Gb card is not giving me dual anything right now.  I've swapped from a 16x running at 8x to an 8x slot and there hasn't been any improvement in functionality.  I can run a single 10Gb stream and sometimes it'll run two x 1Gb.  I'm thinking it is a bios config that is wrong b/c they are VERY intricate.  I am waiting to hear back from Supermicro.

 

Fifth, I ordered the wrong water SLI asymmetrical bridge so right now I'm running my GTX770 until that comes in.

 

Finally, the big issue which I'm working on fixing is that although there are physically enough slots for all my cards, the back plate and front plates and dual-slot nature even watercooled of the SLI setup are making it so that I physically can't fit all three cards (nvme, sound blaster, 10gb) along with the SLi.  Also none of the cards fit on the 4x top slot because of the back plate of the top most card.  So I am now looking into PCI-e riser cables haha.  I wlll rig this thing up so it WILL work however I have to.  I figure I can rise the NVME and mount it on the top of the power supply and then trim the heatsink of the 10Gb card to make it all fit.

 

Oh what else... I dunno here are some horrible pictures I've put no effort into haha--I've put about 5 days into this over the holidays.  Final pictures to come by next weekend I imagine.  Btw, 20 cores and 128gb of memory is f'in GLORIOUS in use.  I look forward to tossing the Titans in.  I also bought another Eaton 5PX 1500w UPS for this new system.  My server/network has its own as well.

 

And the stupid heatsink setup on the mosfets of the heatmaster II have been ditched.  I thought it was worth a try.  I will just glue on some low profile heatsinks I have down the road.

 

 

[sysfailur]

Here is a shot of what I hoped to have as far as my setup but when the waterblocks and joiner were installed I found it to be impossible as the bottom 90 degree fitting would've been 1" inside the power supply.  I *don't* really want to cut a hole in that.

 

 

[sysfailur]

Last update... temperatures.  I am idling at 30C, after running BOINC at 100% cpu use for 30 minutes temps rose to 41C.  The airflow clearly needs to be sorted out.  There's another problem.  The top radiator is for CPUs and I can't physically put two fans on it.  I'm going to create a duct out of acrylic and mount the fans on the door panel and blow them in at a 90 degree angle to the radiator.  The memory is too close to the radiator.  I actually only have one screw in on the one fan on the top rad!!!

[sysfailur]

Also as I only have two 1 TB SSD's in this PC if you're wondering where my storage is.  Here's my network/server/rack:

 

 

[LittleCarrot]

Google Translate can translate pictures.

[sysfailur]

Here's the completed setup.  A few things I've noticed... it will be tight to fit in one of the four new sticks as the graphics card backplate is close to it but it should work.  I'll probably run electrical tape on the backplate to avoid contact when I get them.  I've removed the 10Gb card because it's glitching out and I also have no room for it.  I plan on getting another to install where the PCIE M2 SSD is, and then use a PCIE extension to also hook up the M2.

 

This whole project has been a "too much in too small of a space" but I've made it work.  And yes, I realize there's plenty of extra tubing.  I wanted flexibility to move things around without draining.  Anyway, few small things to do but there you have it.