Research Rig

[RichL]

Hi All,

I am currently a PhD student, interested in deep learning. I am building a personal rig to use for my research and some challenges (and likely gaming because I won't be able to resist). I have a good amount of savings, and I may be compensated if my research using it is successful, so the price is not such a huge issue (just sadly can't splash out $15000 on this!).

Here are the parts I have picked out:
https://uk.pcpartpicker.com/user/RichLewis42/saved/btwXsY

My main goal is extensibility and performance - I hope to add a couple additional Titan X in SLI later, and at that time add water cooling to the system.

(https://uk.pcpartpicker.com/user/RichLewis42/saved/thsCmG).

My worries are:

1) Motherboard

 

I have been suggested Asus for stability and overclocking, so that is why I'm using what I have in the build currently.  I could go with the X99-A for cheaper, and sacrifice the possibility of 3 SLI - anyone have any suggestions for good motherboards?

 

2) Power supply

 

Obviously its overkill at the moment, but with 3 Titan X would this still be overkill?  Any advice for another one? Power efficiency is quite important as the system will be running at max for extended periods of time.

 

3) GPU

 

is there any difference between the different models, or is buying the EVGA one fine? Are the others just with different casings etc. and/or overclocked in the factory, or is there a fundamental difference?

 

4) Storage

 

I don't know anything about storage, I assume I will need an SSD for quick read/write as my datasets may be large and not fit in memory.  For large storage, will a single mechanical drive be fine or should I do something more elaborate (e.g. RAID)?  

Thanks for your time,
Rich

[_JohnB330ci]

1) I think the current board is fine.

2) It isn't overkill it will draw quite a high wattage and its nice to have a bit of head room.

3) Buying the EVGA one is fine, even if you decide to watercool them.

4) Get 2x 256GB Evo 850 SSD's, and a 2tb WD Black 

 

Thinking of it I would think about switching that case to an S340 instead, the H440 isn't great for cooling so your system might run a little bit hot.

[REX.exe]

1.) motherboard is great, will allow for a huge e peen to.

2.) Please don't do 3-way SLI or even get a Titan X for that matter. Scaling is horrible and a 5820K bottlenecks 3 Titan Xs. 850W is more than enough.

3. The GTX 980Ti is a much better price to performance card (JUST worse than the Titan X) and is MUCH cheaper and comes with much more different models. I would suggest a MAX of two of these. 3-way and bottlenecks and scaling issues will start.

4. Honestly for an all out system like this 730 is a fine choice. You could always go 2X256GB 850 Pro though.

 

Other things I changed:

 

Case: For an all out system you need an all out case, I think the Enthoo Luxe fits into that bracket JUST nicely

Power Supply: Downgraded to 850W, 1200W is overkill as fuck and not necessary.

Cooler: I think you need better cooling than a 212 EVO for a high end system

GPU: Changed the G1 Gaming 980Ti, it'll look better and even perform better in some cases when overclocked (if you want to).

 

Besides that your system is great! Here is what I'd suggest. Also, go for a 1440p monitor or 4K if you really want to. For 1440p ASUS ROG Swift is AMAZING. There also plenty of sexy LG Curved Ultrawides out that that Linus has reviewed that are also amazing options.

 

http://uk.pcpartpicker.com/p/yT2Lgs

@RichL Please follow your topics.

[Delusional Phil]

PCPartPicker part list / Price breakdown by merchant

 

CPU: Intel Core i7-5820K 3.3GHz 6-Core Processor  (£296.34 @ Aria PC) 

CPU Cooler: Cooler Master Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler  (£24.95 @ Aria PC) 

Motherboard: Asus SABERTOOTH X99 ATX LGA2011-3 Motherboard  (£272.94 @ More Computers) 

Memory: Corsair Vengeance LPX 32GB (4 x 8GB) DDR4-2400 Memory  (£209.55 @ Amazon UK) 

Storage: Intel 730 Series 240GB 2.5" Solid State Drive  (£125.53 @ CCL Computers) 

Storage: Seagate Barracuda 3TB 3.5" 7200RPM Internal Hard Drive  (£73.14 @ Aria PC) 

Video Card: EVGA GeForce GTX Titan X 12GB Video Card  (£803.22 @ More Computers) 

Case: Fractal Design Define R4 (Black Pearl) ATX Mid Tower Case  (£72.48 @ Aria PC) 

Power Supply: SeaSonic X Series 1250W 80+ Gold Certified Fully-Modular ATX Power Supply  (£170.54 @ Scan.co.uk) 

Total: £2048.69

Prices include shipping, taxes, and discounts when available

Generated by PCPartPicker 2015-07-10 14:31 BST+0100

[Dzzope]

 

My worries are:

1) Motherboard

 

2) Power supply

 

3) GPU

 

4) Storage

 

First, follow your own topics..

 

1 Motherboard is fine, though it has a 64GB limit vs MSI / Asrock / Gigabyte etc 128GB.. Does have the best bios though.

 

2, Way overkill.. a  system with a 4930K @ 4.7GHz (Overclocked) will use around 165W.. each Titan will use about 230w without an overclock (at full load) 100W is fine for an overclock and some head-room on 3 way titans 1200 is a little tight for 4-way..

 

3. stock is stock.. Pick the best warranty.. Though I think you can get a custom version of the titan now.. which would be good as the cooler isn't quite up to the job at keeping it running the best.

 

4. Really depends on what your actually doing.. As does if your best going for titans.. is the cuda cores going to help?

Would you be better off with double precision

Would you be better off with a better compute performance of AMD?

Do you need all that Vram. cos a 980Ti is so close to titan but allot less money.. only reason to go for it is the 12GB.

 

If this is for important work, you need a more robust storage solution.

 

Look at ebuyer for PSU's.. they sometimes have great ones at really good prices with free delivers (special offers) 

 

This is what I would build.. this is a work PC so the case is just a box (with good airflow and 3 or 4 included fans)

980ti's in, just in-case they suit.. they are like 95% of the power at 65% of the cost.. though 50% the Vram..

EVGA's G2 PSU is awesome and half the price..

Intel 750 ssd.. you will not have a bottleneck with that in there... if Fast storage isn't so important, drop back to a samsung 850 evo, ocz arc 100 or whatever fits the budget.

 

PCPartPicker part list / Price breakdown by merchant

 

CPU: Intel Core i7-5820K 3.3GHz 6-Core Processor  (£296.34 @ Aria PC) 

CPU Cooler: be quiet! PURE ROCK 87.0 CFM Sleeve Bearing CPU Cooler  (£22.74 @ Aria PC) 

Motherboard: MSI X99A SLI PLUS ATX LGA2011-3 Motherboard  (£164.99 @ Ebuyer) 

Memory: Crucial 32GB (4 x 8GB) DDR4-2133 Memory  (£163.07 @ More Computers) 

Storage: Intel 750 Series 400GB PCI-E Solid State Drive  (£310.74 @ Aria PC) 

Storage: Seagate Barracuda 3TB 3.5" 7200RPM Internal Hard Drive  (£73.14 @ Aria PC) 

Video Card: EVGA GeForce GTX 980 Ti 6GB Superclocked ACX 2.0+ Video Card (2-Way SLI)  (£563.65 @ Dabs) 

Video Card: EVGA GeForce GTX 980 Ti 6GB Superclocked ACX 2.0+ Video Card (2-Way SLI)  (£563.65 @ Dabs) 

Case: Zalman Z3 Plus ATX Mid Tower Case  (£25.29 @ Amazon UK) 

Power Supply: EVGA SuperNOVA 1300 G2 1300W 80+ Gold Certified Fully-Modular ATX Power Supply  (£146.98 @ Dabs) 

Total: £2330.59

Prices include shipping, taxes, and discounts when available

Generated by PCPartPicker 2015-07-10 14:47 BST+0100

[Nena Trinity]

That build is insane also bottleneck!

[RichL]

Hi all,

 

Thanks for all the feedback, apologies for not following the topic! I'm new here...

 

Just to put some sources that I used to when putting my build together for anyone interested:

Hardware guide for Deep Learning

NVIDIA DevBox build guide (you may need to register with NVIDIA to read this) 

 

Composing comments by the worry areas:

 

Motherboard - No one really disagreed, I don't think I'll need more than 64GB so this could be alright.

PSU - It certainly is overkill for the given, but won't be if I do put 3 Titan X in

GPU - @Dzzope and @REX.exe were concerned about the Titan X being too much, and the planned 3 way SLI would bottleneck severely.

Storage - Possibly get a more robust storage system.

Case - Possible cooling problem @SherifsDog22

 

I appreciate the worries about the Titan X, but the 12GB VRAM is the reason why it is used in deep learning, as you can train much larger networks with it.  Most algorithms out there can only run on a single card - I would rather not limit my research by buying two smaller cards when I could buy one larger one now.

 

I am sure that bottlenecking would be a huge problem for gaming with 3 in SLI, but I don't know if that is the case with deep learning - NVIDIA suggest to use 4 in SLI with the i7-5930K in their DevBox.  Certainly willing to be corrected though.

 

I am tempted to scrap the theoretical 3rd card and go for:

PCPartPicker part list: http://uk.pcpartpicker.com/p/x6srQ7

Price breakdown by merchant: http://uk.pcpartpicker.com/p/x6srQ7/by_merchant/

 

CPU: Intel Core i7-5820K 3.3GHz 6-Core Processor  (£296.34 @ Aria PC) 

CPU Cooler: Cooler Master Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler  (£24.95 @ Aria PC) 

Motherboard: Asus X99-A ATX LGA2011-3 Motherboard  (£196.04 @ Dabs) 

Memory: Corsair Vengeance LPX 32GB (4 x 8GB) DDR4-2400 Memory  (£209.55 @ Amazon UK) 

Storage: Samsung 850 EVO-Series 250GB 2.5" Solid State Drive  (£68.34 @ Aria PC) 

Storage: Samsung 850 EVO-Series 250GB 2.5" Solid State Drive  (£68.34 @ Aria PC) 

Storage: Western Digital BLACK SERIES 2TB 3.5" 7200RPM Internal Hard Drive  (£95.74 @ Amazon UK) 

Video Card: EVGA GeForce GTX Titan X 12GB Video Card  (£803.22 @ More Computers) 

Case: NZXT H440 (White/Black) ATX Mid Tower Case  (£88.07 @ Aria PC) 

Power Supply: Corsair 860W 80+ Platinum Certified Fully-Modular ATX Power Supply  (£164.26 @ CCL Computers) 

Total: £2014.85

Prices include shipping, taxes, and discounts when available

Generated by PCPartPicker 2015-07-10 16:56 BST+0100

 

This will be 2 SLI, but cheaper. I downgraded the motherboard and power supply. I changed the storage to that recommended by @SherifsDog22 - is there a particular reason for these?  Do I run the SSDs in RAID0?  Why are they so much cheaper than the one I picked out?

 

The case I chose is 'silent' which will be a plus with the gf if it is going to be running for days - water-cooling should address any cooling problems right?

 

Thanks again,

Rich

[Dzzope]

For the Titan.. Fair enough.. as long as the Vram is that important to you, then it's worth it.

 

If the programs your using will only run on 1 card, stick to a single titan unless you going 4k or something for the games.

 

If you can use 3 cards for non-gaming reasons.. go for the EVGA G2 I had in my build.. GREAT unit and costs a little less than even the 860 from corsair. and will give you all the room you need... If your going to stick to 2 cards, the G2 850W is plenty, much cheaper and top notch quality too.

 

You can pout the savings into either a beefier CPU cooler to boost the clocks if your overclocking.. If not overclocking then you May be able to squeeze in some extra cores...

 

The SSD's are much newer, they are fast and cheap.. The 730 is an awesome drive but just outclassed by it's age.. if they were about the same price I'd go the 730..

 

The 750 is a new type of drive using NVME and is MANY times faster than anything else and about the same per GB as the 730..

 

The H440 will be fine for your uses and the Titan having a blower style heatsink will help keep case and cpu temps in check with the more restrictive airflow. However the Define R4 has similar style and about as quiet but better airflow (and cheaper).. 

[Mattias Edeslatt]

How important is error correcting (ECC) memory for your research? Not fun to do big simulations and there shows to be a error related to the memory.

 

So the question is if you need workstation grade components or regular hardware is good enough?

[RichL]

How important is error correcting (ECC) memory for your research? Not fun to do big simulations and there shows to be a error related to the memory.

 

So the question is if you need workstation grade components or regular hardware is good enough?

 

Deep learning is pretty much a minimisation problem, so there doesn't tend to need error correction.  Most research is done on gaming cards.  We have a 2 Tesla K40s in our lab that I've tried, but they are usually in use for long running simulations, which is why I'm building my own thing. 

 

Thanks for thinking about that though, their simulations don't run well on gaming PCs.

[RichL]

For the Titan.. Fair enough.. as long as the Vram is that important to you, then it's worth it.

 

If the programs your using will only run on 1 card, stick to a single titan unless you going 4k or something for the games.

 

If you can use 3 cards for non-gaming reasons.. go for the EVGA G2 I had in my build.. GREAT unit and costs a little less than even the 860 from corsair. and will give you all the room you need... If your going to stick to 2 cards, the G2 850W is plenty, much cheaper and top notch quality too.

 

You can pout the savings into either a beefier CPU cooler to boost the clocks if your overclocking.. If not overclocking then you May be able to squeeze in some extra cores...

 

The SSD's are much newer, they are fast and cheap.. The 730 is an awesome drive but just outclassed by it's age.. if they were about the same price I'd go the 730..

 

The 750 is a new type of drive using NVME and is MANY times faster than anything else and about the same per GB as the 730..

 

The H440 will be fine for your uses and the Titan having a blower style heatsink will help keep case and cpu temps in check with the more restrictive airflow. However the Define R4 has similar style and about as quiet but better airflow (and cheaper).. 

 

Thanks for the suggestion - I was going on the power efficiency, as the electricity costs add up if you run the system at full power - I'll definitely consider that G2 though.

 

The CPU cooler is cheap as I plan to water-cool everything eventually.  I was thinking about getting a H100i for the time being, but it is pretty expensive.

[Dzzope]

Thanks for the suggestion - I was going on the power efficiency, as the electricity costs add up if you run the system at full power - I'll definitely consider that G2 though.

 

The CPU cooler is cheap as I plan to water-cool everything eventually.  I was thinking about getting a H100i for the time being, but it is pretty expensive.

It would want to be VERY expensive electricity to Justify the price difference... From gold efficiency to Platinum isn't a huge jump

[REX.exe]

1. Yeah motherboard is fine.

2. 850W is more than enough for one. And if you plan to only utilizize the Titan's VRAM. Two or more won't stack so you'll still have 12GB.

3. I would just stick with the Luxe as it' got lots of water cooling support and is just plain sexy. Phanteks makes some of the best cases. the Luxe being one of them!

4. Just add more drives if you need more space...

5. Just stick with Titan as having multiple ones will ONLY add more computing power and NOT more VRAM. And I'm sure if you don't have enough power to run some simulations you can just borrow a K40 or something.

[REX.exe]

Deep learning is pretty much a minimisation problem, so there doesn't tend to need error correction.  Most research is done on gaming cards.  We have a 2 Tesla K40s in our lab that I've tried, but they are usually in use for long running simulations, which is why I'm building my own thing. 

 

Thanks for thinking about that though, their simulations don't run well on gaming PCs.

Well a Titan is a gaming card...

[REX.exe]

Thanks for the suggestion - I was going on the power efficiency, as the electricity costs add up if you run the system at full power - I'll definitely consider that G2 though.

 

The CPU cooler is cheap as I plan to water-cool everything eventually.  I was thinking about getting a H100i for the time being, but it is pretty expensive.

If you're going to be swapping out cards frequently (for better performance) than I'd suggest you don't water cool. Unless you plan to cool ONLY the CPU.

H100i is a good option. You could also look at the 240m or 280L, Raijintek Triton or a Cryorig R1 Ultimate (it's an air cooler).

[brob]

The i7-5820K only supports 28 PCIe lanes. See the caution in https://timdettmers.wordpress.com/2015/03/09/deep-learning-hardware-guide/. You will need/want at least an i7-5930K if you wish to communicate at 16 lane speeds with each of the gpu.

 

You may want to consider a 4 lane PCIe ssd drive like one of the Intel 750, Intel 750 Series 400GB PCI-E Solid State Drive and Intel 750 Series 1.2TB PCI-E Solid State Drive; or a Samsung M.2 like Samsung SM951 256GB M.2-2280 Solid State Drive and Samsung SM951 512GB M.2-2280 Solid State Drive. These drives offer significantly better performance than SATA III ssd.

 

Aside: RAID 0 may not provide better performance than a single larger ssd. It really depends on the pattern of i/o.

 

I am not sure that the Hyper 212 EVO is going to be able to do a decent job given how cpu bound your work may be. You might look at Noctua NH-U14S 55.0 CFM CPU Cooler or even an AIO, depending on how quickly you plan to move to a custom cooling loop.

 

If you are planning to add a second gpu, 860W is a bit tight. Especially if the gpu are going to be overclocked.

[RichL]

It would want to be VERY expensive electricity to Justify the price difference... From gold efficiency to Platinum isn't a huge jump

Yeah I didn't think it would be that much, but there was a mention of the price in the guide that suggested it would build up. I'll probably end up going with your recommendation, thanks!

 

 

1. Yeah motherboard is fine.

2. 850W is more than enough for one. And if you plan to only utilizize the Titan's VRAM. Two or more won't stack so you'll still have 12GB.

3. I would just stick with the Luxe as it' got lots of water cooling support and is just plain sexy. Phanteks makes some of the best cases. the Luxe being one of them!

4. Just add more drives if you need more space...

5. Just stick with Titan as having multiple ones will ONLY add more computing power and NOT more VRAM. And I'm sure if you don't have enough power to run some simulations you can just borrow a K40 or something.

 

 

1. Thanks

2. May need more if I'm getting more than 1

3.It does look nice, will add it to the list.

4.Will do!

5. Yeah more GPU cards allow more models to be trained simultaneously, with different parameters (usually the tuning takes 10s of iterations, so multiple cards are very worthwhile so long as they don't bottleneck).

 

The i7-5820K only supports 28 PCIe lanes. See the caution in https://timdettmers.wordpress.com/2015/03/09/deep-learning-hardware-guide/. You will need/want at least an i7-5930K if you wish to communicate at 16 lane speeds with each of the gpu.

 

You may want to consider a 4 lane PCIe ssd drive like one of the Intel 750, Intel 750 Series 400GB PCI-E Solid State Drive and Intel 750 Series 1.2TB PCI-E Solid State Drive; or a Samsung M.2 like Samsung SM951 256GB M.2-2280 Solid State Drive and Samsung SM951 512GB M.2-2280 Solid State Drive. These drives offer significantly better performance than SATA III ssd.

 

Aside: RAID 0 may not provide better performance than a single larger ssd. It really depends on the pattern of i/o.

 

I am not sure that the Hyper 212 EVO is going to be able to do a decent job given how cpu bound your work may be. You might look at Noctua NH-U14S 55.0 CFM CPU Cooler or even an AIO, depending on how quickly you plan to move to a custom cooling loop.

 

If you are planning to add a second gpu, 860W is a bit tight. Especially if the gpu are going to be overclocked.

Huh I thought I checked the PCIe lanes, great spot - you're a hero!  That'll put the price up though

 

Yeah I was reading about the PCIe SSDs, they look interesting.  I'll look into it.

I think I'll get a preowned H100i for CPU cooling, before doing the custom loop.

[REX.exe]

If you want 3 or more Titans the 5820K will definitely bottleneck. Even the 5960X bottlenecks. Only a little bit for 3.

[brob]

...

Huh I thought I checked the PCIe lanes, great spot - you're a hero!  That'll put the price up though

 

Yeah I was reading about the PCIe SSDs, they look interesting.  I'll look into it.

I think I'll get a preowned H100i for CPU cooling, before doing the custom loop.

 

So I have reworked the build in post #7 based on a better understanding of the Hardware Guilde.

 

You will note a much less expensive Xeon cpu is selected. It supports 40 PCIe lanes but only has 4 hyperthreaded cores. Given the nature of cpu usage descibed in the Hardware Guide this should be more than sufficient to handle up to three gpu comfortably. The cpu has 10MB of cache. While that is only 2/3 of the 15MB in an i7-5930K, it should be more than sufficient to keep the cpu well fed. (Aside: algorithm designs significantly affect cache hit patterns. Handling the same data processing task, a good design can maximize cache hits, while a poor design may maximize misses.)

 

PCPartPicker part list / Price breakdown by merchant

CPU: Intel Xeon E5-1620 V3 3.5GHz Quad-Core Processor  (£231.66 @ Scan.co.uk)

CPU Cooler: Noctua NH-U14S 55.0 CFM CPU Cooler  (£48.12 @ CCL Computers)

Motherboard: Asus X99-A ATX LGA2011-3 Motherboard  (£191.28 @ Amazon UK)

Memory: Corsair Vengeance LPX 32GB (4 x 8GB) DDR4-2400 Memory  (£209.55 @ Amazon UK)

Storage: Samsung SM951 512GB M.2-2280 Solid State Drive  (£235.44 @ Aria PC)

Storage: Western Digital BLACK SERIES 2TB 3.5" 7200RPM Internal Hard Drive  (£95.74 @ Ebuyer)

Video Card: EVGA GeForce GTX Titan X 12GB Video Card  (£803.22 @ More Computers)

Case: NZXT H440 (White/Black) ATX Mid Tower Case  (£88.07 @ Aria PC)

Power Supply: XFX ProSeries 1050W 80+ Gold Certified Fully-Modular ATX Power Supply  (£120.54 @ More Computers)

Total: £2023.62

Generated by PCPartPicker 2015-07-12 23:36 BST+0100

[g4m3cr34tor]

Just on a quick side-note, if you want FULL 24/7 reliability in your motherboard, definitely take a look at the X99E-WS from Asus or even the Sabertooth X99. The WS board (super expensive though) will take everything you throw at it and them some. It has simply unbeatable quality for consumer/enthusiast grade boards before you go into the deep end with stuff like Supermicro. 

 

I'm not exactly sure about this, but maybe take a look into Nvidia Quadro GPUs? Don't quote me or ask for my assistance on this at all as my knowledge in them is extremely limited.

 

Another quick point on reliability, definietly go for air cooling as it offers much less points of failure and a good cooler like those from Noctua or BeQuiet will do you just fine, even if you slightly overclock the system.

[Mattias Edeslatt]

ASRock also has a good X99-WS board. quite a bit cheaper than the ASUS counterpart.

[RichL]

Hi all,

 

Sorry for being away from the topic - our lab has got some more funding so I can get the required hardware without building it for me, so I am now able to wait for the Pascal GPUs - I hope there will be something more affordable that still has the VRAM etc. that I am looking for.  Thanks to everyone for the advice, will post here when I'm ready to put together a build again.

 

Rich