Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Whaler_99

Whaler's GPU Folder Build Log - Folding3

Recommended Posts

Posted · Original PosterOP

Folding3 is just the hostname I use, for my HFM.NET. :)

 

Update July 2013 - to check out the build log on my second GPU folder click here. Also, updated pics of both units...

 

 

When you are this Big, they call you Mr.

 

OVERVIEW

 

So… the purpose of this build log will be to detail my adventures in consolidating my GPU based folding systems.

 

Right now I have 3 systems folding 6 GPU’s. One of the issues, the system are all older dual or quad core systems, which generate pretty much zero PPD versus the amount of power they draw and the amount of heat they dump when CPU folding.

 

So, the idea is to get a new system that can do the SMP folding and turn out a decent amount of PPD and hopefully bring the GPU’s into one box. Why you ask? And how do you fit that many GPU’s??

 

First off, power consumption – most of my systems all run single rail, 750 watt PSU’s – but I don’t think any one of them is drawing enough overall to take advantage of the efficiencies. So, by consolidating, I can make better use of a single, high quality PSU, maximize wattage usage and hopefully get into that sweet spot of 85% for the best efficiency.

 

Second, the GPU’s. If some of you have been watching the folding section, I detailed where you can actually use PCI-e x1 to x16 adapters to connect GPU’s to x1, x4 and x8 slots and be able to fold on them, here. Folding does not require a video card to be in an x16 slot and running at x1 doesn't affect the folding performance either. Note – this isn’t something you would do on your main system, would bugger SLI/XFire, etc. But, does work for folding…

 

So, onto main component selection…

 

CASE

 

I went with the HAF XB case. Couple of reasons, the design lends itself to a folding rig – can leave it open, things like hard drives and PSU don’t get into the way of the airflow for the mobo and GPU’s. Also, in looking at it, if I ever built a second one, I think I could stack em. J

The case itself is great quality and easy to work with, will update on this more as the build progresses. Also, was on sale – that helps!

 

Motherboard & CPU

 

First of CPU – if buying a new one, I would go with a 3770k, as this wouldn't break the bank, as compared to one of those extreme ones, and can overclock it. But, I ended up going with a 3770 non K edition. Why? I got an awesome deal on an almost new one from a buddy. Can’t beat that. J

 

As for the mobo – I was looking around for an inexpensive option that allows me to maximize my PCI-e slots. I ended up going with a Gigabyte Z77X-UD3H. Was, again, able to get this one sale. It has three PCI-e x16 slots and three PCI-e x1 slots. Also, is has an optional SATA power adapter on the mobo for helping to deliver more power to the PCI-e slots when multiple cards used. This WILL come in handy. Other than that, don’t need much more.

 

Update - Now using an MSI board that can support more cards!

 

GPU’s

 

My entire GPU selection to date has been based on NVidia cards, simply because they preform very consistently versus AMD and all their driver issues. Also, they do not hit the CPU near as much as AMD cards do. The new Core17 for AMD seems to have fixed some of this, but only time will tell. Another factor into the specific models has been simply based on what deal I could get on used cards.

The one issue I foresee is getting the indexing to work properly. Anyone familiar with multi GPU folding knows this – the FAH client can sometimes kack out when there are too many GPU present, so you need to ply with the indexing/slot assignments. This can be a pain to get working.

 

Update July 2013 - the main 4x GPU folder was recently updated with all AMD cards - a second GPU Folder, detailed later, is running the NVidia cards.

 

PSU

 

Going with a Corsair TX750v2 unit. First, because it was on a really good sale. Noticing a theme here when it comes to building folding systems? J Next – the Corsair TX750 is a Seasonic OEM model, so you know it is top quality and has a single 12v rail.

 

Other Components

 

Not much to say here. Basic hard drive to run Windows 7, some ram. Don’t need anything fancy as folding doesn't really care about that. J

 

Parts List

 

Case – Cooler Master HAF XB

Motherboard – MSI-Z77A-GD65

CPU – Intel 3770

Cooler – stock cooler

RAM – 4x1GB of generic 1600

Hard Drive – Seagate 7200RPM, 160GB SATA

Fans – 2x140mm front intake, 1x120mm rear exhaust

PSU – Corsair TX750 v2

GPU1 – AMD 7770

GPU2 – AMD 7750 - this is a custom single slot model

GPU3 – AMD 6850

GPU4 - AMD 6850

 

Day one... a new Dawn...

 

 

So… first thing first, tearing this case apart. J No, I am not modding it, painting, dressing up with LED’s or anything like that, it’s just for Folding. But, need to pull off the sides and top, pull out the SSD cage as it won’t be needed.

 

Overall, really good build quality on this case – impressed for the cost. LOTS of tie down spots. They even lay out some possible cable routing areas and tips in the instructions.

 

Pulling off the included front 120mm fans – replacing them with 140mm ones. Want to maximize that airflow CFM. Going to use a couple of Bitfenix Spectre Pro ones I have and put the same, but 120mm, in the rear.

 

Now here is one minor issue with Cooler Master – they do not provide any extra fan screws, for the rear fans. Everything else on the case is black, but depending on what fans you buy, you may end up with silver fan screws, which are the common. Be nice to see 12 extra fan screws, black.

 

Mounting the mobo was straight forward. I wasn't going to, but ended up pulling out the mobo tray, only because you have to, to get at two of the screws securing the SSD cage, which I wanted gone. Everything went back in nicely.

 

My one gripe with the mobo – the I/O shield. They have this foam on it. Basically when you slid the board in, everything gets kind of sealed up. Nice, but, made it so you had to put a lot of pressure on the board to push it in and hold it while you secure it down. I always worry about putting just that little bit too much pressure on the board.

 

Got the CD-ROM mounted with no issue. The hard drive tray – amazing tooless solution. The entire frame is hard plastic and it actually has a locking mechanism and can slide apart to get the drive into the tabs, click it back into place, done. Not like the flimsy ones, bending and prodding and such so get the drive screw tabs in. It’s so elegant and works so well…

 

Now, loaded in the PSU. Word of advice, plan your build, and mount the PSU first and run most of you cables leads where you will need them. Then install the mobo on the tray. I did the PSU last and it is somewhat difficult running the power supply leads. If, you had a modular PSU, forget it. Almost impossible to get your hand in there to plug extra ones in…

 

Ok, cards all loaded in. Time to power up this puppy and run through all the detection's and install drivers..

 

 

Day two... Umm Scotty, we have an issue...

 

Update - we are now down to FIVE GPU's in the rig. Due to the layout of the pcie-x1 slot beside the x16 slot and using dual slot coolers, my expander cannot actually go in there. It stops the x16 card from properly loading in. Hmm...

 

So... I get windows loaded, load up updates, get NVidia drives on and voila... three video cards. HUH? Ok... trouble shoot mode... I spend a few hours checking power leads.... re-seating cards.... checking expander cables... nope, everything looks fine. The computer will just not see the other two GPU's...

 

Also noticing another issue. On all my Windows folder's I run HWmonitor to track temps. With the the first 560 packed up against the second one it is getting REALLY hot. Not able to pull in much air... Geezzz... 98C??? Time to shut this down and figure things out, like WHERE are my other cards and how to get this card cool...

 

Detour....

 

Ok, time to vent and rage, although the fault is somewhat my own. RTFM!!! <sigh>  I crack open the manual and start looking at the electrical connections on the pcie lanes... Turns out, in gigabytes wisdom, that if you plug a card into the third pice-x16 slot, ALL the x1 slots are disabled. WHAT? Seems like someone took some shortcuts. 

 

Lesson learned - although not typically an issue, if you are going to take a mobo to the limits of what may be normal, maybe read up on it before hand. :)   I did check some manuals for some other boards that were a bit more expensive with three x16 and a few x1 and all of them had a something like this, though not as extreme. Typically if you use slot X then slot Y is disabled. So.. drawing board...

 

Day three... A new Dawn...

 

OK, so - going with two cards in x16 slots 1 and 2, and then an additional two cards in x1 slots. Using my one single slot card, I could have used it in the first x16 slot, and then had three x1 slots to use (allowing my to fold on 5 cards), but I ended up selling that card. And a GT 640 doesn't do that great... So for those of you keeping score at home...

 

I am now down to four GPU's in the rig. OK, think I got everything cabled right, power on cross the fingers... and YES - four GPU's detected. <wipes sweat of forehead>

 

So, going to move this into the "folding" area - aka my crawl space. I decided to put the sides back on. This will allow more focused airflow through the case. Although the case does use pop rivits, a lot of the places you may want to do come work on, they put in screws. In order to get the left panel on, I need to get the handle off, as if impacts on the card. Three screws later and off she is. I also put the top cover back on with a 200mm Cooler Master fan I had, also intaking air, down onto the cards..

 

GRIP ALERT - I tried to mount the front panel back on as well, hey, might as well keep some dust out... as you will see from the pics, it won't fit. Turns out, there is a little play in mounting the 140mm fans, and mine happen to be sitting at the "bottom". Just above the PCB that hold the power switch, LED's, etc, for the case. Turns out the front cover has a "ridge" inside that goes just above this PCB and... my 140 fans are now blocking it. Only fix, move the fans up a little. Caveat - screws are inside the case, meaning to get a screwdriver AT them - you need to rip the GPU's out... at this point, not happening. :)

 

Conclusion

 

System is folding away nicely, temps are ok. This is always the issue with Folding - trying to get max performance and control temps. FYI for everyone - I am NOT overclocking any of the GPU's or CPU. I also set the CPU to fold on all 8 cores as NVidia are not so reliant on a free core as AMD.

 

In the next section I will post up all the pics I took with some descriptions and such.

 

Temps are manageable - between 60 and 75 Celcius and generate between 80K and 90k PPD. Going to play with the CPU settings - see if freeing one core up has any difference. A couple of the cards don't seem to be getting quite as much PPD as they used to.

 

 

Hope ye all enjoyed....


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites
Posted · Original PosterOP

First off, some pics of the case, mobo and install...

 

post-765-0-33089800-1367463352_thumb.jpg

 

post-765-0-82950000-1367463354_thumb.jpg

 

post-765-0-15481100-1367463357_thumb.jpg

 

post-765-0-10947900-1367463360_thumb.jpg

 

post-765-0-64929000-1367463362_thumb.jpg

 

Install those BitFenix fans... Man, they are nice...

 

post-765-0-61683100-1367463365_thumb.jpg

 

post-765-0-90058400-1367463367_thumb.jpg

 

 

Top down shots...

 

post-765-0-69185600-1367463370_thumb.jpg

 

post-765-0-33430200-1367463373_thumb.jpg

 

Power supply mounted in - ya, it's a rats nest. I would highly recommend going modular on this case.. but hey, can't pass up a deal.

 

post-765-0-77456200-1367463375_thumb.jpg

 

post-765-0-09138500-1367463644_thumb.jpg

 

post-765-0-74629100-1367463646_thumb.jpg

 

More to come...


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites
Posted · Original PosterOP

Ok, some pics of the drive cage... Notice on the left hand side... pulls apart there. Really nice design....

 

post-765-0-05121000-1367463768_thumb.jpg

 

post-765-0-56202200-1367463770_thumb.jpg

 

post-765-0-86446200-1367463772_thumb.jpg

 

And just slips in nice....

 

post-765-0-44837300-1367463775_thumb.jpg

 

Now, some shots of the original layout with 5 GPU's.... Still shed a tear over that...

 

post-765-0-22603600-1367463778_thumb.jpg

 

post-765-0-92978300-1367463780_thumb.jpg

 

post-765-0-74699400-1367463783_thumb.jpg

 

post-765-0-41812400-1367463786_thumb.jpg

 

post-765-0-04036000-1367463789_thumb.jpg

 

And some final ones coming...

 


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites
Posted · Original PosterOP

Ok - Final layout after all the messing about...  here I have the 660 in the first x16 slot, a 560 in the second, a 560 there between the 660 and the CPU and the 460 over on the far left...

 

post-765-0-20638400-1367463937_thumb.jpg

 

post-765-0-81290600-1367463939_thumb.jpg

 

Pic of the side panel, with the "handle" remove so it can fit flush and allow the 460 to fit properly and have some nice airflow...

 

post-765-0-12981000-1367463942_thumb.jpg

 

post-765-0-63327000-1367463944_thumb.jpg

 

Here is a close up the PCB issue I mentioned, notice my fans sitting right on top? PITA

 

post-765-0-53667500-1367463946_thumb.jpg

 

And it's final resting home...

 

post-765-0-94329800-1367463948_thumb.jpg

 

 

Now... some pics of the FAH client.  First one was after running for a few minutes...

 

post-765-0-58619900-1367464140.jpg

 

And now after a day and some stabilization...

 

post-765-0-25244000-1367464141.jpg

 

Hope you all enjoyed!

 

Whaler

 


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites

have this case for my litecoin mining rig, have an 7950 (for now single) with the stock intake fans in the front direct on psu and it only reaches 70C.

 

This case is awesome in airflow and I was thinking about stacking them up to.


MSI GT70-One-899NL | i7-3630QM | Nvidia GTX675MX | 12GB 1600mhz ddr3 | 128gb mSata ssd | 1TB hdd | 17.3 Full HD non-glare | Killer E2200 

Link to post
Share on other sites

 

post-765-0-94329800-1367463948_thumb.jpg

 

Sweet project, but curiosity gets the best of me... what in the world is in the bottom left of this picture?


Workstation: 3930k @ 4.3GHz under an H100 - 4x8GB ram - infiniband HCA  - xonar essence stx - gtx 680 - sabretooth x79 - corsair C70 Server: i7 3770k (don't ask) - lsi-9260-4i used as an HBA - 6x3TB WD red (raidz2) - crucia m4's (60gb (ZIL, L2ARC), 120gb (OS)) - 4X8GB ram - infiniband HCA - define mini  Goodies: Røde podcaster w/ boom & shock mount - 3x1080p ips panels (NEC monitors for life) - k90 - g9x - sp2500's - HD598's - kvm switch

ZFS tutorial

Link to post
Share on other sites

Sweet project, but curiosity gets the best of me... what in the world is in the bottom left of this picture?

 

It looks allot like a KVM to me, so he can control all his crazy hardware lol


My Meme Proudly Featured on the Live-Stream: http://prodigydoo.com/ImageStorage/memes/linus-meme-6.jpg
Some Threads You Should Get involved in: Linus and Slick Memes     Youtubers on the LTT Forum      The List of Tech YouTubers (Contribute!)      Disable AdBlock for Linus (Guide)

Link to post
Share on other sites

nice man!


| Contact Information |
My Teamspeak : Austs1.gameservers.com:9334  |  Steam: Iamtictac456  |  My other aliases include Scruffy and Scruffy Biggems :)
 
 
 

 

 

Link to post
Share on other sites

I like it. very nice and well done.


i5 3570 | MSI GD-65 Gaming | OCZ Vertex 60gb ssd | WD Green 1TB HDD | NZXT Phantom | TP-Link Wifi card | H100 | 5850


“I snort instant coffee because it’s easier on my nose than cocaine"


 

Link to post
Share on other sites

I kinda figured, but I'm curious as to what brand/model.


Workstation: 3930k @ 4.3GHz under an H100 - 4x8GB ram - infiniband HCA  - xonar essence stx - gtx 680 - sabretooth x79 - corsair C70 Server: i7 3770k (don't ask) - lsi-9260-4i used as an HBA - 6x3TB WD red (raidz2) - crucia m4's (60gb (ZIL, L2ARC), 120gb (OS)) - 4X8GB ram - infiniband HCA - define mini  Goodies: Røde podcaster w/ boom & shock mount - 3x1080p ips panels (NEC monitors for life) - k90 - g9x - sp2500's - HD598's - kvm switch

ZFS tutorial

Link to post
Share on other sites

Jesus Whaler and here is me trying to compete with you, but when I get the GT 630 I'm on to you:P

 

Here is what I have, only getting 4,600 PPD and my folding rig is in 'show of your setup' so no point re posting that again  lol

 

Untitled-6_zps9e222007.png


DESKTOP - Motherboard - Gigabyte GA-Z77X-D3H Processor - Intel Core i5-2500K @ Stock 1.135v Cooling - Cooler Master Hyper TX3 RAM - Kingston Hyper-X Fury White 4x4GB DDR3-1866 Graphics Card - MSI GeForce GTX 780 Lightning PSU - Seasonic M12II EVO Edition 850w  HDD -  WD Caviar  Blue 500GB (Boot Drive)  /  WD Scorpio Black 750GB (Games Storage) / WD Green 2TB (Main Storage) Case - Cooler Master 335U Elite OS - Microsoft Windows 7 Ultimate

Link to post
Share on other sites

nice!


PC Builder, Engineer... BACON    Project Cobalt: http://linustechtips.com/main/topic/38058-project-cobalt-copper-piping-laser-etching-and-more/#entry489258

| NZXT Switch 810 | i5-3570k | gigabyte UD-5H | Corsair Vengeance 8gb ram | GTX 670 | 2x 60gb intel 330 series ssd's in raid 0 | 1tb seagate barracuda hdd | Corsair tx750m | XSPC razor GPU and CPU waterblocks | XSPC d5 vario pump | Thermochill Pa140.3 | phoyba 280mm radiator | Chromed Copper tubing |

Link to post
Share on other sites

great work dude, keep it up.


Fractal Design Define R4 | Intel i7-4770K | Corsair H80i | Corsair SP120 PWM | MSI Z87-GD65 GAMING | 8GB Corsair Vengeance Pro | MSI GTX770 GAMING | Samsung 840EVO 120GB | 2x WD Red 2TB | Corsair AX760 | Windows 8 Pro

Link to post
Share on other sites
Posted · Original PosterOP

Jesus Whaler and here is me trying to compete with you, but when I get the GT 630 I'm on to you:P

 

Here is what I have, only getting 4,600 PPD and my folding rig is in 'show of your setup' so no point re posting that again  lol

 

 

 

The main thing - just keep at it and slowly grow your gear. I buy a lot of stuff on kijiji and ebay...


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites
Posted · Original PosterOP

Thanks for all the well wishes guys - more parts for the 4P systems showed up - so if you haven't, check out the thread here...


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites

The main thing - just keep at it and slowly grow your gear. I buy a lot of stuff on kijiji and ebay...

 

What could a Core i3-2120 (slighly overclocked to 3.46GHz an running good:P) and a GT 630 kick out on PPD combined?


DESKTOP - Motherboard - Gigabyte GA-Z77X-D3H Processor - Intel Core i5-2500K @ Stock 1.135v Cooling - Cooler Master Hyper TX3 RAM - Kingston Hyper-X Fury White 4x4GB DDR3-1866 Graphics Card - MSI GeForce GTX 780 Lightning PSU - Seasonic M12II EVO Edition 850w  HDD -  WD Caviar  Blue 500GB (Boot Drive)  /  WD Scorpio Black 750GB (Games Storage) / WD Green 2TB (Main Storage) Case - Cooler Master 335U Elite OS - Microsoft Windows 7 Ultimate

Link to post
Share on other sites
Posted · Original PosterOP

Hard to say... every system will give you somewhat different numbers based on projects. Core i3, guessing maybe 2-3k PPD - the GT 630, again 3-4k?

Depending on project you may see a total sometimes of around 5k and could hit as high as maybe 9k...


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites
Posted · Original PosterOP

Yeeehaw!  :) All four GPU's with the new Beta Core17 Project 7663 - the 660 though isn't seeing much of an improvement, the 560's some and the 460 a HUGE boost

 

post-765-0-07647900-1367928670.jpg

post-765-0-07647900-1367928670.jpg


Forum Links - Community Standards, Privacy Policy, FAQ, Features Suggestions, Bug and Issues.

Folding/Boinc Info - Check out the Folding and Boinc Section, read the Folding Install thread and the Folding FAQ. Info on Boinc is here. Don't forget to join team 223518. Check out other users Folding Rigs for ideas. Don't forget to follow the @LTTCompute for updates and other random posts about the various teams.

Follow me on Twitter for updates @Whaler_99

 

 

 

 

Link to post
Share on other sites

That is pretty sweet, also looking at one of the projects credits at 16,000 is that a big project?


DESKTOP - Motherboard - Gigabyte GA-Z77X-D3H Processor - Intel Core i5-2500K @ Stock 1.135v Cooling - Cooler Master Hyper TX3 RAM - Kingston Hyper-X Fury White 4x4GB DDR3-1866 Graphics Card - MSI GeForce GTX 780 Lightning PSU - Seasonic M12II EVO Edition 850w  HDD -  WD Caviar  Blue 500GB (Boot Drive)  /  WD Scorpio Black 750GB (Games Storage) / WD Green 2TB (Main Storage) Case - Cooler Master 335U Elite OS - Microsoft Windows 7 Ultimate

Link to post
Share on other sites

Very well written build log. I laughed at the way you worded your GPU mishaps :)


export PS1='\[\033[1;30m\]┌╼ \[\033[1;32m\]\u@\h\[\033[1;30m\] ╾╼ \[\033[0;34m\]\w\[\033[0;36m\]\n\[\033[1;30m\]└╼ \[\033[1;37m\]'


"All your threads are belong to /dev/null"


| 80's Terminal Keyboard Conversion | $5 Graphics Card Silence Mod Tutorial | 485KH/s R9 270X | The Smallest Ethernet Cable | Ass Pennies | My Screenfetch |

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×