My 24 & 16 Core 4P Folding Rigs (Updated!) (large pics)

[Imaginaerum]

This information was current as of 12/11/2013.

 

Currently neither rigs are folding and I've sold both of them. The BigAdv folding requirements were raised and I now neither meet the specifications. Because of the costs to upgrade I have no intention of changing my rigs. Updated 1/26/2014.

 

This is a thread showing off my two 4P Folding Rigs! The first rig (top shelf) makes ~120k PPD at minimum, the second rig (bottom shelf) makes ~90k minimum, combined for ~210k PPD with BigAdv folding. This only happens when Project 8101 comes along. Rig 1 cost me $250 out of pocket, Rig 2 cost me $200. Combined I pay roughly $60 in energy a month, and that's just for these two rigs... As far as noise levels go, it's about as loud as one of those giant 20" fans you find at the store on the lowest setting.

 
Since they're both complete, I won't have a huge amount of pictures of my progress. I didn't feel the need to detail each and every step, but I'll keep a text-log about my experiences for those interested.
 
1st 4P Rig Specs:

• Motherboard: Supermicro H8QME-2+
• Processors: 4 AMD Opteron 8425 HE @ 2.1GHz (24-cores total)
• PSU: Seasonic 760w Platinum
• RAM: 16GB DDR2
• HDD: Seagate 500GB (overkill for folding, but it's multi-use. Mounted in my unused Fractal Define R4 cage)
• Heatsinks: 4 copper heatsinks, included with bundle
• Fans: 4 80mm Antec with 3 speed options (I usually have them on medium for noise reasons), 200mm Cooler Master fan & 120mm Bitfenix Spectre fan
• Case: Custom-built, but mostly went naked. I put the shelf together, named it the Plank 2.0

2nd 4P Rig Specs:

• Motherboard: Supermicro H8QM3-2+
• Processors: 4 AMD 8386 SE processors @ 2.8GHz (16-cores total)
• PSU: Seasonic 760w Platinum
• RAM: 24GB DDR2
• HDD: 32GB PNY USB Drive
• Heatsinks: 4 Generic Copper-Aluminum heatsinks
• Fans: 4 80mm Antec 3-speed (kept on high because these heatsinks are picky), 200mm Cooler Master & 120mm Bitfenix Spectre
• Case: Custom-built, but mostly went naked. I put the shelf together, named it the Plank 2.0

For both rigs, I use an IBM Model M Keyboard, a $10 Walmart brand mouse, a Dell monitor found at a garage sale, Arctic Silver 5 thermal paste, and a Mionix Deskpad. I thought the deskpad was ridiculous when I first saw it, but after the plywood on my shelf made my skin sting, I freaking love it. The motherboards are held off the surface by a bunch of (wait for it) Rubber Erasers. I had fancy rubber standoffs but lost them sometime during the coffee-table-to-shelf transfer, and the erasers are working perfectly.
 
1: Initial testing for Rig 1. The 200mm fans were all I had at the time to cool the heatsinks, and this was before I built the shelf.
 

 
2: Both rigs fully assembled, also initial testing for Rig 2. I put the wooden shelf together with four 2x4 boards, metal rails, and two plywood sheets for each shelf. Measures 24"x24"x46". The purpose of this was to get the coffee table back, allow me to put two rigs in a more compact space, and have room to add another shelf in the unlikely event I want to add a third rig. I may, or may not, organize the cables one day.
 

 
3: The rest are close ups of each rig and different sides of the shelf. I would paint the Plank 2.0, but I don't currently have a garage or any well-enough-ventilated workspace. Both systems are on in the pictures, even though the fans aren't moving because of my camera.
 

 
Rig 1
 

 
Rig 2

 

 

 
 4: Wattage draw of Rig 1, Rig 2 is only a few tens of watts difference.
 

 
5: My friend suggested this, it's somewhat meaningless but fun to look at. All 24 cores at 100% in Rig 1. Rig 2 takes way too long to load to bother with this, as I'm running it on a USB-drive at USB 1.1 speeds.
 

 
 
 
 That's it for pictures - now for the story of Rig 1.

What caused me to get a 4P rig? I was getting about 6,000 PPD on my personal desktop, which is fine if that's all you can do, but I wanted better; not only for points, but because I wanted to help the cause. I considered buying more graphics cards to run 24/7, but the energy cost was going to be crazy, and I'd have to upgrade my PSU (which I ended up doing anyway). That's when I read about 4P folding rigs and saw how much more PPD you could get for a fraction of the power draw. Heat output doesn't bother me too much, so I was sold.
 
The first thing I bought was the PSU on Newegg for over half off after the rebate and a sale. Second, I bought a Supermicro H8QME-2+ for $120 on eBay. The seller wanted $150 originally, but I offered $100 and he accepted. $20 extra was for shipping. That seller was the only person I could find selling Supermicro motherboards that I could afford...
 
The bundle came with four AMD Opteron dual cores, 16GB DDR2 RAM, and 4 copper heatsinks. Seeing as DDR2 RAM is $500 on Newegg, I felt like I got a pretty good deal.
 
After getting the motherboard and powering it on, loading the BIOS and deciding it wasn't going to die out on me, I looked for the processors I actually wanted: four AMD Opteron 8425 HE 6-core chips. While browsing eBay, I found they were generally being sold around $100 each. On the day I had the money, I found a guy selling 4 of his for under $350, and of course, bought them. It's possible I could have done better if I bought the processors one at a time.
 
At this point, I had everything I needed, and spent a little under $550. I had some Paypal money left over from a computer I recently sold, so I only had to pay about $250 out of pocket. As for my keyboard/mouse/monitor/hard drive/thumb drive/any cables I needed, I already had them all, which is always useful as that can easily drive up the cost.
 
Eventually the processors arrived - and the mailman didn't ring the doorbell. I end up going to the post office, picking up the processors, going back home, and putting them in the motherboard. Upon powering it on!... nothing happens. I can't get any video, so straight to Google (useless), Whaler (helpful), and [H]ardForum (savior) I go.
 
I actually had the right idea for solving my problem: I needed a BIOS update. I made a Win-98 bootable DOS thumb-drive, loaded the BIOS flash files onto the motherboard, and get an invalid ROM error. [H]ard told me to add /X to a line in the .bat file, which forced the update, flashed the BIOS, and recognized all 24 of my cores. I was ready to get started!
 
...almost. I had previously tried Ubuntu, but it seemed to have a personal vendetta against my internet. Every 5 minutes it would just murder my router, requiring me to restart it. After hours and hours of Googling and talking to one of my networking friends, we discover the cause might be the default DNS servers. I changed them to Google's & OpenDNS servers, and problem solved!
 
From that point on, following this guide at [H]ard was extremely easy to get F@up and running, and then I setup HFM.net to get a better visual representation on what is actually happening. The V6 client isn't very informative.
 
Afterwards I ran into a few more issues, like missing my first BigAdv deadline thanks to local power outages, then missing a second deadline because I screwed up and made my computer run two F@clients at start up. I have, thankfully, figured out all of the problems (for now) and should be smooth folding from now on.

 
The story of Rig 2.

About two weeks after my first rig is up and running, my dad's work had an auction for their old server equipment. By what I consider to be sheer chance, it seemed like a perfect setup for another folding rig, so I went on over to the auction and won a single one (that's all an individual was allowed) for $200! It came with mostly everything I needed, except I didn't have a whole lot of space.
 
That's when I made the Plank 2.0 for around $50 with materials from Lowe's. I would love to paint and finish it, but without a garage it would smell too badly for too long. I can't complete it outside either since it's too windy, and because Fall makes all the leaves fall off, some of them might stick to the wood while it's drying. In other news, plywood is terrible for your skin and sucks the oil right out of it. After using it for 10 minutes my arms were stinging like crazy, so I bought a Mionix 9000 Mouse Pad and can honestly say that was the best decision of this entire project.
 
The rig isn't without its share of problems. I'm convinced the motherboard is cursed. It only turns on when it wants to, it doesn't shut down when Ubuntu tells it to, and if you so much as brush a fan it completely turns off. You then have to recite an incantation, wait until midnight, and hope all the planets align as you flip the switch on the power supply. If all goes well, it works again.
 
On top of that the USB 2.0 ports, for whatever reason, run at USB 1.1 speed, which has a whopping 1.5MB/s transfer rate. This makes doing anything extremely slow (it took 8 hours to install Ubuntu). This seems to be fine for folding but I cannot use it for anything else. I don't know how long my 32GB USB drive will survive this, maybe forever but maybe only a week. If it does die I'll probably restart on a real hard drive to save the headache of USB 1.1.
 
Once I got everything running, the stupid processors started overheating. CPU1 was getting over 72C, which is when the motherboard starts to scream at you. After deciding the heatsinks weren't good enough, I left it alone for a few days but decided to give it another try before buying replacements. While tapping around in the BIOS I found some fan profiles which happened to be set to Silent. I change it to Full and now temperatures are fine and I'm folding away.

 
 
Since I'm already here... My personal rig I used previously for folding.
 
http://i.imgur.com/GHP325T.jpg
 
Personal Rig Specs
Motherboard: ASUS P6X58D-E
Processor: Intel i7-930 2.8GHz (too hot to overclock, idles at 50C)
RAM: 10GB DDR3 triple-channel
Graphics: Nvidia GeForce GTX 660 Ti
HDDs: WD 7200RPM 500GB (games) and Seagate 7200RPM 500GB (for OS/temporary storage)
Heatsink: Hyper 212+ Evo
Case: Fractal Define R4
Fans: Two 120mm Bitfenix Spectres (front), two 140mm Fractal fans (exhausts), one generic 120mm (heatsink)
PSU: OCZ 550w Gold Fully Modular (the Seasonic PSU in the picture is now being used for the second folding rig, but I can't be bothered to perfect my cable management again so I don't feel like updating that image)

 

As for periphals, I use a Ducky Shine Cherry Black keyboard, Mionix NAOS 3200 mouse, Bose 2.1 speakers, Panasonic RP-HTF600s, ASUS 1080p monitor, Mionix mousepad, Blue Snowball mic, and various other hardware lying at my desk at any given time.
 
Final thanks go to Whaler, [H]ard, Google (for some things), my friend who helped with some Ubuntu knowledge, and Linus & Slick and whoever else for Build of the Week (Nov 1, 2013). Also, thanks for reading!

[Maskot]

Nice desk case.

[J4M35]

epic rig! i would love to build a 4p folder, but i don't have the cash-money to do that. also, how many ppd are you getting on that rig? enjoy them!

[jmrios]

Nice rig man!!! My ultimate dram is to build a 4P folding rig. But I'm in the gpu folding wagon, and since I can't buy for ebay for limitations regarding my location, I think that I'll always be a gpu folder. It's nice, but since I'm a huge hardware fan, the 4P would be a new playground for me.. 

Please updade this thread about how much PPD is producing.

I expect to see you in the charts soon 

[D3MON]

Beautiful! seems like you got a steal on the chips, ram and board (why are they called supermicro though???)
would love to have something like this...or 2 piled 2 high in a 4U rack mount or something, I nearly got wood at the loaded cores picture....

edit: would be cool if you could pay for your nvidia grid to give you some PPD under your name...hmm

[Imaginaerum]

Beautiful! seems like you got a steal on the chips, ram and board (why are they called supermicro though???)

would love to have something like this...or 2 piled 2 high in a 4U rack mount or something, I nearly got wood at the loaded cores picture....

Supermicro is just the motherboard brand. Chips are AMD, no clue what the RAM is.

 

As for PPD, around ~120k. I put that near the end of the post, glad to see few people read so far... I'll put it at the beginning now

[Commissar]

lol, if(really when) I get another 7970, i'll be at ~180kppd....

[BenDreadful]

lol, if(really when) I get another 7970, i'll be at ~180kppd....

the main point of the 4p folders is that it uses less power and the temp are lower. comparing this to my 7970 it's folding on the same amount of power and it's about 30 degrees cooler. plus there's the life expectancy, folding 24/7 the 4p wins every time

[qwertywarrior]

hmm arnt there 4p board with ddr3 support ?

[legopc]

fancy but isnt it better to put it in a case?? 

[Whaler_99]

fancy but isnt it better to put it in a case?? 

 

Nope - most of these big guys are left "naked"

[Whaler_99]

Awesome post - thanks very much for the pics and details!! Couple of observations...

 

1 - raise the mother board up, off the desk - using standoffs or something. You want to make sure that air can get under the board, as it can get VERY hot under there.

 

2 - On those two heatsinks that are on the chipsets, get a fan on them. They run stupid hot as well. Some boards come with them, some don't. Just plop a fan on top to get some air moving through them. Will help keep things more stable.

 

3 - any way to get a fan blowing at the front of the case? If you look at mine, I have two... getting some air flow across the board helps keep the board cooler and move some of that hot air from the cpu's away as well.

 

4 - LOVE the keyboard. Adds that extra something to the build!

 

Check out this post over at [H]ard and the custom case solution.

http://hardforum.com/showthread.php?t=1783282

 

Will add you in the the new 4P post - Welcome and congrats on the build.

[Commissar]

the main point of the 4p folders is that it uses less power and the temp are lower. comparing this to my 7970 it's folding on the same amount of power and it's about 30 degrees cooler. plus there's the life expectancy, folding 24/7 the 4p wins every time

soz... the life span of my 7970 is less if i'm folding on it all the time?

[Xsilent(X)]

looks puny but awesome

[BenDreadful]

soz... the life span of my 7970 is less if i'm folding on it all the time?

 

yup, it's all in the temps. the higher the temps, the shorter the lifespan. if you can get it to fold and have like 30 degrees under 100% load then it wouldn't be a problem.

 

I thought about trying to do this since there are waterchillers which are basically meant for fish tanks, but some people use them for cooling the water in their custom loop. you can see them here http://specialtech.co.uk/spshop/customer/Water-Chillers-cid-1904.html

 

it's basically a mini fridge in which you can stick tubes to cool your loop.

 

but yeah, heat will lower the lifespan of your components. went a bit off topic lol

[piplupgao]

Supermicro is just the motherboard brand. Chips are AMD, no clue what the RAM is.

 

As for PPD, around ~120k. I put that near the end of the post, glad to see few people read so far... I'll put it at the beginning now

How much was this?

[ajoy39]

that board has no expansion slots an it looks like it has SODIMM sockets. wut.

[Raboo]

Its not a looker but a performer.

[Imaginaerum]

Awesome post - thanks very much for the pics and details!! Couple of observations...

 

1 - raise the mother board up, off the desk - using standoffs or something. You want to make sure that air can get under the board, as it can get VERY hot under there.

 

2 - On those two heatsinks that are on the chipsets, get a fan on them. They run stupid hot as well. Some boards come with them, some don't. Just plop a fan on top to get some air moving through them. Will help keep things more stable.

 

3 - any way to get a fan blowing at the front of the case? If you look at mine, I have two... getting some air flow across the board helps keep the board cooler and move some of that hot air from the cpu's away as well.

 

4 - LOVE the keyboard. Adds that extra something to the build!

 

Check out this post over at [H]ard and the custom case solution.

http://hardforum.com/showthread.php?t=1783282

 

Will add you in the the new 4P post - Welcome and congrats on the build.

I have it standing off now with some makeshift rubber standoffs, and I have those 200mm fans blowing across it too. My "finished" build was "almost finished lol. I have the 120mm fan blowing directly on the chipset heatsinks, I should probably replace the picture.

 

How much was this?

Also mentioned this... $250 out of pocket, $550 total thereabouts.

[Spannerhands]

Totally love it..... good work...

[Imaginaerum]

that board has no expansion slots an it looks like it has SODIMM sockets. wut.

It has 1 16x slot and a SODIMM. I ended up taking out the SODIMM card that came with it, solved some weird startup issues.

[Commissar]

yup, it's all in the temps. the higher the temps, the shorter the lifespan. if you can get it to fold and have like 30 degrees under 100% load then it wouldn't be a problem.

 

I thought about trying to do this since there are waterchillers which are basically meant for fish tanks, but some people use them for cooling the water in their custom loop. you can see them here http://specialtech.co.uk/spshop/customer/Water-Chillers-cid-1904.html

 

it's basically a mini fridge in which you can stick tubes to cool your loop.

 

but yeah, heat will lower the lifespan of your components. went a bit off topic lol

even just 60c? i may just underclock it while folding then.... or not gpu fold at all.....

[Kayn]

even just 60c? i may just underclock it while folding then.... or not gpu fold at all.....

60c is fine even 70c is fine. GPUs can stand high temps. It will degrade a little faster but it won't be anuthong significant.

[Xsilent(X)]

even just 60c? i may just underclock it while folding then.... or not gpu fold at all.....

 

never OC the CPU it causes allot of errors in the logs. only OC the GPU. but ether not to mutch.

[Kayn]

never OC the CPU it causes allot of errors in the logs. only OC the GPU. but ether not to mutch.

Switch that around. F@H is less picky with CPU as long as its absolutely stable. GPU overclocks are very picky