Whaler's 4P Folders Build Log (32 Cores and 24 Cores)

[Whaler_99]

Folding1 is just the hostname I use, for my HFM.NET. 

 

Update July 2013 - I have now also finished the 24 core Folder - follow this to go to the post - http://linustechtips.com/main/topic/15102-whalers-primary-4p-folder-build-log-32-cores/?p=168806

 

Update March 2014 - Due to the changes in the bigadv project (core requirements increase this spring and announced discontinuation of project in December) I no longer have either unit. But, I keep these up and can answer any questions you might have in regards to these or your specific builds.

 

 

When you are this Big, they call you Mr.

(haven’t I used ^^^^ before? Nahhh….)

 

OVERVIEW

 

So… the purpose of this build log will be to detail my adventures in building out my first 4P folder.

 

So, up to now, 4P systems have been the way to Fold for major points (with the new Beta core for GPU’s this may change). So, some issues that come to mind are 1) how to do it and 2) it can cost a lot of money. So, start the reading, I spent many a night reading various threads and forums and doing research.

 

I decided not to go whole hog to start but to get a decent system to start. The other issue, costs. I can hunt around and get some deals, but the board and CPU’s and such are pretty exclusive. So, I hunted around and found some Folding forums that sell used gear to other folders. Bonus.

 

So, got all the gear ordered and here and ready to start building. I will detail various components below. For the record though, here is some of the links I liked and/or referenced (or will be) a lot.

 

Computer tech station:

http://richchomiczewski.wordpress.com/

 

Design and Build Guides:

http://www.evga.com/forums/tm.aspx?m=1638127&mpage=1&print=true

http://hardforum.com/showthread.php?t=1652906

 

Home Rack Garage Build:

http://www.evga.com/forums/tm.aspx?high=&m=1678549&mpage=1%20-%201678549

 

Ubuntu stuff:

http://hardforum.com/showthread.php?t=1601608

http://rog.asus.com/forum/showthread.php?4298-Setting-up-Linux-bigadv-folding

 

Musky’s Hyper 212 mounting method:

http://hardforum.com/showthread.php?t=1662099

 

 

CASE

 

Original I was not going to bother, Fold naked and just have everything laying there. Naked means no case, everything open and exposed. Because of the size of the board and the heat sinks, you typically need expensive, custom case, like what SuperMicro make.

 

BUT – on browsing a forum post I came across custom case builder Spotswood. His cases are simply amazing. And he makes an open frame case for folding systems that allows you to tie down the PSU, hard drive, etc, leave it all open. Looks so clean and does the job. And… you can stack multiple ones. Awesome. See below for pics.

 

Motherboard & CPU

 

So… I managed to pick up via one of the Folding Forum sites I discovered, 4 x AMD Opteron 6136 chips. These are 8 core units, and can be overclocked. The motherboard I ended up getting was a SuperMicro H8QGL-IF-O. Was able to get an ok deal on this on Ebay

 

PSU

 

So, the one place you don’t want to scrimp on is the PSU – so I went and picked up a Corsair AX850 unit. This puppy gets a 9/10 rating over at jonnyguru.com. I think I now had 5 AX series PSU’s between various systems.

 

Heatsinks

 

Need to pay some attention here. The stock ones I got are the typical low profile, with SUPER noisy fans (think Delta’s). So, I picked up four Cooler Master Hyper 212 units. The great thing, direct from the CM store, you can buy them refurb. Now, the socket is G34, not exactly compatible. But, musky from [H]ard forum has a detailed mod to adapt the Hyper 212 to work on a G34 board. Better cooling and a lot quieter. J

 

Memory

 

Again, paying some special attention here. I got some GSkill ripjaws 2GB sticks. I needed 4 sticks per CPU, so had to find 16 of them and want the CL7, XMP versions. You need these ones to support the overclock on the CPU. Finally managed to find 16 of them through newegg. Funny enough, considering, this was probably the most expensive part.

 

Other Components

 

Not much to say here. Basic hard drive to run Ubuntu.

 

Parts List

 

Case – Spotswood 4p/MEV/SWTX Open Frame tray

Motherboard – SuperMicro H8QGL-IF-O.

CPU – 4 x Opteron 6136 (8 core)

Cooler – Hyper 212 Plus + musky mod

RAM – 16x2GB GSkill

Hard Drive – Seagate 7200RPM, 80GB SATA

Fans – CoolerMaster 120mm x 7, Noctua NF-P12 x 2

PSU – Corsair AX850

 

Week one...

 

My biggest issue, waiting for all the bloody parts to arrive from various sources.

 

Got the cooler and CPU’s. The ordered the case, the mobo, memory, etc…

 

Now, it is finally all here. I started with case assembly. I have to say, the workmanship by Spotswood on this A+. The case just looks beautiful. I won’t lie, took a bit of work putting together, got the old level out for a couple parts, various adjustments. But nothing some patience couldn’t solve. Once done, magnificent. I also assembled the second one I got for the other 4P unit I am putting together and stacked them to get a look, perfect fit. And such a clean way of doing this, especially if you have 5 or 6 systems… For the cost his systems destroy anything you can get from say Highpoint.

 

So case is assembled. Check. Mount in the PSU and hard drive. Check…

 

Now, onto my mods to get the Hyper 212’s I ordered to fit on the board. I basically follow musky’s guide to the letter and everything works. Some minor notes – have a dremel or other power tool for cutting the brackets. I tried one with my steel hand saw and forget that. J Now, when you get to the point when you are swinging the x brace and brackets around (after slipping them through the cooler base) to attach to the other end of the x brace, he states that although “snug” he had no issue. So, maybe my brackets were just a bit wider, don’t know, but I couldn’t attach the other end without crimping the copper heat pipes. And I DIDN’T want to do that. So, pulled the x-brace back out and slight bent the inside edge of the bracket inward, just a wee bit. I only had to do it on one end. (see attached pic) This then gave me just enough clearance to get everything screwed in nice and tight. And yes, the brackets are up against the copper heat pipes, but that isn’t an issue.

 

Now, next step is mobo mods. To get the 6-32 thread through the mounting post in the board he gives a couple of options. I honestly couldn’t figure out how cutting the head off would allow this to work. The 6-32 is still a bit too wide to fit. So, I completed the drill out method he talk about. How I did this…. Got a strong cardboard box to support the motherboard. Got the drill with the 5/32 bit and drilled from the top down. Main thing to remember here, don’t force the drill, let the bit do the work. If you force it, you could warp/bend the board. Also, by using a cardboard box, once through, you just go through the cardboard. J Once through, back out and onto next. Remember, when done, pay special attention to getting all the metal filings off the board. Surprisingly, took quite a few shakes (being careful) and blasts of compressed air to get all the filings out.

 

 

And this is where I take a break kids... going to be travelling for work for a few days and mother's day this past weekend tied me up. So, hopefulle make some progress this weekend...

[Whaler_99]

Some of these pix have been in other threads and tweeted - this will be the consolidated bunch with comments.

 

Hyper 212 and CPU's arrive.

 

 

Some CPU pics.

 

 

The SuperMicro board arrives....

 

 

All the "gear" for the musky mod on the Hyper 212's

 

 

Memory and moded x-braces for the Hyper 212's

 

 

Unpacking my case from Spotswood...

 

 

Lots of case assembly ones :)

 

 

Couple pics of the two cases I ordered, stacked...

 

 

Hyper 212 mod - notice, the inside edge slightly bent inward...

 

 

Mobo - ready for drilling (REALLY?!?! on a $700 board? Yikes...)

 

 

And the nice drill holes in the cardboard...

 

 

Close of of the CPU...

 

 

The board with 6-32 1.5" bolts through, ready for Hyper 212 mounting...

 

[Whaler_99]

Week two…

 

Well, back to work. J

 

So, following musky’s guide, I get ready to mount the heat sinks onto the chips. I put a bit more thermal paste then I normally would on a CPU as these are pretty big. I ran it along a line. Then slowly settle the HS down onto the chip. Getting the washers, lock washers and nuts on were a bit of a pain, very space constrained. Had to use needle nose pliers to drop the washers on and then with my fat fingers, threading the nuts on was a pain, but a lot of patience saw it done. Hardest thing was just not moving the HS to much and smearing the thermal paste everywhere…

 

Tightening down the nuts is a bit of a guessing game – how tight to make it. I made then fairly tight, to the point where the X brace was starting to bend just a tad. I figure that be good enough. If my temps suck, I can always tighten them down a bit more.

 

Now, to describe, in looking at the back of the board, or as per the manual, I installed the heat sink on CPU1 slot, then CPU3, then CPU2 and then CPU4. Found that easiest for getting my wrench in. When installing CPU2 HS, which is very close to the CPU1, I put the “push” fan on, because it looked like if I tried after both HS mounted, there would be no way to get it in there.

After installing the HS’s, but before adding the fans, I install all the ram, as some slots are going to be covered by fans.

 

With all the HS mounted and memory in, I proceed to install fans. I did push/pull on all of them. Probably not required, maybe only gaining a degree or two of cooling, but I have the extra fans and tons of fan connectors on the board, so why not. J

 

Putting A and B together…

 

Ok, board is ready, case is ready. Time to marry the two…  Weird did I leave that priest?  Oh, sorry, not that type.

 

I screwed in all the motherboard standoffs to the frame and then dropped the board in. Really, have I mentioned how nice it is dealing with an open frame solution? Maybe not in this paragraph yet eh? J

Now, start screw down the mobo and I notice a problem. Some of the holes in the mobo are not lining up properly with the standoffs. <sigh> So, I email Spotswood and within a few minutes he has an answer back to me – loosen the standoffs on the frame to get some play. Doh. Yep, that was it, with everything loose, was able to get everything threaded up properly and then tightened down nicely.

 

Run my power cables. Note, on these boards there is actually 3 x 8 pin EPS conectors, but you only need to connect the first and second one. Which also means you have to make sure your PSU supports two of them. Don’t use converters for this type of thing please? It will only lead to bad power feed. Get a proper power supply.

 

Drive cables and temp CD-Rom hooked up. All looks nice. Also mounted a couple of fans on the front of the frame. Help blow some cool air underneath the board, which does get very hot, and across mobo components.

Power up – YEA! She lives!

 

Fold, Fold and away we go….

 

So, following another one of musky’s guides, I set about installing Ubuntu onto the system. Honestly, following his guide couldn't be easier. Within an hour or so, I have it up and running and Folding away.

 

Will add some more pics on this. Currently Estimated PPD of the system is 340K. :D

 

Sweetness! Temps on all the components on the board are between 36 and 40 C. And I have to say, those Hyper 212 are an awesome investment. Keeping things cool, but more than that, the whole solution is quiet!

[Whaler_99]

Pic of partial install. See how when I installed HS for CPU1 then 2 I did it with the fan installed. Pretty tight...

 

 

All the HS's and fans installed, ram in...

 

 

Close up of ram. Notice the clearance of the ram and the fan... or lack thereof. :)

 

 

Here we see the assembly of the HS to the board...

 

 

Mobo installed in the fame...

 

 

Additional front fans installed (BTW - it's not supposed to look pretty - just crank out PPD)

 

 

 

IT'S ALIVE!!!

 

 

Screen Shot of HFM - Check FOLDING1 - the 32 Core system... yes, 341,282 PPD :D

 

 

BTW - a lot read a zero because HFM cannot detect PPD yet on Beta core projects.

 

[Whaler_99]

My 24 Core system, named.... Folding2 :)

 

OVERVIEW

 

So… the purpose of this second section of the build log will be to detail my adventures in building out my second 4P folder.

 

Please read the initial post for a lot of the details and pictures. Most of what is there is relevant for this Folder as well.

 

This time to save on costs over the original 4P system, I decided to go with AMD 8000 series CPU’s. You can typically still get say a 24 Core, 4P AMD 8000 series solution for $400-$500 plus incidentals.

Now, these systems won’t put out the big numbers of a 6000 series box, but for the cost, they are effective.

 

This time I managed to find a “turnkey” solution, basically someone selling pretty much everything you need as a total package to get up and folding. I got the motherboard, CPU’s, heat sinks, memory and even a drive with Ubuntu 12.04 installed on it. The only things I provided where the power supply and the case (which if folding naked, is optional)

 

CASE

 

Original I was not going to bother, Fold naked and just have everything laying there. Naked means no case, everything open and exposed. Because of the size of the board and the heat sinks, you typically need expensive, custom case, like what SuperMicro make.

 

BUT – on browsing a forum post I came across custom case builder @Spotswood. His cases are simply amazing. And he makes an open frame case for folding systems that allows you to tie down the PSU, hard drive, etc, leave it all open. Looks so clean and does the job. And… you can stack multiple ones. Awesome. See the original post for some pics of the case itself and links to his site. Some pics of the entire solution are below

 

Motherboard, CPU and RAM

 

Here are the specs on the gear I got…

Motherboard - SuperMicro H8QME-2 http://www.supermicro.com/Aplus/moth...55/H8QME-2.cfm

CPU - 4x Opteron 8425's Hexacore at 2.1ghz (80w cpus) - 24 threads http://www.cpu-world.com/CPUs/K10/AM...25PDS6DGN.html

Memory - 8x2GB (2 sticks per CPU) DDR2 ECC Ram

 

PSU

 

So, the one place you don’t want to scrimp on is the PSU – so I went and picked up a NZXT Hale90 850 Watt, module PSU. Like the Corsair on the other system, this is an excellent PSU. Single 12v rail, Gold rated efficiency, module and got it at a great price.

 

Parts List

 

Case – Spotswood 4p/MEV/SWTX Open Frame tray

Motherboard – SuperMicro H8QME-

CPU – 4 x Opteron 8425 (6 core)

Cooler – basic 2u aluminum heat sinks – I just screwed down some 80mm fans I had to help cool

RAM – 8x2GB

Fans – generic 80mm heat sink fans x 4, XSPC fans from a rad I had (XIRULIAN I think) x 2

PSU – NZXT Hale90-850-M

 

Get ‘er done…

 

So unlike the 32 core system where I was putting a lot of it together, figuring things out as I went, with experience come proficiency.

Also helped, I bought a turnkey system, so once in the case, plugged into power, booted it up and away I went. J

 

In the post below will be a bunch of pics and another screen shot of HFM – she’s doing about 160K PPD. :)

[Whaler_99]

And here are some updated pics!

 

Base unit...

 

 

 

Original unit in the crawl space...

 

 

 

24 Core unit stacked on top - fits perfect...

 

 

 

 

 

And here is the screen shot.

 

Folding1 is the 32 Core Folder

Folding2 is the 24 Core Folder

Folding3 is the 4xGPU Folder...

 

[XiaoPigs]

32 cores? What do you have? Dual Xeons?

[bradscoolio]

Oooh, looking forward to this!

[bradscoolio]

32 cores? What do you have? Dual Xeons?

4P means 4 processors. He's got 4 amd opterons ;)

[XiaoPigs]

4P means 4 processors. He's got 4 amd opterons ;)

Ah I don't know anything about folding :x

[Whaler_99]

32 cores? What do you have? Dual Xeons?

 

4 x AMD Opteron 6136 - 8 core CPU's

[XiaoPigs]

4 x AMD Opteron 6136 - 8 core CPU's

All in one machine?

[Mentalguy]

All in one machine?

yep, he could get a 4p with 4 16 core cpus

[warmon6]

All in one machine?

 

Yep.

 

While this isn't whaler's machine, he's a good example from Hardforum of what this kind of system can look like.

 

http://hardforum.com/showthread.php?t=1652906

 

 

[Mentalguy]

*drools*

[ha1o2surfer]

While he does have a lot of cores, those are pretty power inefficient if I recall. (the power per performance ratio) I have an 8 core Opteron in my older server and it really hangs up on any folding.. he will get impressive numbers (maybe) but will suck up a lot of power.. He would use way less power building 2 machines with Core 39xx series while also saving money most likely. A single 3970 can get 26kPPD while my opteron can only do 8-9kPPD using 135watts. I mean, my lonely 3630QM is faster.. BUT, throwing more cores at it might prove to work..I could be way off, but i can't wait to see any more predictions and then the real thing! ;) or maybe I am just jealous and being mean... :P

 

EDIT; well looking around I see impressive numbers.. I was wrong! I wonder why... trying to figure this out. don't kill me whaler!! haha

[Whaler_99]

What Opteron chip do you have? The ones I am getting are the HE ones, so they are very power efficient and con overclock to plus 3.2 Ghz, hence why I am getting the advance timing RAM.

[ha1o2surfer]

What Opteron chip do you have? The ones I am getting are the HE ones, so they are very power efficient and con overclock to plus 3.2 Ghz, hence why I am getting the advance timing RAM.

Opteron 4256, granted, they are low cost ones... I was wrong for basing my opinion on the Opteron I have. 

[Whaler_99]

Yep - that series is a much "slower" ones. Typically higher end 4P systems are based on 6000 series cards followed up by 8000... 

[FireFox]

Yep - that series is a much "slower" ones. Typically higher end 4P systems are based on 6000 series cards followed up by 8000... 

 

I don't mean to sound like a moron but what does 4P mean?

[bennyino]

I don't mean to sound like a moron but what does 4P mean?

4 Processors :)

[Whaler_99]

Updated with Data and Pics! :D

[bennyino]

Folding1 is just the hostname I use, for my HFM.NET. 

 

 

When you are this Big, they call you Mr.

(haven’t I used ^^^^ before? Nahhh….)

 

OVERVIEW

 

So… the purpose of this build log will be to detail my adventures in building out my first 4P folder.

 

So, up to now, 4P systems have been the way to Fold for major points (with the new Beta core for GPU’s this may change). So, some issues that come to mind are A) how to do it and B) it can cost a lot of money. So, start the reading, I spent many a night reading various threads and forums and doing research.

 

I decided not to go whole hog to start but to get a decent system to start. The other issue, costs. I can hunt around and get some deals, but the board and CPU’s and such are pretty exclusive. So, I hunted around and found some Folding forums that sell used gear to other folders. Bonus.

 

So, got all the gear ordered and here and ready to start building. I will detail various components below. For the record though, here is some of the links I liked and/or referenced (or will be) a lot.

 

Computer tech station:

http://richchomiczewski.wordpress.com/

 

Design and Build Guides:

http://www.evga.com/forums/tm.aspx?m=1638127&mpage=1&print=true

http://hardforum.com/showthread.php?t=1652906

 

Home Rack Garage Build:

http://www.evga.com/forums/tm.aspx?high=&m=1678549&mpage=1%20-%201678549

 

Ubuntu stuff:

http://hardforum.com/showthread.php?t=1601608

http://rog.asus.com/forum/showthread.php?4298-Setting-up-Linux-bigadv-folding

 

Musky’s Hyper 212 mounting method:

http://hardforum.com/showthread.php?t=1662099

 

 

CASE

 

Original I was not going to bother, Fold naked and just have everything laying there. Naked means no case, everything open and exposed. Because of the size of the board and the heat sinks, you typically need expensive, custom case, like what SuperMicro make.

 

BUT – on browsing a forum post I came across custom case builder Spotswood. His cases are simply amazing. And he makes an open frame case for folding systems that allows you to tie down the PSU, hard drive, etc, leave it all open. Looks so clean and does the job. And… you can stack multiple ones. Awesome. See below for pics.

 

Motherboard & CPU

 

So… I managed to pick up via one of the Folding Forum sites I discovered, 4 x AMD Opteron 6136 chips. These are 8 core units, and can be overclocked. The motherboard I ended up getting was a SuperMicro H8QGL-IF-O. Was able to get an ok deal on this on Ebay

 

PSU

 

So, the one place you don’t want to scrimp on is the PSU – so I went and picked up a Corsair AX850 unit. This puppy gets a 9/10 rating over at jonnyguru.com. I think I now had 5 AX series PSU’s between various systems.

 

Heatsinks

 

Need to pay some attention here. The stock ones I got are the typical low profile, with SUPER noisy fans (think Delta’s). So, I picked up four Cooler Master Hyper 212 units. The great thing, direct from the CM store, you can buy them refurb. Now, the socket is G34, not exactly compatible. But, musky from [H]ard forum has a detailed mod to adapt the Hyper 212 to work on a G34 board. Better cooling and a lot quieter. J

 

Memory

 

Again, paying some special attention here. I got some GSkill ripjaws 2GB sticks. I needed 4 sticks per CPU, so had to find 16 of them and want the CL7, XMP versions. You need these ones to support the overclock on the CPU. Finally managed to find 16 of them through newegg. Funny enough, considering, this was probably the most expensive part.

 

Other Components

 

Not much to say here. Basic hard drive to run Ubuntu.

 

Parts List

 

Case – Spotswood 4p/MEV/SWTX Open Frame tray

Motherboard – SuperMicro H8QGL-IF-O.

CPU – 4 x Opteron 6136 (8 core)

Cooler – Hyper 212 Plus + musky mod

RAM – 16x2GB GSkill

Hard Drive – Seagate 7200RPM, 80GB SATA

Fans – TBD

PSU – Corsair AX750

 

Week one...

 

My biggest issue, waiting for all the bloody parts to arrive from various sources.

 

Got the cooler and CPU’s. The ordered the case, the mobo, memory, etc…

 

Now, it is finally all here. I started with case assembly. I have to say, the workmanship by Spotswood on this A+. The case just looks beautiful. I won’t lie, took a bit of work putting together, got the old level out for a couple parts, various adjustments. But nothing some patience couldn’t solve. Once done, magnificent. I also assembled the second one I got for the other 4P unit I am putting together and stacked them to get a look, perfect fit. And such a clean way of doing this, especially if you have 5 or 6 systems… For the cost his systems destroy anything you can get from say Highpoint.

 

So case is assembled. Check. Mount in the PSU and hard drive. Check…

 

Now, onto my mods to get the Hyper 212’s I ordered to fit on the board. I basically follow musky’s guide to the letter and everything works. Some minor notes – have a dremel or other power tool for cutting the brackets. I tried one with my steel hand saw and forget that. J Now, when you get to the point when you are swinging the x brace and brackets around (after slipping them through the cooler base) to attach to the other end of the x brace, he states that although “snug” he had no issue. So, maybe my brackets were just a bit wider, don’t know, but I couldn’t attach the other end without crimping the copper heat pipes. And I DIDN’T want to do that. So, pulled the x-brace back out and slight bent the inside edge of the bracket inward, just a wee bit. I only had to do it on one end. (see attached pic) This then gave me just enough clearance to get everything screwed in nice and tight. And yes, the brackets are up against the copper heat pipes, but that isn’t an issue.

 

Now, next step is mobo mods. To get the 6-32 thread through the mounting post in the board he gives a couple of options. I honestly couldn’t figure out how cutting the head off would allow this to work. The 6-32 is still a bit too wide to fit. So, I completed the drill out method he talk about. How I did this…. Got a strong cardboard box to support the motherboard. Got the drill with the 5/32 bit and drilled from the top down. Main thing to remember here, don’t force the drill, let the bit do the work. If you force it, you could warp/bend the board. Also, by using a cardboard box, once through, you just go through the cardboard. J Once through, back out and onto next. Remember, when done, pay special attention to getting all the metal filings off the board. Surprisingly, took quite a few shakes (being careful) and blasts of compressed air to get all the filings out.

 

 

And this is where I take a break kids... going to be travelling for work for a few days and mother's day this past weekend tied me up. So, hopefulle make some progress this weekend...

you say you buy an ax850 but then on the specs its an ax 750 :P

[Whaler_99]

you say you buy an ax850 but then on the specs its an ax 750 :P

 

Stupid copy paste. :) WIll update... thx

[bennyino]

Stupid copy paste. :) WIll update... thx

no problem, happy to help :D