Jump to content

Tobben's 4P Build Log, 48 core folder. (mini-Malossi)

mini Malossi

 

 

 

i have added all the sources i used at the bottom, a big thanks to whaler for talking to me and helping me for the past month.

 

it ended up costing me about 3000$ and the folding numbers i have been getting so far are.

WU 8103   frame time 11:04    PPD 490k

 

WU 8101   frame time 15:08    PPD 300k 

 

WU 8104   frame time 08:28    PPD 465k

 

WU 8105   frame time 11:04    PPD 460k

 

 

this has been in the making for well over a month, spendt pretty much all that time looking around trying to find parts, the actual assembly and installation process took about a day, i went full out with a g34 board and newer processors, it cost a crapload of money (3000$ to be exact..) but the benefit to this over a 500$ partly pre assembled solution with the lower end hardware is that this is going to live for alot longer(well duh), later on i can just swap out the processors for better ones because the motherboard supports it, it should also be more efficient power to performance wise, not to mention that it is great fun to play around with all this new shiny and amazing hardware :).

 

 

 

 

 
 
 The speccs are:
 
CPU: 4x amd opteron 6172 12 core processors.     (bought on ebay)
 
Motherboard: Supermicro H8QGL-IF+                     (also bought from ebay) 
 
PSU: Corsair rm 850W                                             (the rest were ordered from various online stores within norway)
 
CPU cooler: 4x hyper 212 evo
 
Ram: Corsair 6 memory stick kit 1333 mhz cl9 1,5 volt and all that, these sticks look aswesome! to bad they no longer sell them so i could only get 6 of them.
         10x kingston hyperx ram with the same speccs as the corsair ones to fill the remaining slots.
 
Storage: corsair force 128 GB ssd, the ssd was gifted to me for the sake of helping me get it up and running.

a 500 GB Hitachi harddrive.

2x 4TB WD green.

 
 
 
 
 
 First up is the Musky modded 212 evo heatsinks.
 

 

post-955-0-11756600-1383390682.jpg
 
the standard mounting bracket, just bend off the black clips at the end, and slide off the screws/springs.
 
post-955-0-70350800-1383390827.jpg
 
This is how it looks once you have taken it appart, i kept everything incase i would need it, i got a drawer full of screws and whatnot, il be using the tension springs later when mounting the heatsinks, which later proved to be a not so great solution.. but it worked out.
 
 
post-955-0-28413500-1383390872.jpg
 
i took the rails from an old closet, just took out the rails used to hold the drawers and used the ones that seemed like they would fit the best.
 
 
post-955-0-17453100-1383390985.jpg
 
cut them to the proper length with an old rusted saw, my arm was about to fall off once i was done .. so was the saw, i came to the conclusion that about 5cm beetwen each screw hole should be sufficient, but i was lucky so i just followed the pattern on the rail.
 
 
This is the finished result.
 
post-955-0-64287400-1383391086.jpg
 
 
post-955-0-67582100-1383391116.jpg

Link to comment
Share on other sites

Link to post
Share on other sites

Update 2

 

 

 

 

 

 

 

 

 

post-955-0-54390700-1383421818.jpg

 

This is the shelf i figured i would use, i had to saw off some bits and adjust it somewhat but i think it fits pretty good, i don't think it will fall off anytime soon, it will hopefully keep the cats away aswell.

 

 

 

post-955-0-23991500-1383421912.jpg

 

OH YES, the motherboard has arrived.

 

 

post-955-0-18060100-1383421951.jpg

see that accessory box? yeah it's pretty much empty ... boring right? it only contains a small cd .. it also comes with a checklist that says i should have the following items 

foam, motherboard, cd , box, fantastic right? :P

 

 

 

 

post-955-0-14911600-1383421970_thumb.jpg

 

oh yes, for the most part this is the table of contents, it's every techies dream.

about 3000$ in parts right here.

 

 

 

post-955-0-11201500-1383422015.jpg

 

the picture is terrible but this is what i have and the cpu's are now sitting cozy and comfortably in the motherboard so .. yeah.

 

 

 

 

post-955-0-49413400-1383422091.jpg

 

Ram and cpu's are in, aren't those cpu's just the greatest thing ever? does anyone agree?.

 

 

 

 

 

post-955-0-65657700-1383422137.jpg

 

while i wanted to do everything possible to avoid this i was still stuck with drilling... had to do some drilling and expand the holes to be able to fit the screws.

i got 5 cm long and 4mm wide screws .. im just guessing but i think 4 cm and 2 mm might fit.

 

 

 

post-955-0-90384300-1383422440.jpg

 

arctic MX-2 to the rescue! posting images of thermal paste applying is a dangerous thing ..  this is probably the worst out of the 3 cpu's, this was a failure but im ilustrating a point, im making 2 thin identical stripes across the long side of the cpu.

 

 

 

 

post-955-0-47271700-1383422213.jpg

 

does anyone agree with me that the XMS3 ram looks amazing? it really it a shame that i had to mix ram sticks.

 

now partially done, you can see the screws that go from underneath the motherboard and up, you got to be carefull while mounting the heatsinks they are razor sharp, by the end i had a hard time doing anything because my hands were covered in badages :P, seriously the hardest thing by FAR has been mounting these heatsinks, the aluminium fins were actually bendt pretty badly by the time the whole process was done, i never expected it to be this difficult and nerve wrecking, i would probably die if i tried to do this again, screw thermal paste methodology .. here we just need to make it fit and sit where it should!.

 

 

 

 

 

post-955-0-86927700-1383422857.jpg

 

i had to add 2 washers beetwen the screw head and the motherboard, as you can see the screw would have crashed into the heatsink otherwise, but that turned out in my favor, the screws i used were 5 cm long and 4 mm thick, well .. 5cm is to long and 4mm is also to thick.. there is some distance beetwen my motherboard and where it's standing because it's resting on the screws, i actually had a fair bit of trouble with the hex nuts crashing into the copper heatpipes but was able to surpass it with some twisting and turning, i couldn't really equal the pressure properly on both sides as i did all kinds of horrifying things just to be able to tighten them at all. it could have been worse though, it will be interesting to see the temps... 

 

 

 

 

 

 

post-955-0-72218300-1383423079.jpg

 

this is how it looks with everything mounted, by far the worst thing was mounting the heatsinks.. i swear musky modding is bad for your health.. i might not do it again if i have the choice but .. the upside is you get some fantastic heatsinks and a system that overall looks great and is alot more imposing, ok well .. looking at my machine, i would probably do it again just because it's awesome.

Link to comment
Share on other sites

Link to post
Share on other sites

all you cable managment lovers out there avert your eyes now

 

honestly though, it's a 4p server, it's supposed to look abit messy, i was thinking in the future i might modd my shelf abit, adding piece to cover up the cords, and perhaps some mounting for my optical drive and ssd, i can't make it to heavy though, and that is for the future, for now i have to get it up and running.

 

 

post-955-0-93493700-1383509717_thumb.jpg

yes the fan to the right is tilted abit high, fixed it later.

 

 

 

 

post-955-0-97066300-1383423281.jpg

 

glory to the optical drive! it adds both prestige and personality to every machine!

 

i have lots of great ideas on cleaning it up and generally make it look awesome, but for now that is second priority. i have an awesome solution to mounting a fan at the bottom blowing up undeneath the motherboard and one blowing across the motherboard ontop, and just generally cleaning up the cords, but right now it's functionality above everything else.

 

 

post-955-0-19687200-1383425787.jpg

 

pardon the mess but this place was pretty much uninhabitable by the time i was done.

 

it wasn't very easy to take pictures because there's so much black here, and the walls are white/yellowish so the lighting gets imbalanced.

 

 

 

 

 

 

post-955-0-00731300-1384343782_thumb.jpg

 

soooo this is the current state of the system, got a new fan, tidied up the cables alittle bit and added a 500 GB hardrive, go to page 3 if you are curious about the fan mounting :D

cables hanging down from my shelf was kinda stupid so i fixed that.

Link to comment
Share on other sites

Link to post
Share on other sites

Update 3

 

 

 

 

 

 

it's now up and running, im sitting here with both windows open atm, it's certainly throwing out more heat than any oven i have ever had, good thing i have so many cats.. so i can only put the windows on a slight tiny tilt... seriously this thing produces alot of heat! il try to cook some eggs on it once im done updating the log, i could always try to boil some noodles aswell. (on a serious note i do defrost my frozen berries in the exhaust of this thing)

 

it booted with no problems at all, just put in the cd and installed ubuntu, the whole process was painless and took about 5 minutes, which is a surprise considering all the things i did to this motherboard in order to be able to mount the heatsinks, im just happy it booted.. .. you guys go oh no! don't put it ontop of the anti static bag!  and don't touch it there! 

now here we are drilling holes and dropping bolts on it ... just trying not to bend it or pierce it with the screw driver :P this motherboard cost me 1200$. 

 

seriously, props to the peeps over at hardforum, they have already written all the code and scripts for you, so all you have to do is basicly, just download and run it, which is amazing, the install takes 5 minutes instead of 5 hours, honestly anyone can do it, the job is already done for you.

 

 

 

 

 

 

 

 

post-955-0-18503400-1383505600.jpg

 

 

currently got it plugged into my main monitor, and yes that is vga you see me using on a dell U2412M ips monitor.( only on the 4p folder though), got to switch input in order to check up on the secondary machine.

 

 

 

 

 

 

post-955-0-88840100-1383506109.jpg

 

oh yes, all dem threads, currently seems like it's using about 11 minutes on each % so i guesstimate that's roughly 17-18 hours ish until completion, and yes i called it mini malossi and i love the name :P, my main machine is Malossi so it felt like the right thing to call it.

 

 

 

 

post-955-0-02265000-1383506216.jpg

 

currently got 32 cpu's in system monitor for whatever reason, atleast it's up and running now this system is still in it's very early infancy so there is a bunch of things i have to look up. 

 

 

 

post-955-0-17311500-1383842867_thumb.jpg

 On the 8105 unit im pretty much getting stable 460k ppd, think i might be able to get abit more performance out of it.

 

 

 

 

conclusion:

 

honestly the hardest part by far was just finding and being able to buy the parts.. that's what took me over a month, and my purchases were far from ideal... however! if you are made of money the buying process should be pretty straight forward as long as you know what you want.

mounting the heatsinks was also a horrifying gut wrenching experience i never want to have again, when it comes to assembly mounting the heatsinks was by FAR and wide the hardest part, you think you know what to expect, but trust me it's 5 times worse than you think, i was lucky so the setup after assembly was smooth without a hitch, just booted up put in the cd i burned, rebooted, installed and followed hardforums ubuntu guide, they have done all the difficult things for you, you just have to download and run it according to their description, they basicly "guide" you to download their script and run it, full installation was straight forward quick and easy, however buying parts and mounting non native heatsinks is the suck.

 

You could try buying some used parts and using old parts from previous rigs, i guess the motherboard is the hardest component to buy for anything lower than retail price, processors don't have to be THAT expensive and you could land a lucky purchase, grab some ram sticks that are compitable with your machine, then aside from that there really isn't anything special, get a powerfull fairly efficient psu with atleast 2 x 8 pin supplementary cpu power connectors, heatsinks to your liking that will fit the socket of your machine, find an old harddrive somwhere and format it, grab a usb stick or an optical drive depending on which method of installation you prefer, find a couple of old fans and you should be good to go.

 

 

 

 

 

 

 

i would have a look at some of these sites for guides and whatnot.

 

 

General overview

 

overclock.net    http://www.overclock.net/t/1212613/guide-building-a-2p-4p-from-start-to-finish-along-with-ppd-benchmarks-gallery

 

Evga   http://forums.evga.com/tm.aspx?m=1638127&mpage=1

 

Hardforum   http://hardforum.com/showthread.php?t=1652906

 

 

Musky modded heatsinks. (hardforum) http://hardforum.com/showthread.php?t=1662099

this one is for H8QGL motherboards, he also has one for H8QGI which is easier because you have more space beetwen the cpu's, i like this method because it seems to cover the cpu better, i used 212 evo so those can be used aswell.

 

Ubuntu (Hardforum)  http://hardforum.com/showthread.php?t=1601608

 

 

and here is some ppd numbers on various projects and processor, take them with a grain of salt though

https://docs.google.com/spreadsheet/ccc?key=0Aspl6-RkaxtFdHdTdUdmUjhWalpXWVZ2S2xvejBDcHc#gid=0

 

 

 

(continue flipping through the pages for more pictures.)

Link to comment
Share on other sites

Link to post
Share on other sites

48 core?! ;O 

[CPU: AMD FX-6100 @3.3GHz ] [MoBo: Asrock 970 Extreme4] [GPU: Gigabyte 770 OC ] [RAM: 8GB] [sSD: 64gb for OS] [PSU: 550Watt Be Quiet!] [HDD: 1TB] [CPU cooler: Be Quiet! Shadow Rock Pro Sr1]  -Did i solve your question/problem? Please click 'Marked Solved'-

Link to comment
Share on other sites

Link to post
Share on other sites

Moment of silence for the ol' nokia phone -_- , your phone have been sacrificed for a good cause.  But the door shall be repaired... with duct tape.

 

 

Ram: Corsair 6 memory stick kit 1333 mhz cl9 1,5 and all that, these sticks look aswesome! to bad they no longer sell them so i could only get 6 of them.
         10x kingston hyperx ram with the same speccs as the corsair ones to fill the remaining slots.

Certified or ECC Ram?  Supermicro boards are picky.  Good luck.  They can work, but can kill you at high ram usage. 
 
 

Storage: corsair force 128 GB ssd
yes i know .. i shouldn't be using an ssd for this but .. it was gifted to me to help me with the project.

How come you think ssd is not good in the first place?  Would be a shame not to.

My Rigs (past and present)

Link to comment
Share on other sites

Link to post
Share on other sites

You mean @Whaler_99 right?

yeah yeah have it your way 

If you tell a big enough lie and tell it frequently enough it will be believed.

-Adolf Hitler 

Link to comment
Share on other sites

Link to post
Share on other sites

 

Certified or ECC Ram?  Supermicro boards are picky.  Good luck.  They can work, but can kill you at high ram usage. 

 

 

How come you think ssd is not good in the first place?  Would be a shame not to.

hah, i wanted to add a memorial for my nokia phone, it felt like the right thing to do :P

 

in fact im using non buffered non ecc memory .. but they fit all the criteria so it should be good, i hope .. il find out soon as i picked up the motherboard just now, at first they couldn't give me the package because the servers were down (yikes!) but they got it sorted out in a couple of hours... thank god i didn't have to wait until monday! the ram, heatsinks, and all the other parts have been sitting here collecting dust for weeks and weeks.

 

well i kinda feel a ssd is wasted on this system simply because it isn't needed, but it's worth doing just for the sake of going overkill, i also got a old 500 GB harddrive il throw in there for the sake of server storage.

Link to comment
Share on other sites

Link to post
Share on other sites

Weak startpost :( No motherboard picture yet!

i have mounted 2 out of 4 heatsinks, don't worry pictures are coming up, i won't update the build log before im done with putting it all together, if you had read it you would have known i uploaded some minor things before i went to pick up the motherboard because i didn't have it yet, i have it now though :P, the motherboard was the last thing i aqcuired, everything else was collecting dust in the meantime.

 

this is in progress, i will probably redo the entire build log once im finished.

Link to comment
Share on other sites

Link to post
Share on other sites

I read it! But 4way motherboards must be shown!! Priorities ;)

Link to comment
Share on other sites

Link to post
Share on other sites

I see you also have an electric stove... you won't be needed that anymore.

 

hah, i wanted to add a memorial for my nokia phone, it felt like the right thing to do :P

 

in fact im using non buffered non ecc memory .. but they fit all the criteria so it should be good, i hope .. il find out soon as i picked up the motherboard just now, at first they couldn't give me the package because the servers were down (yikes!) but they got it sorted out in a couple of hours... thank god i didn't have to wait until monday! the ram, heatsinks, and all the other parts have been sitting here collecting dust for weeks and weeks.

 

well i kinda feel a ssd is wasted on this system simply because it isn't needed, has no benefit and has no purpose :P but i guess it's worth doing just for the sake of it, i also got a old 500 GB harddrive il throw in there for the sake of server storage.

How much memory did you get?

 

I got ecc modules for a couple xeon system, and went according to the memory module models they recommended (not the stick, but the chips) which worked fine under windows server and windows 7, but had issues when I starting using Virtual machines on top.  Fortunately those vm's are not critical, and just to access old systems data.  Issue has been resolved.  Turns out there was some odd spec in the memory board (newegg had it at the time but wasn't in stock, so I got the other one  :rolleyes: .)  Btw, the original incompatible ones did pass memory tests and folding... over a week continuous too with system loaded.  I guess memory test are not random enough.

 

My personal system uses the same xeon board, but I went to kingston and used their system to choose motherboard.  They have a nice system setup that gives you like to retailers like newegg or buy directly from them.   All 32GB are working fantastic loaded up with a bunch of VMs 24/7.  

 

Well, just a heads up in case you run into funny behavior.  

 

I guess as a folder, ssd is kind of a waste...  A usb stick with linux would probably do the job just as well  :lol: .

My Rigs (past and present)

Link to comment
Share on other sites

Link to post
Share on other sites

I see you also have an electric stove... you won't be needed that anymore.

 

How much memory did you get?

 

I got ecc modules for a couple xeon system, and went according to the memory module models they recommended (not the stick, but the chips) which worked fine under windows server and windows 7, but had issues when I starting using Virtual machines on top.  Fortunately those vm's are not critical, and just to access old systems data.  Issue has been resolved.  Turns out there was some odd spec in the memory board (newegg had it at the time but wasn't in stock, so I got the other one  :rolleyes: .)  Btw, the original incompatible ones did pass memory tests and folding... over a week continuous too with system loaded.  I guess memory test are not random enough.

 

My personal system uses the same xeon board, but I went to kingston and used their system to choose motherboard.  They have a nice system setup that gives you like to retailers like newegg or buy directly from them.   All 32GB are working fantastic loaded up with a bunch of VMs 24/7.  

 

Well, just a heads up in case you run into funny behavior.  

 

I guess as a folder, ssd is kind of a waste...  A usb stick with linux would probably do the job just as well  :lol: .

 

hah, i actually have 2 big electrical stoves here, my plan was actually to get this up and running before the winter really kicks in so that it would provide supplementary heat, it might provide enough heat alone,

and yeah the ssd is definitely wasted on the system .. i will use it for now, might swap it out in the future, as previously mentioned it was gifted to me to help me with geting the system up and running, i kinda ran into trouble on every possible thing there could be trouble with .. :P

 

i just got 2 GB sticks so it's 32 gigs of total memory, i have corsair and kingston ram mixed on cpu 1, hopefully it will work out.

 

 

Build log updated btw, will continue tomorrow.

Link to comment
Share on other sites

Link to post
Share on other sites

You should do a custom loop for it.

doing a custom loop would actually be very awesome, i'm pretty sure you wont be able to actually find any water blocks for the g34 socket though, wait ... now that you mention it i think i have seen it the past sometime, but after all the hassle and trouble the current heatsinks has caused me i want to keep them for abit.

 

and not to mention cost! it will be a while until i can even spend a single penny more on this machine .. sadly.

Link to comment
Share on other sites

Link to post
Share on other sites

Does RAM effect Folding performance?

Awesome build BTW.

The first step to insanity is believing in your sanity.

Link to comment
Share on other sites

Link to post
Share on other sites

doing a custom loop would actually be very awesome, i'm pretty sure you wont be able to actually find any water blocks for the g34 socket though, wait ... now that you mention it i think i have seen it the past sometime, but after all the hassle and trouble the current heatsinks has caused me i want to keep them for abit.

 

and not to mention cost! it will be a while until i can even spend a single penny more on this machine .. sadly.

There is a G34 version of the Aquacomputer Kryos.

BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to comment
Share on other sites

Link to post
Share on other sites

sweet, that would be way to expensive though :P, im going to keep my 212 heatsinks for a while, making a custom water loop would probably cost around 450$

(scratch that, it would cost small fortune), and for me it would easily exceed that ammount with importing taxes and whatnot (making is a medium sized fortune!), while doing a custom loop would be cool, when it comes to these types of machines just practicality and simplicity is beautiful in itself, just straight barebones hardware that does the job it should do, not to fancy shmancy 

Link to comment
Share on other sites

Link to post
Share on other sites

Does RAM effect Folding performance?

Awesome build BTW.

 

 

you need some, but i doubt anything over 16GB would help much.

~Judah

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×