Home Super Computer Build - We need your numbers!

[NowThatsDamp]

Hello fellow geniuses.. My name is Michael O'Lear. My colleague Bailey Tincher and I are long time computer enthusiasts, and we've come with one goal: Advance the technologies of Mankind.

 

We don't care why.

We don't care what field.

We don't care who for.

All we care about is leaving this measly little race just a bit smarter than when we joined it.

 

100 TFLOPS by 2017 smarter, that is.

 

Contributions to Folding at home and BONIC are our method of doing this. We have our own personal businesses set up to fund a massive computing project,

In order to build our array we need some test results. As all of you smart people know, the initial building cost of supercomputers is not the main worry. Instead, the cost of powering and maintaining them are far greater. Therefore, we would like your input on the performance per watt of your GPUs. To be clear, we are not looking for the Performance/Initial_purchase_price, this is hardly relevant. What we are looking for is the Performance/Watt, the most efficient GPU on the market for these types of computing applications. (Currently, the focus is folding. Other projects may be contributed to in the future, but for now, the focus is folding).

 

Of course, if we were LMG like Linus, we could just order every GPU on the market for free, test them all and then neglect to return them, instead zip tying aftermarket heat sinks to them just to see if they work in the rain. Hell, while we're at it, why not just strap an Antec XBONE cooler jammed with duritoes to a titan X, then throw it off the roof too see if the packaging is up to par, scratch it on Austin Evans's nipple, tell people that it's dumb but they should buy it anyway like the Galaxy S6 Edge, and drop it in a cookie tray full of water sealed with sticky eraser for better cooling? Actually, we're using an Antec XBONE cooler so it would be best to change the water out with Mountain Dew. Then Luke could overclock that Titan to 7.6 jigawortz and render transformers in 1.3 milliseconds. Liquid nitrogen guys be like WHAAAAAAAAAAA?

^Possibly the greatest people on the internet^

 

Unfortunately, we're not Linus Media Group.

 

That's why we need you guys.

 

For all you forum trolls (You know who you are): Don't tell us this is a waste of money. Don't make a comment like "Who would ever spend that much money on a computer for folding?".

 

Here is the answer to why we are doing this: One day we will die. Whatever money we have will mean nothing. Whatever car, boat, gaming PC, house, swimming pool, whatever we have will count for nothing. This computer will accelerate the minds of mankind, and that is a contribution which will carry on far after we are gone. Our time and efforts put into this will have an absolute, long lasting effect. And that in it'self is an encouraging thought.

 

So be loud, be proud, and shout to the world "My GPU is the most power efficient lump of 28nm silicon on the market!". And then back it up with credible facts and tests, of course.

 

Also, if anyone spots any baller deals on 80+ gold PSU's, simple folding - geared RAM, LGA 1155 motherboards with two 16x PCI slots for cheap and other less important PC components, leave a link. 

 

 

UPDATE 6/12/15:

Picked up an Asus H87M Plus on Ebay today for $32.70 shipped today. Important specs are: LGA 1150, 2 DDR3 RAM slots 8GB each, 1 PCIE 3.0 x16 lane, and 3 PCIE 1x lanes. Now this changes things because my research says that you can run cards in 1x lanes with no performance impact for computational purposes... so I'm putting 4 cards on this board. Looks like my G3258 is not gonna work out. Anyone know what the best bang for buck 4 core i5 is?

 

Also, now I need to power 4 970's on one board. Here's my thought: Buy a PSU for two of the cards and the rest of the system. Buy another PSU which will do nothing but power the other two cards. Nobody online is saying this wont work. A lot of people say not to do it, but none of them give a reason. A few threads had stories of people doing it. Will it work?

[Elliott Game]

that came out of nowhere...

[Me1z]

If you want cheap PSUs then I suggest this deal on Newegg: http://www.newegg.com/Product/Product.aspx?Item=N82E16817139131&cm_re=ax1200i-_-17-139-131-_-Product

 

You get both an AX1200i(80+ Platinum) and an RM850(80+ Gold) for $240. I was tempted to grab them for my own folding rig, but my bank account is still hurting from my last upgrade.

 

Not sure where you and your colleague are located, but if its anywhere near New York I'd love to get involved with your project. 

[Mew]

I think you can ease your insults on Linus and his team down a little bit. Since both the GTX 970 and 980Ti are both known for their value, I would assume that both of these would work well for folding while also being power efficient, quiet, and not outputting too much heat. Ram prices are slowly going down, but afaik, folding doesn't require much ram (someone correct me) so you can be fine with 4gb or 8gb of 1333 or 1600mhz ram that you can easily get for around $50. Based on this, I would get the 850 evo if performance is also a priority, or just the BX100.

[Tedster]

I think you can ease your insults on Linus and his team down a little bit. Since both the GTX 970 and 980Ti are both known for their value, I would assume that both of these would work well for folding while also being power efficient, quiet, and not outputting too much heat. Ram prices are slowly going down, but afaik, folding doesn't require much ram (someone correct me) so you can be fine with 4gb or 8gb of 1333 or 1600mhz ram that you can easily get for around $50. Based on this, I would get the 850 evo if performance is also a priority, or just the BX100.

I agree with you on the RAM, 8GB should be plenty. I'm folding on my GTX 680 and have 44 chrome tabs open, using 6GB of RAM.

And storage really doesn't matter much for folding - just get a good cheap SSD and be done with it?

[NowThatsDamp]

I think you can ease your insults on Linus and his team down a little bit. Since both the GTX 970 and 980Ti are both known for their value, I would assume that both of these would work well for folding while also being power efficient, quiet, and not outputting too much heat. Ram prices are slowly going down, but afaik, folding doesn't require much ram (someone correct me) so you can be fine with 4gb or 8gb of 1333 or 1600mhz ram that you can easily get for around $50. Based on this, I would get the 850 evo if performance is also a priority, or just the BX100.

I apologize. Those where NOT insults to Luinus Media Group. I LOVE them. I've followed Linus since he was in his garage. It was just joking banter. That's all.

[NowThatsDamp]

You guys have been helpful thus far. But remember, I'm not looking for pure computational output. Im looking for highest computational output per watt of power used. This is an essential detail as this will be the mindset the entire computer is built around. 

I am planning on getting a good cheap SSD, just looking for pointers towards sales.

[Tedster]

You guys have been helpful thus far. But remember, I'm not looking for pure computational output. Im looking for highest computational output per watt of power used. This is an essential detail as this will be the mindset the entire computer is built around. 

I am planning on getting a good cheap SSD, just looking for pointers towards sales.

I'd have to say that the GTX 970 is probably the best performance per watt.

And for a cheap SSD, I'd just like to make you aware that the Kingston V300 isn't a good SSD. Read more about it here: http://www.anandtech.com/show/7763/an-update-to-kingston-ssdnow-v300-a-switch-to-slower-micron-nand

[Mew]

Wait a minute..... why would you need a hard drive for a system that is only used for folding? 

[Warren Pnigel]

 

 

Here is the answer to why we are doing this: One day we will die. Whatever money we have will mean nothing. Whatever car, boat, gaming PC, house, swimming pool, whatever we have will count for nothing. This computer will accelerate the minds of mankind, and that is a contribution which will carry on far after we are gone. Our time and efforts put into this will have an absolute, long lasting effect. And that in it'self is an encouraging thought.

 

 

We will all die one day, so you should spend your money on stuff that makes you happy because the truth is in 30 years any graphics card being used to fold with will look ancient lol. 

[blackadder]

Hi, i understand where your coming from.

Im hard pressed not to think of overall cost to, im just cheap. So the higher rated psu may not tanslate to lower cost over all because of the initial cost difference.

Gpu. Boinc projects use different things some only nvidia some amd. Some only use cpus. 970 or 980ti would be best power/work. You can have all kinds of compute units so you can mix or match or get second hand.

Storage. Power saved over a hdd would be. Minor, put the money in gpu or cpu.

Cpu. This is difficult. You want enough cores to "feed whatever gpus you have" but, it depends on how many gpus because of pci lanes to get the one that supports the amount of lanes you need.

Work. This isn't well defined in boinc you get vastly different amounts from different projects so you should define what you feel is work, do you want to find primes, look for pulsars or do field algebra. Look at what projects you want to do and see whats the most useful.

[NowThatsDamp]

Bailey and I are currently thinking this:

Redundant arrays of simple motherboards, each with a Pentium G3258 feeding 2 970's.

The G3258 will be slightly undervolted and slightly overclocked, as our own personal tests have proven to be most efficient. (We got my sister's old Vista laptop to put out 88PPD/Watt using this method. Not bad for an old cheap CPU). 

Once we have our hands on the GPU's we will do some testing to figure out the most efficient clock rates. We will see if the slight undervolt/overclock system works again.

4GB of simple RAM, likely 1333MHZ

A 80+ gold PSU. Platinum is overkill.

The entire system will be Linux based. Hopefully we can create a central computer and use this fore mentioned setup as satellites.

As for hardware sourcing, we are looking for 970's. We will buy two for testing now, however to hit the 75 TFLOP Nvidia based goal, we will need at least 8 full satellites one day. We are always looking for great deals on hardware.

That, we plan, will eventually make up 75TFLOPS of our compute output. As Blackadder said, some Bonic projects are AMD based. For versatility, the other 25TFLOPS will be AMD based. This is if, and only if, we can find AMD cards with reasonable efficiency. Any recommendations?

[Me1z]

Bailey and I are currently thinking this:

Redundant arrays of simple motherboards, each with a Pentium G3258 feeding 2 970's.

The G3258 will be slightly undervolted and slightly overclocked, as our own personal tests have proven to be most efficient. (We got my sister's old Vista laptop to put out 88PPD/Watt using this method. Not bad for an old cheap CPU). 

Once we have our hands on the GPU's we will do some testing to figure out the most efficient clock rates. We will see if the slight undervolt/overclock system works again.

4GB of simple RAM, likely 1333MHZ

A 80+ gold PSU. Platinum is overkill.

The entire system will be Linux based. Hopefully we can create a central computer and use this fore mentioned setup as satellites.

As for hardware sourcing, we are looking for 970's. We will buy two for testing now, however to hit the 75 TFLOP Nvidia based goal, we will need at least 8 full satellites one day. We are always looking for great deals on hardware.

That, we plan, will eventually make up 75TFLOPS of our compute output. As Blackadder said, some Bonic projects are AMD based. For versatility, the other 25TFLOPS will be AMD based. This is if, and only if, we can find AMD cards with reasonable efficiency. Any recommendations?

 

It depends on how the project strains the GPU, as every project is different, but I have found that an R9 295x2 can be just as efficient as as GTX 970s when it comes to compute applications like folding. I have one in my folding rig; the full system wattage averages around 480W and it earns between 460-550k PPD (F@H). AMD may not have the best reputation when it comes to efficiency, but I'd like to think that having higher binned chips in a dual GPU card helps in that department.

[NowThatsDamp]

It depends on how the project strains the GPU, as every project is different, but I have found that an R9 295x2 can be just as efficient as as GTX 970s when it comes to compute applications like folding. I have one in my folding rig; the full system wattage averages around 480W and it earns between 460-550k PPD (F@H). AMD may not have the best reputation when it comes to efficiency, but I'd like to think that having higher binned chips in a dual GPU card helps in that department.

I've other people with these number too. And that's huge considering how one 295x2 is cheaper then two 970's used, with higher computational output. What I need is two 970's and a 295x2, slap them down on our satellite board, then compare the numbers. Being able to get 4 GPU's on one board then cuts in half the number of satellites needed and cuts initial cost. Plus I can only imagine what 10 water cooled GPU's would look like all on the same shelf.

Does anyone know of severe AMD cripplings which would make an entirely AMD build silly?

[mmmbrrrr]

i used to run my cpu thinking im getting the most ppd but i have found it to be a waste of time and you are just better to let the gpu do there work with no cpu bottle neck. if you are going to put 4 gpu in one rig for folding i would recomend unclocking the cpu and not folding with it at all

[Imakuni]

Also, if anyone spots any baller deals on 80+ gold PSU's, simple folding - geared RAM, 5400 RPM low power HDD's, small size cheap SSD's (For power efficiency, of course) and other less important PC components, leave a link. 

Wait. If you are doing a massive project dedicated for Boinc, you don't use HDs and SSDs. You just grab ONE huge HD, put in a "master" PC and have all other machines boot to it via Ethernet. That's a great measure to spare initial costs of purchase and reduce power consumption. It's a bit tricky to set up, but once done, it works like a charm.

 

At any rate, you have to define what you want a bit better. Are aiming specifically for FAH, mostly for FAH (but with some Boinc) or just anyone will do?

[NowThatsDamp]

Wait. If you are doing a massive project dedicated for Boinc, you don't use HDs and SSDs. You just grab ONE huge HD, put in a "master" PC and have all other machines boot to it via Ethernet. That's a great measure to spare initial costs of purchase and reduce power consumption. It's a bit tricky to set up, but once done, it works like a charm.

 

At any rate, you have to define what you want a bit better. Are aiming specifically for FAH, mostly for FAH (but with some Boinc) or just anyone will do?

The machine will be doing FAH at first, then used for other applications later as it makes sense. And yes, that boot system in what I was planning on using. I'm not sure what I was planning on doing when I asked for low price SSD links...

It was late I suppose.

[Imakuni]

The machine will be doing FAH at first, then used for other applications later as it makes sense. And yes, that boot system in what I was planning on using. I'm not sure what I was planning on doing when I asked for low price SSD links...

It was late I suppose.

Oh well. Then I guess you are looking into a bunch of Gtx 970/80s. While they wont do that great in FAH (damn you nvidia...), they'll be awesome, once you move to Boinc. In regards to 970 VS 980, I would make some calculations and see if by going with 970s, you can afford to buy more systems. If not, I guess you might as well just stick to 980s.

 

Remember that you need 1 core per GPU (in FAH's case), so you are looking at a bunch of i5s. More specifically, the i5 5675C. They offer as much performance as the Haswell CPUs, but they come at a lower TDP. Also, their Iris Pro graphics is a powerfull GPU that can also help in Boinc tasks....I think. I know you can use 3rd and 4th gen's iGPUs for Boinc, but I'm not sure if the 5th is supported already.

 

Last (still about FAH), remember the PCIe requirements. Now, you need 4 PCIe x16 slots, that's obvious. But not every mobo that has it actually supports all slots being used at once. And even for the ones that do, you need each slot to have 3.0 x4 capability, if I recall the exact numbers correctly. Maybe it was 2.0 x4 or x8 instead, I'm not sure. At any rate, pay attention to that.

[EvilNugget]

You guys have been helpful thus far. But remember, I'm not looking for pure computational output. Im looking for highest computational output per watt of power used. This is an essential detail as this will be the mindset the entire computer is built around. 

I am planning on getting a good cheap SSD, just looking for pointers towards sales.

Cheap SSDs are not the best, but it will do for your purpose (as far as I know) (mostly)

[NowThatsDamp]

Oh well. Then I guess you are looking into a bunch of Gtx 970/80s. While they wont do that great in FAH (damn you nvidia...), they'll be awesome, once you move to Boinc. In regards to 970 VS 980, I would make some calculations and see if by going with 970s, you can afford to buy more systems. If not, I guess you might as well just stick to 980s.

 

Remember that you need 1 core per GPU (in FAH's case), so you are looking at a bunch of i5s. More specifically, the i5 5675C. They offer as much performance as the Haswell CPUs, but they come at a lower TDP. Also, their Iris Pro graphics is a powerfull GPU that can also help in Boinc tasks....I think. I know you can use 3rd and 4th gen's iGPUs for Boinc, but I'm not sure if the 5th is supported already.

 

Last (still about FAH), remember the PCIe requirements. Now, you need 4 PCIe x16 slots, that's obvious. But not every mobo that has it actually supports all slots being used at once. And even for the ones that do, you need each slot to have 3.0 x4 capability, if I recall the exact numbers correctly. Maybe it was 2.0 x4 or x8 instead, I'm not sure. At any rate, pay attention to that.

Yes I'm aware of the 1 core per GPU. When I said 4 GPU's I was referring to two 295's, bringing 2 physical cards with 4 physical processors. And that leaves me with a question. Because the 295 has two GPU's, do you think one card would require 2 CPU cores?

[Imakuni]

Yes I'm aware of the 1 core per GPU. When I said 4 GPU's I was referring to two 295's, bringing 2 physical cards with 4 physical processors. And that leaves me with a question. Because the 295 has two GPU's, do you think one card would require 2 CPU cores?

If I recall it right, yes, it will. However, I'd advise against the 295x2. Mainly because you'll be doing stuff other than FAH, and when you do, power consumption will just skyrocket.

[Whaler_99]

Some points....

 

99% of people do not have a Kill-o-Watt so are guesstimating usage. Also, every single system gives different numbers, even is very close in hardware config. Hence one main reason we have never bothers with a database. At any time your 970 may be getting 70K PDD or 200K PPD... it varies...

 

Right now, best bang for buck, 970's, esp with the the updated Core18 projects. Haven't seen results from 980ti yet though.

 

GPU's do not run on Linux, you will need to install windows, on each system. No central boot solution. Except if running something virtualized, but then you be srewing yourself over for various other reasons I won't go into to.

 

For each GPU you only need a x1 slot. You can use x1->x16 expanders if needed. And don't need to be PCIe v3 or anything like that. Pretty much any board will work that has the proper number of slots and electrical signalling (read my GPU Build Guide).

[NowThatsDamp]

If I recall it right, yes, it will. However, I'd advise against the 295x2. Mainly because you'll be doing stuff other than FAH, and when you do, power consumption will just skyrocket.

Alright. Thanks for the advice. I think 970's are gonna be the way to go.

[Tedster]

 

I've other people with these number too. And that's huge considering how one 295x2 is cheaper then two 970's used, with higher computational output. What I need is two 970's and a 295x2, slap them down on our satellite board, then compare the numbers. Being able to get 4 GPU's on one board then cuts in half the number of satellites needed and cuts initial cost. Plus I can only imagine what 10 water cooled GPU's would look like all on the same shelf.

Does anyone know of severe AMD cripplings which would make an entirely AMD build silly?

AMD CPUs have far worse performance per watt.

[ImBleu]

 

Say no more.