The Big Hasher

[riceusc2]

Hey guys, 

 

I'm little new here but i'm thinking I'm going to fit right in. I'm in the process of a build that i would like some help/thoughts on. I currently have 4 AMD 7990s and 4 7970s in my possession and need to create a linux base system capable of running all of these. In the past I've only been able to strap two of these bad boys in a computer because of the massive 3U heat sinks. The application that will be running on this will be a Hash cat server so needless to say, heat is my biggest battle. I have often ran these cards around 90C and i need to stop doing that. So to solve all of my issues, I'm looking to build a mineral oil solution to put these 8 GPUs. I would like to use one of those kits linus has used on his show but as some of us know, they are no longer being made because of copyright issues.

 

Some specifics for this build:

 

Motherboard:

No mother board purchased yet but it needs be able able to handle the 8x GPUs

 

Power Supply:

Haven't purchased one yet but i have a spare 1600 Watt laying around that it can be built into

 

SSD:

looking for a M.2

 

Case:

Of course probably a fish tank but open to suggestions

 

Water Cooling stuff:

I have none of it yet

 

GPUs:

I have these in hand and they are roaring to go

 

Ram:

DDR4 PLZ

 

CPU:

Yet to be purchased

 

Cost:

Since it's for a college i don't have an unlimited budget but i can always ask for money. To start, lets say $1,000.

 

Give me some ideas to help get this creative process flowing! Build will happen in mid March, so i have time to plan.

[Samfisher]

You have GPU's that have coolers that are...3U? Like, 5.25inches thick?

[riceusc2]

Hey guys, 

 

I'm little new here but i'm thinking I'm going to fit right in. I'm in the process of a build that i would like some help/thoughts on. I currently have 4 AMD 7990s and 4 7970s in my possession and need to create a linux base system capable of running all of these. In the past I've only been able to strap two of these bad boys in a computer because of the massive 3U heat sinks. The application that will be running on this will be a Hash cat server so needless to say, heat is my biggest battle. I have often ran these cards around 90C and i need to stop doing that. So to solve all of my issues, I'm looking to build a mineral oil solution to put these 8 GPUs. I would like to use one of those kits linus has used on his show but as some of us know, they are no longer being made because of copyright issues.

 

Some specifics for this build:

 

Motherboard:

No mother board purchased yet but it needs be able able to handle the 8x GPUs

 

Power Supply:

Haven't purchased one yet but i have a spare 1600 Watt laying around that it can be built into

 

SSD:

looking for a M.2

 

Case:

Of course probably a fish tank but open to suggestions

 

Water Cooling stuff:

I have none of it yet

 

GPUs:

I have these in hand and they are roaring to go

 

Ram:

DDR4 PLZ

 

CPU:

Yet to be purchased

 

Cost:

Since it's for a college i don't have an unlimited budget but i can always ask for money. To start, lets say $1,000.

 

Give me some ideas to help get this creative process flowing! Build will happen in mid March, so i have time to plan.

[Decon]

why do you need a hashcat server >_>

 

so basically you want us to help you build a massive password cracker... riiiiiight...

[riceusc2]

This is the card with the 3U cooler i was referring too: Link
Also yes, We have a cyber security team that needs to be able to hash passwords for penetration testing and competitions.

 

 

You have GPU's that have coolers that are...3U? Like, 5.25inches thick?

not exactly, when referring to U in this instance, i mean the distance from one X16 slot to the next, if you fallow the new thread, i have posted a link to it there.

[brandishwar]

To run all 8 of those cards off the same mainboard, you're going to need PCI-Express extenders. I'd highly recommend using multiple power supplies on this as well as I doubt even a 1600W PSU can support all of what you're intending to run how you're intending to run it. Better yet, get 3 PSUs: run 4 GPUs off 1, run 4 GPUs of another, and use the last to power the mainboard.

On that mark, for the PCI-Express extenders, I'd recommend going with ones that use USB 3.0 cables. Those will be much easier to power since the side connecting to the graphics card is powered separately from the rest of the mainboard. I used those in a project.

Water cooling will be beneficial as well, but you're going to need probably two quad radiators at minimum to handle that heat.

Out of curiosity, why are you doing all of this?

[brandishwar]

To amend my previous, if you're wanting one board to push all 8 of those cards, you're going to be hard pressed to find one with only a 1000 USD budget. Most mainboards won't have enough PCI-Express slots. And I'm skeptical that any that do have that many can support that many graphics cards. The ATX specification typically allows for only 7 slots at most, though some 8-slot boards do exist. None that support DDR4 currently, though. X79 is the last chipset I've seen with boards that have 8 PCI-Express slots. Again, though, whether those boards can support those graphics cards and run stable? I'm doubtful.

It's only server boards that I've seen that can support more, and they're typically custom server boards. Otherwise you're looking at trying to run PICMG 1.3 system to support up to a 20-slot backplane. If any of the GPUs have blocks available that will allow you to swap the I/O plate to a single-slot plate, then you might be able to use those in a 14-slot backplane.

If you can get one of the X79 boards that has all 8 PCI-Express slots (with none of them being disabled based on where cards are plugged in), then you're going to need a chassis to hold all that. And the only one I know of is the Ascension Gold Digger by Mountain Mods. Otherwise you're going to have to do something custom.

[Redicat]

How much lanes do you need for 4 way crossfire?

[riceusc2]

 

How much lanes do you need for 4 way crossfire?

I don't need crossfire exactly. Crossfire would only be used for gaming use but since this wont be used for gaming at all, i have no need for it. So I'm not sure if it would need the specific lane numbers.

 

 

To amend my previous, if you're wanting one board to push all 8 of those cards, you're going to be hard pressed to find one with only a 1000 USD budget. Most mainboards won't have enough PCI-Express slots. And I'm skeptical that any that do have that many can support that many graphics cards. The ATX specification typically allows for only 7 slots at most, though some 8-slot boards do exist. None that support DDR4 currently, though. X79 is the last chipset I've seen with boards that have 8 PCI-Express slots. Again, though, whether those boards can support those graphics cards and run stable? I'm doubtful.

 

It's only server boards that I've seen that can support more, and they're typically custom server boards. Otherwise you're looking at trying to run PICMG 1.3 system to support up to a 20-slot backplane. If any of the GPUs have blocks available that will allow you to swap the I/O plate to a single-slot plate, then you might be able to use those in a 14-slot backplane.

 

If you can get one of the X79 boards that has all 8 PCI-Express slots (with none of them being disabled based on where cards are plugged in), then you're going to need a chassis to hold all that. And the only one I know of is the Ascension Gold Digger by Mountain Mods. Otherwise you're going to have to do something custom.

With this case you suggested, it wouldn't be able to be used as a tank. Im trying to avid buying water cooler blocks for each of the GPUs which i would need to do if i used that type of case. What if i just went with the four 7990s?

[brandishwar]

I don't need crossfire exactly. Crossfire would only be used for gaming use but since this wont be used for gaming at all, i have no need for it. So I'm not sure if it would need the specific lane numbers.

 

With this case you suggested, it wouldn't be able to be used as a tank. Im trying to avid buying water cooler blocks for each of the GPUs which i would need to do if i used that type of case. What if i just went with the four 7990s?

Looking back at your original post, I recalled that you're considering mineral oil for doing this. I'll address that in a moment, but it does mean you won't have to worry about water blocks.

Your first hurdle, however, is figuring out how to get however many GPUs you want to run stable. 7990s are dual-GPU cards, so that would still be 8 GPUs the system would need to be able to power. I'm skeptical you'll be able to find a mainboard capable of doing that while remaining stable. I don't even know if the Linux AMD drivers will be able to account for that. For the 7970s, you should be able to find a board capable of running just those and remaining stable, since that would still be 4 discrete GPUs. But to power all 8 cards, 12 discrete GPUs total, from one mainboard.... again I'm skeptical you'll be able to do it, meaning you'll likely need two or three mainboards to run all 8 of those cards, with the 7970s on one, and two 7990s on the remaining two.

You'll be able to find better information through research into Bitcoin mining rigs. The 7970 and 7990 I know were the cards of choice for Bitcoin miners before ASICs started dominating the scene. This will give you an idea as to what you're going to require in order to set up something. A lot of them were running 3 or 4 discrete GPUs and remaining stable. On the 7990s, again, you may only be able to run 2 of them and be stable, but you might be able to run all 4. Whether you can will depend on the mainboard you select. Again research into Bitcoin mining is highly recommended, and see if you can find someone who was able to make use of more than 2 Radeon 7990s connected to 1 mainboard.

I do know that you will need PCI-Express extenders to connect all the cards, and I'd recommend the ones that use USB 3.0 cables instead of just being long ribbon cables. You won't see any detriment in performance doing that, but what it will give you is the ability to relocate the cards elsewhere away from the motherboard. This will also allow you to power the cards separate from the mainboard, allowing you to run the mainboard and primary storage off as little as a 250W power supply without any difficulty. The cards themselves will then be powered entirely by a separate high-wattage power supply. You'll likely need two power supplies, each powering 4 cards, just to ensure you have all the connectors you'll need.

Mineral oil will be a much more cost-effective way of cooling all of these cards since you won't need blocks -- those alone will likely add up to over 1000 USD combined depending on where you source them. And you'll really only need it for the cards themselves. A good AIO will allow the CPU(s) powering all of this to stay cool without a problem. If your super-computing setup won't be all that stressful on the CPU, the stock cooler should suffice, though a good, quiet after-market cooler would be beneficial to avoid adding more noise to this.

And then there are the radiators. At minimum you'll need 1 fan per discrete GPU, meaning a total of 12 fan spaces, 12x120mm minimum, or 12x140mm to really be safe.

Don't try to submerge all 8 in the same tank either. Again go only with 4 at a time, and depending on the oil temperature and the reported GPU temperatures, you may have to take it down to 2 cards per tank. On the 7970s, you should be good having all 4 in one tank, but the 7990s might get the oil a little too warm. This will depend heavily on the tank. You'll need to do some research to figure out what pumps to use and the overall way of setting up the tank. Your research might bring you across someone who had a mineral-oil cooled Bitcoin mining rig for all I know.

[riceusc2]

Looking back at your original post, I recalled that you're considering mineral oil for doing this. I'll address that in a moment, but it does mean you won't have to worry about water blocks.

Your first hurdle, however, is figuring out how to get however many GPUs you want to run stable. 7990s are dual-GPU cards, so that would still be 8 GPUs the system would need to be able to power. I'm skeptical you'll be able to find a mainboard capable of doing that while remaining stable. I don't even know if the Linux AMD drivers will be able to account for that. For the 7970s, you should be able to find a board capable of running just those and remaining stable, since that would still be 4 discrete GPUs. But to power all 8 cards, 12 discrete GPUs total, from one mainboard.... again I'm skeptical you'll be able to do it, meaning you'll likely need two or three mainboards to run all 8 of those cards, with the 7970s on one, and two 7990s on the remaining two.

You'll be able to find better information through research into Bitcoin mining rigs. The 7970 and 7990 I know were the cards of choice for Bitcoin miners before ASICs started dominating the scene. This will give you an idea as to what you're going to require in order to set up something. A lot of them were running 3 or 4 discrete GPUs and remaining stable. On the 7990s, again, you may only be able to run 2 of them and be stable, but you might be able to run all 4. Whether you can will depend on the mainboard you select. Again research into Bitcoin mining is highly recommended, and see if you can find someone who was able to make use of more than 2 Radeon 7990s connected to 1 mainboard.

I do know that you will need PCI-Express extenders to connect all the cards, and I'd recommend the ones that use USB 3.0 cables instead of just being long ribbon cables. You won't see any detriment in performance doing that, but what it will give you is the ability to relocate the cards elsewhere away from the motherboard. This will also allow you to power the cards separate from the mainboard, allowing you to run the mainboard and primary storage off as little as a 250W power supply without any difficulty. The cards themselves will then be powered entirely by a separate high-wattage power supply. You'll likely need two power supplies, each powering 4 cards, just to ensure you have all the connectors you'll need.

Mineral oil will be a much more cost-effective way of cooling all of these cards since you won't need blocks -- those alone will likely add up to over 1000 USD combined depending on where you source them. And you'll really only need it for the cards themselves. A good AIO will allow the CPU(s) powering all of this to stay cool without a problem. If your super-computing setup won't be all that stressful on the CPU, the stock cooler should suffice, though a good, quiet after-market cooler would be beneficial to avoid adding more noise to this.

And then there are the radiators. At minimum you'll need 1 fan per discrete GPU, meaning a total of 12 fan spaces, 12x120mm minimum, or 12x140mm to really be safe.

Don't try to submerge all 8 in the same tank either. Again go only with 4 at a time, and depending on the oil temperature and the reported GPU temperatures, you may have to take it down to 2 cards per tank. On the 7970s, you should be good having all 4 in one tank, but the 7990s might get the oil a little too warm. This will depend heavily on the tank. You'll need to do some research to figure out what pumps to use and the overall way of setting up the tank. Your research might bring you across someone who had a mineral-oil cooled Bitcoin mining rig for all I know.

Your exactly right, the primary original purpose of these cards was mining and thats why we purchased them (and they worked great until the US changed their policy on crypto). Anywho, excluding the primary factor of heat, is the reason for cutting down the number of 7990s to mainboard because of space on the board or board limitations? Because i was planning on dunking the GPUs without heat sinks at all and I'm not sure if thats going to make a difference. Again, I'm just trying to find out why you are recommending the PCI extenders/USB 3.0s rather than using the onboard PCI ports.

 

as for radiators, should i go with a true radiator system or a chilling system?

[Miguel Batista]

12 gpus in oneineral oil tank... Would you like fries with that sir? Seripusly, mineral oil has its limitations meaning if you put too much heat to it, it will keep that heat longer than air due to more limited circulation. I would go with a server board if I was youbut even then... 12 gpus is a lot of juice. You will have to custom build something and that is probably way North of 1000 usd.

Why don't you use a couple of GPUs tp start with (lile the four 7970s) and have it run stable. Then think about adding the rest. I mean even with a.splitter for the pci connections sp you would split a x16 connection into two 8x you would probably have stability problems... For your budget I dpn't see it happening.

[brandishwar]

Your exactly right, the primary original purpose of these cards was mining and thats why we purchased them (and they worked great until the US changed their policy on crypto). Anywho, excluding the primary factor of heat, is the reason for cutting down the number of 7990s to mainboard because of space on the board or board limitations? Because i was planning on dunking the GPUs without heat sinks at all and I'm not sure if thats going to make a difference. Again, I'm just trying to find out why you are recommending the PCI extenders/USB 3.0s rather than using the onboard PCI ports.

 

as for radiators, should i go with a true radiator system or a chilling system?

Go with radiators. Don't use a chilling system. I don't think they'll work well with mineral oil anyway. Plus with mineral oil, viscosity goes up as temperature goes down, so a chilling system could be problematic.

Board limitations are why I say to cut down the number of cards. You may find a board that will support all those GPUs and run them stable. Again researching Bitcoin mining rigs will be to your advantage on this. Like I said above, you're going to have difficulty finding a board that has 8 PCI-Express slots, let alone a board that can run GPUs in all of those slots and remain stable.

The extenders are for several reasons. First, you're not going to be able to plug all 8 of those graphics cards directly onto one ATX mainboard. It's not happening. It's impossible. The right board will support 4 double-wide cards, but then with air coolers they'll be choked off. Which brings me to the second reason: airflow.

Even with mineral oil cooling, you still need to have the cards spaced out. And the extenders allow you to do that. If you research Bitcoin mining rigs you'll find that virtually everyone who used more than two cards had PCI-Express extenders. Using extenders allowed for the cards to be spaced out better to run cooler. These are graphics cards, not supercomputing cards, so they're not tested for being run under a near 100% load, 24 hours a day, for weeks or months on end. This is one of the reasons why graphics cards that were used for any kind of supercomputing -- including crypto mining -- have a much, much lower value than if they were used only for gaming. Those cards, even if they were water cooled, are going to have a significantly lower life span.

The reason I'm specifically mentioning using the USB 3.0 extenders over ribbon cables is power isolation. There are several types of extenders. Riser cards are one, but immaterial to this discussion. Ribbon cables are another. And the third are data-cable extenders. On the data cable extenders I've seen two types: USB 3.0 and SATA. On the SATA extender, I've only company make one: ASRock. I don't recommend it for this application. It uses two SATA cables plus has a proprietary power connector.

USB 3.0 extenders come in two parts. You have a small PCI-Express x1 module that plugs into the mainboard, and a PCI-Express x16 receiver that the graphics card sits into. And the data cable in this case is a USB 3.0 type A male to male. The x1 channel won't inhibit communication with the card, so don't worry about that. Anyway, the receiver is separately powered, so you'll have power connectors for the graphics card, and power connectors for the receiver. This is in part because the graphics card expects to receive some of its power through the PCI-Express slot. So what that'll allow, as I've mentioned, is to use one low-wattage power supply to power the mainboard and primary storage, while using separate high-wattage PSUs to power the graphics cards and the receiver side of the extenders.

I hope that makes sense....

To give you an idea of what I'm talking about here, check out the link in my signature "Berkeley (BOINC) setup". In that, I have several graphics cards connected to an older mainboard via USB 3.0 PCI-Express extenders. The graphics cards are powered separately from the host system, but the host system recognizes the cards and is able to use them.