What is the Max number of GPUs in 1 system for a GPGPU?

[arisgardelis]

Hello, I am preparing my GPGPU machine.

 The components so far are:

ASRock Extreme 11

2x Asus ARES 5870x2

2x Sapphire 5970 Toxic

2x XFX 5970 Black Edition Limited

4x 5970

8x Samsung 840 PRO 256GB

2x Enermax Platimax 1200w

2x 8GB 1600MHz DDR3 ECC REG

1x Xeon E5-2665

 

I am using the ATI 5xxx platform, because the programs that I will use are written in BROOK and since ATI has dropped support from the 7xxx series, there's no other option. 6xxx platform has fewer stream cores, so no point to bother.

 

The question I want to ask is, how many GPUs a system can handle? The scenario is to use all the 6 beefy 5970 and one normal 5970, but has anyone any experience working with a configuration like this? Don't really want to burn something or experiment with this equipment, so any help would be appreciated.

 

Thanks.

[mvitkun]

I don't think you could run that without using atleast another 2x 1200 watt.

the 5870x2/5970 takes 313 watts during gaming and 436 watts during furmark,GPGPU would probably take it closer to the furmark value.

 

and you have 7x 5970's.

 

you'd need somewhere  around 3000 watts to run that system at full GPGPU load.

[ionbasa]

Also I believe the AMD drivers only support up to 7 Cards on a system, unless you use some sort of hardware virtualization like zen + linux + hardware pass-through. Ymmv.

[arisgardelis]

I don't think you could run that without using atleast another 2x 1200 watt.

the 5870x2/5970 takes 313 watts during gaming and 436 watts during furmark,GPGPU would probably take it closer to the furmark value.

 

and you have 7x 5970's.

 

you'd need somewhere  around 3000 watts to run that system at full GPGPU load.

3rd Enermax is scheduled to be purchased soon.

[arisgardelis]

Also I believe the AMD drivers only support up to 7 Cards on a system, unless you use some sort of hardware virtualization like zen + linux + hardware pass-through. Ymmv.

Can you please be more specific? When you mean 7 cards, you mean 7 GPUs or 7 dual GPU cards?

[ionbasa]

See here: https://bitcointalk.org/index.php?topic=51280.0

and http://devgurus.amd.com/thread/158863

 

There is no straight forward answer, but it seems to be a bios limitation, each GPU takes up ~256 System memory on bootup. So with your current config you would have 12 Gpu cores.

12 Cores * 256 Mb = 3Gb of memory.

 

I believe that it would spear that your system would have to have EFI even to just get it to boot. (the asrock board doesn't have Uefi i believe, only BIOS).

[Dsgreek]

What is GPGPU??

[JORDANonPC]

I don't think you could run that without using atleast another 2x 1200 watt.

the 5870x2/5970 takes 313 watts during gaming and 436 watts during furmark,GPGPU would probably take it closer to the furmark value.

 

and you have 7x 5970's.

 

you'd need somewhere  around 3000 watts to run that system at full GPGPU load.

I might have missed something but where did you get 7 from because I thought he had 10 cards there and 20 GPUs

[mvitkun]

I might have missed something but where did you get 7 from because I thought he had 10 cards there and 20 GPUs

reread his post.

 

The scenario is to use all the 6 beefy 5970 and one normal 5970

and while there are 2 5870 gpu's per 5970,and he has 10 5970's, I used the power consumption for a 5970.

so it would be (max power consumption of 5970 x 7) + power consumption of CPU + power consumption of ___  = max power consumption.

 

so (436 x 7) + 115 [cpu] + 24 [sSDs] = ~3191 absolute max

[piplupgao]

All I can say is have fun with your power bill

[Godlygamer23]

What is GPGPU??

General Purpose Graphics Processing Unit.

[JORDANonPC]

reread his post.

and while there are 2 5870 gpu's per 5970,and he has 10 5970's, I used the power consumption for a 5970.

so it would be (max power consumption of 5970 x 7) + power consumption of CPU + power consumption of ___  = max power consumption.

 

so (436 x 7) + 115 [cpu] + 24 [sSDs] = ~3191 absolute max

thanks for that I just kept reading the part about different 5970s he had :p my bad there

[arisgardelis]

See here: https://bitcointalk.org/index.php?topic=51280.0

and http://devgurus.amd.com/thread/158863

 

There is no straight forward answer, but it seems to be a bios limitation, each GPU takes up ~256 System memory on bootup. So with your current config you would have 12 Gpu cores.

12 Cores * 256 Mb = 3Gb of memory.

 

I believe that it would spear that your system would have to have EFI even to just get it to boot. (the asrock board doesn't have Uefi i believe, only BIOS).

The motherboard is ASRock X79 Extreme 11. It has UEFI. Although this is a cutting edge motherboard, there are reports for lot's of RMAs. I think it has hit me also, because the LSI controller is dissapeared when linking with PCIE 3.0, though it works normal as linked with PCIE 2.0. Actually, there is no competition when you want max transfer rates  (4GB/s) and max IOPS.

I'll check those links, but I hoped to find someone who's experienced with this kind of setup.

By the way, an Enermax Platimax 1500 is on the way, so replying also to the latest messages, power is not an issue. I just wanted to find someone with solid experience in this setup.

Thanks anyway.

[ionbasa]

Great, as long as you have enough RAM on the ASRock board you should be fine, although some UEFI implementations in the GPU are hacked together to support the UEFI, so it may or may not work as planned.

 

Basically look if the card manufactures pushed a BOIS update for the GPUs, one such examle is this: http://rog.asus.com/forum/showthread.php?30670-GOP-UEFI-BIOS-Available-for-ASUS-GeForce-GTX-680-Series-Graphics-Cards

 

Where the card will then have a UEFI hybrid bios, if your cards do not have this it may not work, because in order to address that many address spaces in the GPUs you need a 64 bit boot loader (UEFI)