Building GPU server for AI Training

[ajmal]

Hello guys i dont know if this right area for posting this but i am coder and i am trying to build new 5 gpu server for my AI work but i am pretty noob at hardware so i need recommendations and help in the build

what hardware should i choose and what things i need i have budget of up to 10k usd so if any one can take few minutes of their time to help me out

[Electronics Wizardy]

do you want rack mount or tower?

 

Getting 5 gpus in a tower is hard, and boards arevery rare for this setup/

 

Id probably get a used rckmount server with 8x gpu slots, and fill it with the gpus you want

[ajmal]

 

2 minutes ago, Electronics Wizardy said:

do you want rack mount or tower?

 

Getting 5 gpus in a tower is hard, and boards arevery rare for this setup/

 

Id probably get a used rckmount server with 8x gpu slots, and fill it with the gpus you want

rackmount is better to get for this setup which i am planning can u point me to some good direction for that

[manikyath]

your easiest bet would be one of supermicro's 8-GPU options, the only downside there (besides cost..) is the ridiculous boot time.

 

that said, finding a consumer board to fit 8 GPU's is a nightmare in itself, and given it is a corporate investment it's in itself a value that you can have an external party do all the fiddling for you. (most supermicro resellers are also system integrators)

 

that said.. if you're doing this out of your own pocket, drop that down to 4 GPU's, and go with some top tier consumer ("prosumer"?) hardware

[Electronics Wizardy]

Just now, ajmal said:

 

rackmount is better to get for this setup which i am planning can u point me to some good direction for that

Supermicro has a lot of gpu servers like this one. https://www.thinkmate.com/system/superserver-4029gp-trt

 

WIth a 10k budget, id get a used one like this one on ebay.  https://www.ebay.com/itm/SuperMicro-SYS-4028GR-TR2-Server-Up-to-10x-GPU-Incl-4x-PSU-2x-Xeon-Processors/324068410649?hash=item4b73fb8519:g:onkAAOSwgo9dyjUg. THen fill it with gpus.

 

Just saying, there are very loud, and get get very power hungry(to the point that you can run off a normal hose outlet_

[ajmal]

1 minute ago, manikyath said:

your easiest bet would be one of supermicro's 8-GPU options, the only downside there (besides cost..) is the ridiculous boot time.

 

that said, finding a consumer board to fit 8 GPU's is a nightmare in itself, and given it is a corporate investment it's in itself a value that you can have an external party do all the fiddling for you. (most supermicro resellers are also system integrators)

 

that said.. if you're doing this out of your own pocket, drop that down to 4 GPU's, and go with some top tier consumer ("prosumer"?) hardware

its not corporate investment i am doing it from my own savings what do u suggest for "prosumer" hardware

[manikyath]

Just now, Electronics Wizardy said:

Supermicro has a lot of gpu servers like this one. https://www.thinkmate.com/system/superserver-4029gp-trt

 

WIth a 10k budget, id get a used one like this one on ebay.  https://www.ebay.com/itm/SuperMicro-SYS-4028GR-TR2-Server-Up-to-10x-GPU-Incl-4x-PSU-2x-Xeon-Processors/324068410649?hash=item4b73fb8519:g:onkAAOSwgo9dyjUg. THen fill it with gpus.

 

Just saying, there are very loud, and get get very power hungry(to the point that you can run off a normal hose outlet_

going with a used box actually wont be a bad idea indeed, backing this idea.

[ajmal]

1 minute ago, Electronics Wizardy said:

Supermicro has a lot of gpu servers like this one. https://www.thinkmate.com/system/superserver-4029gp-trt

 

WIth a 10k budget, id get a used one like this one on ebay.  https://www.ebay.com/itm/SuperMicro-SYS-4028GR-TR2-Server-Up-to-10x-GPU-Incl-4x-PSU-2x-Xeon-Processors/324068410649?hash=item4b73fb8519:g:onkAAOSwgo9dyjUg. THen fill it with gpus.

 

Just saying, there are very loud, and get get very power hungry(to the point that you can run off a normal hose outlet_

i looked in supermicro alot but they are really power hungry things would not fit for home setup

[manikyath]

Just now, ajmal said:

its not corporate investment i am doing it from my own savings what do u suggest for "prosumer" hardware

well, plenty of "gamer" motherboards that can fit 4 GPU's, 5 on the other hand may be problematic.

 

for example, this board could do the trick:

 

all in all, going this route easily cuts a few K usd off the total cost.

[Electronics Wizardy]

1 minute ago, ajmal said:

i looked in supermicro alot but they are really power hungry things would not fit for home setup

If you want a lot of gpus, its gonna be very power hungry no matter what you do. 5x high end gpus is about 1500w + other parts and psu, so thats beyond a normal outlets limit.

 

If you can deal with 4 gpus, a desktop should be ok .

 

How much cpu power, ram and bandwidth between gpus do you need?

[ajmal]

3 minutes ago, Electronics Wizardy said:

If you want a lot of gpus, its gonna be very power hungry no matter what you do. 5x high end gpus is about 1500w + other parts and psu, so thats beyond a normal outlets limit.

 

If you can deal with 4 gpus, a desktop should be ok .

 

How much cpu power, ram and bandwidth between gpus do you need?

i am planning to have around 8gb gpu cards and for cpu i am planing like a good server grade cpu like xeon but dont know what consumer grade mother support that type of cpu and ram would aound 128gb

[manikyath]

13 minutes ago, ajmal said:

i am planning to have around 8gb gpu cards and for cpu i am planing like a good server grade cpu like xeon but dont know what consumer grade mother support that type of cpu and ram would aound 128gb

dont take this for a full answer, but this would be a (very) rough draft of the sort of system i'd suggest you go with, if it's out of your own pocket:

 

PCPartPicker Part List

CPU: AMD Threadripper 3960X 3.8 GHz 24-Core Processor  ($1399.00 @ Best Buy) 
CPU Cooler: Noctua NH-U14S TR4-SP3 82.52 CFM CPU Cooler 
Motherboard: ASRock TRX40 Creator ATX sTRX4 Motherboard  ($429.99 @ Newegg) 
Memory: Corsair Vengeance LPX 128 GB (4 x 32 GB) DDR4-3600 Memory  ($699.99 @ Newegg) 
Storage: Samsung 970 Pro 1 TB M.2-2280 NVME Solid State Drive  ($349.99 @ B&H) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Video Card: EVGA GeForce RTX 2080 SUPER 8 GB GAMING Video Card (2-Way SLI)  ($703.98 @ Newegg) 
Video Card: EVGA GeForce RTX 2080 SUPER 8 GB GAMING Video Card (2-Way SLI)  ($703.98 @ Newegg) 
Case: Fractal Design Define XL R2 (Titanium Grey) ATX Full Tower Case  ($151.98 @ Newegg) 
Power Supply: Corsair 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply  ($549.99 @ Newegg) 
Total: $5378.87
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2020-03-24 16:07 EDT-0400

 

some notes:

- the storage is "placeholder"

- it only let me add 2 GPU's

- keep in mind that if you're packing a system full of graphics cards, you're best to get "blower" style cards, so that they help with total system airflow, rather than just swim in their own heat.

- i've added a 1600W power supply, that is a NECESSITY for this level of system.

- i've gone for amd EPYC because most of the boards on that side have MANY pcie options, if your heart is with intel there's options on that side as well.

- if this is just a hobby thing, and wont directly relate to revenue, this level of system is absolutely ridiculous.

 

(EDIT: fixed paste of pcpp)

[ajmal]

3 minutes ago, manikyath said:

dont take this for a full answer, but this would be a (very) rough draft of the sort of system i'd suggest you go with, if it's out of your own pocket:

 

PCPartPicker Part List

CPU: AMD Threadripper 3960X 3.8 GHz 24-Core Processor  ($1399.00 @ Best Buy) 
CPU Cooler: Noctua NH-U14S TR4-SP3 82.52 CFM CPU Cooler 
Motherboard: ASRock TRX40 Creator ATX sTRX4 Motherboard  ($429.99 @ Newegg) 
Memory: Corsair Vengeance LPX 128 GB (4 x 32 GB) DDR4-3600 Memory  ($699.99 @ Newegg) 
Storage: Samsung 970 Pro 1 TB M.2-2280 NVME Solid State Drive  ($349.99 @ B&H) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Storage: Toshiba X300 4 TB 3.5" 7200RPM Internal Hard Drive  ($129.99 @ Best Buy) 
Video Card: EVGA GeForce RTX 2080 SUPER 8 GB GAMING Video Card (2-Way SLI)  ($703.98 @ Newegg) 
Video Card: EVGA GeForce RTX 2080 SUPER 8 GB GAMING Video Card (2-Way SLI)  ($703.98 @ Newegg) 
Case: Fractal Design Define XL R2 (Titanium Grey) ATX Full Tower Case  ($151.98 @ Newegg) 
Power Supply: Corsair 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply  ($549.99 @ Newegg) 
Total: $5378.87
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2020-03-24 16:07 EDT-0400

 

some notes:

- the storage is "placeholder"

- it only let me add 2 GPU's

- keep in mind that if you're packing a system full of graphics cards, you're best to get "blower" style cards, so that they help with total system airflow, rather than just swim in their own heat.

- i've added a 1600W power supply, that is a NECESSITY for this level of system.

- i've gone for amd EPYC because most of the boards on that side have MANY pcie options, if your heart is with intel there's options on that side as well.

- if this is just a hobby thing, and wont directly relate to revenue, this level of system is absolutely ridiculous.

 

(EDIT: fixed paste of pcpp)

thanks for such good sheet and yeah this is not hobby thing i need system for real AI and 3d work so this would help me out and generate good revenue as well for me

[igormp]

5 hours ago, ajmal said:

thanks for such good sheet and yeah this is not hobby thing i need system for real AI and 3d work so this would help me out and generate good revenue as well for me

Do you have any kind of idea on what kind of models you'll run? Maybe going for some 2080 TIs would be better due to the extra vram. 8g isn't that useful for bleeding edge stuff nowadays.