Build Check and Advice for Local LLM machine for work. Down a Rabbit Hole

[Bear_99]

I've been watching LTT for a long time and figure might as well plea here for people's advice and recommendations. Thanks!

Budget (including currency): $4000

Country: US.  Can only buy from B&H, NewEgg, Amazon(if required)

Games, programs or workloads that it will be used for: Running local LLM

Other details Was given $4000 budget to build a Local LLM machine. Plan is to buy 1 4090 now and then buy a second one later. 

Problems and questions
Is my Motherboard Good enough to support 2? I think my CPU has 24. The GPU's will eat 16 so we should be ok?

So Far I've seen

2x4090's is a waste because we would lose NVLink 

 

Stuck between these two Mobo's

Asus ProArt B650-Creator $250

Asus ProArt X650e - $425

PSU

some concerns for the new 4090's catching fire due to the 12vhpwr. I saw that seasonic will be releasing some newer cables for ATX 12V-2x6 Cables. I assume this will resolve that fear? Will it work with the PSU I picked?

SeaSonic PRIME TX-1600 ATX 3.0 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply

My build so far

PCPartPicker Part List

CPU: AMD Ryzen 9 7900 3.6 GHz 12-Core Processor  ($369.00 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: SeaSonic PRIME TX-1600 ATX 3.0 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply  ($568.12 @ Amazon) 
Total: $4004.34
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:18 EDT-0400

[igormp]

38 minutes ago, Bear_99 said:

Is my Motherboard Good enough to support 2? I think my CPU has 24. The GPU's will eat 16 so we should be ok?

 

Yes. 2 GPUs will give you x8/x8 each, which is not really that relevant for inference on the small models you can run on it anyway.

38 minutes ago, Bear_99 said:

2x4090's is a waste because we would lose NVLink 

NVLink is irrelevant for inference if you're working with larger models that require more than 24GB of vram. If you plan on fine tuning models then you could go for 2x3090s + nvlink instead, but I don't think the performance would be that different.

39 minutes ago, Bear_99 said:

Stuck between these two Mobo's

Asus ProArt B650-Creator $250

Asus ProArt X650- $425

Depends on if you care about PCIe 5.0 (irrelevant IMO), if you need the extra NVMe slot, or if you need 10GbE. I guess that's the most relevant difference. I'd personally go with the cheaper B650 option.

41 minutes ago, Bear_99 said:

SeaSonic PRIME TX-1600 ATX 3.0 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply

Pretty overkill, but you do you.

[PDifolco]

51 minutes ago, Bear_99 said:

I've been watching LTT for a long time and figure might as well plea here for people's advice and recommendations. Thanks!

Budget (including currency): $4000

Country: US.  Can only buy from B&H, NewEgg, Amazon(if required)

Games, programs or workloads that it will be used for: Running local LLM

Other details Was given $4000 budget to build a Local LLM machine. Plan is to buy 1 4090 now and then buy a second one later. 

Problems and questions
Is my Motherboard Good enough to support 2? I think my CPU has 24. The GPU's will eat 16 so we should be ok?

So Far I've seen

2x4090's is a waste because we would lose NVLink 

 

Stuck between these two Mobo's

Asus ProArt B650-Creator $250

Asus ProArt X650- $425

PSU

some concerns for the new 4090's catching fire due to the 12vhpwr. I saw that seasonic will be releasing some newer cables for ATX 12V-2x6 Cables. I assume this will resolve that fear? Will it work with the PSU I picked?

SeaSonic PRIME TX-1600 ATX 3.0 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply

My build so far

https://pcpartpicker.com/list/czmZyg

The B650 board will be fine, 2 GPUs will run at x8 on a Creator

I'd make some changes, get a beefier CPU, and no need to put that much money in the SSD, case and PSU

 

Edit : changed the case as you need room for 2x360 eventually

 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon)
CPU Cooler: Thermalright Phantom Spirit EVO 69 CFM CPU Cooler  ($49.90 @ Amazon)
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H)
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon)
Storage: TEAMGROUP MP44 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($233.99 @ Amazon)
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg)
Case: Phanteks ECLIPSE G500A DRGB ATX Mid Tower Case  ($129.99 @ Amazon)
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg)
Total: $3644.61
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:18 EDT-0400

 

[Bear_99]

5 minutes ago, igormp said:

Yes. 2 GPUs will give you x8/x8 each, which is not really that relevant for inference on the small models you can run on it anyway.

NVLink is irrelevant for inference if you're working with larger models that require more than 24GB of vram. If you plan on fine tuning models then you could go for 2x3090s + nvlink instead, but I don't think the performance would be that different.

Depends on if you care about PCIe 5.0 (irrelevant IMO), if you need the extra NVMe slot, or if you need 10GbE. I guess that's the most relevant difference. I'd personally go with the cheaper B650 option.

Pretty overkill, but you do you.

Thanks for your input and advice.
Yea went with the Season TX-1600 because of potential 2 4090's later on or 2 5090's when that comes out. Whats the deal with the cable? 12V-2x6 

I plan to buy the MSI Liquid 4090 to save some space in between
I have 1x 4tb Samsung 990. Maybe plan to buy another one down the road

[igormp]

Just now, Bear_99 said:

Yea went with the Season TX-1600 because of potential 2 4090's later on or 2 5090's when that comes out.

I still believe that's overkill. I run my 2x3090 with a 850W PSU. There's no need to crank both GPUs to their max PL since you're losing on efficiency, but I guess to each their own.

1 minute ago, Bear_99 said:

Whats the deal with the cable? 12V-2x6 

I'm not really into those power cable shenanigans, but I'd just use the adapter that comes with the GPU and let it be. Worst comes your manufacturer should be responsible for the warranty.

2 minutes ago, Bear_99 said:

I plan to buy the MSI Liquid 4090 to save some space in between

Good plan.

 

Oh, one detail that I missed, if you ever plan on getting another 2x48GB later (which you probably will if you get another 4090), you will likely need to downclock those sticks to 5200~5600MHz, getting 4x high density sticks to run higher than that is really troublesome. If I were you, I'd just go with 2x48gb at 5600MHz already in order to avoid headaches in the future, the performance difference is insignificant and having stuff running in a stable manner is more important IMO.

[Bear_99]

5 minutes ago, PDifolco said:

The B650 board will be fine, 2 GPUs will run at x8 on a Creator

I'd make some changes, get a beefier CPU, and no need to put that much money in the SSD, case and PSU

 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon)
CPU Cooler: Thermalright Phantom Spirit EVO 69 CFM CPU Cooler  ($49.90 @ Amazon)
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H)
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon)
Storage: TEAMGROUP MP44 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($233.99 @ Amazon)
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg)
Case: Phanteks Eclipse G360A ATX Mid Tower Case  ($120.33 @ Amazon)
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg)
Total: $3634.95
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:16 EDT-0400

 

Wow thanks! 

For your Corsair HX1500i i see that it comes wiht the newer PCIe 5.1 12V-2x6  I assume this is safer? I swapped out a bit with your PSU instead.

 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg) 
Total: $3818.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:22 EDT-0400

[brob]

There is no ProArt X650 motherboard. Did you mean the ProArt X670-E?

 

Most of the problems with the 12VHPWR connector have been attributed to improper connector seating. As long as the connection is carefully done it shouldn't be a worry.

 

Two RTX 4090 GPU do not require 1600W. A 1200W - 1300W PSU would be sufficient. 

 

80+ Titanium are rarely worth the price premium. Use a good 80+ Gold model.

 

[PDifolco]

1 minute ago, Bear_99 said:

Wow thanks! 

For your Corsair HX1500i i see that it comes wiht the newer PCIe 5.1 12V-2x6  I assume this is safer? I swapped out a bit with your PSU instead.

 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg) 
Total: $3818.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:22 EDT-0400

I suppose the latest produced PSU will have the new connectors indeed

You could make easy money with non overpriced SSD like the 990 is, do your work involves transferring petabytes of sequential data ?

 

[Bear_99]

2 minutes ago, igormp said:

I still believe that's overkill. I run my 2x3090 with a 850W PSU. There's no need to crank both GPUs to their max PL since you're losing on efficiency, but I guess to each their own.

I'm not really into those power cable shenanigans, but I'd just use the adapter that comes with the GPU and let it be. Worst comes your manufacturer should be responsible for the warranty.

Good plan.

 

Oh, one detail that I missed, if you ever plan on getting another 2x48GB later (which you probably will if you get another 4090), you will likely need to downclock those sticks to 5200~5600MHz, getting 4x high density sticks to run higher than that is really troublesome. If I were you, I'd just go with 2x48gb at 5600MHz already in order to avoid headaches in the future, the performance difference is insignificant and having stuff running in a stable manner is more important IMO.

Thanks! Yep I went down to a  Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply to save a bit of money then buffing up to the 79050x

 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg) 
Total: $3818.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:28 EDT-0400

[PDifolco]

1 minute ago, brob said:

There is no ProArt X650 motherboard. Did you mean the ProArt X670-E?

 

https://www.asus.com/fr/motherboards-components/motherboards/proart/proart-b650-creator/

 

[Bear_99]

5 minutes ago, brob said:

There is no ProArt X650 motherboard. Did you mean the ProArt X670-E?

 

Most of the problems with the 12VHPWR connector have been attributed to improper connector seating. As long as the connection is carefully done it shouldn't be a worry.

 

Two RTX 4090 GPU do not require 1600W. A 1200W - 1300W PSU would be sufficient. 

 

80+ Titanium are rarely worth the price premium. Use a good 80+ Gold model.

 

Yep the Asus Proart x670e

4 minutes ago, PDifolco said:

I suppose the latest produced PSU will have the new connectors indeed

You could make easy money with non overpriced SSD like the 990 is, do your work involves transferring petabytes of sequential data ?

 

Yea we have a strange discount on the Samsung ones through our partner reseller. 

Final build seems to be is this correct?
 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg) 
Total: $3818.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:28 EDT-0400

[brob]

3 minutes ago, PDifolco said:

https://www.asus.com/fr/motherboards-components/motherboards/proart/proart-b650-creator/

 

 

1 hour ago, Bear_99 said:

Stuck between these two Mobo's

Asus ProArt B650-Creator $250

Asus ProArt X650e - $425

 

[PDifolco]

1 minute ago, Bear_99 said:

Yep the Asus Proart x670e

Yea we have a strange discount on the Samsung ones through our partner reseller. 

Final build seems to be is this correct?
 

PCPartPicker Part List

CPU: AMD Ryzen 9 7950X 4.5 GHz 16-Core Processor  ($501.76 @ Amazon) 
CPU Cooler: be quiet! Dark Rock Pro 5 CPU Cooler  ($79.90 @ Amazon) 
Motherboard: Asus PROART B650-CREATOR ATX AM5 Motherboard  ($229.99 @ B&H) 
Memory: Corsair Vengeance RGB 96 GB (2 x 48 GB) DDR5-6000 CL30 Memory  ($349.99 @ Amazon) 
Storage: Samsung 990 Pro 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive  ($318.35 @ Amazon) 
Video Card: MSI SUPRIM LIQUID X GeForce RTX 4090 24 GB Video Card  ($1899.00 @ Newegg) 
Case: Corsair 7000D AIRFLOW ATX Full Tower Case  ($189.99 @ Newegg) 
Power Supply: Corsair HX1500i (2023) 1500 W 80+ Platinum Certified Fully Modular ATX Power Supply  ($249.99 @ Newegg) 
Total: $3818.97
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 14:28 EDT-0400

The BQ DR5 is more expensive and less effective on a Zen4 CPU than a TR PS120, nuff said

[Bear_99]

2 hours ago, PDifolco said:

The BQ DR5 is more expensive and less effective on a Zen4 CPU than a TR PS120, nuff said

Ah yes was reading reviews on that. Do you have another CPU Cooler you recommend. I dont know why the TR PS120 is glitched on our ordering portal at work. I can buy the trps120 myself  what fans do you recommend swapping with?

 

Any recommendations on buying more fans as well? or anything else i might missing like fan controllers.

 

Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2024-05-18 16:40 EDT-0400

For the case I picked the 7000d would it work for 2x the MSI Liquid 4090? I assume we slap it on the front two in the future.

[igormp]

29 minutes ago, TheDarkCanuck said:

You can save a lot of money on the processor. Multi-threaded/core performance does not seem to be relevant here. 

For models that are bigger than the vram size, the cpu becomes relevant since you're offloading only some layers to the gpu, the rest still runs on the cpu.

30 minutes ago, TheDarkCanuck said:

Are you also going to try inference with the cpu or something? Otherwise you could save money on the RAM as well. Unless you have other plans for this box.

It's not a good idea to go with less ram, unless you mean getting slower ones at the same capacity.

[igormp]

2 hours ago, TheDarkCanuck said:

It's still single threaded, no? Seems like there is a lot of performance (and $) being left on the table as even i7 14 (even 13!) gen has better performance for this workload, eg. https://www.cpubenchmark.net/singleThread.html

 

Good to know about the ram. 

No, it's embarrassingly multi threaded.