Budget Build for deploy Large-Language Model locally?

Alkaid · July 28, 2023

Budget (including currency): ideally $500, and the maximum is $1500.

Country: USA

Games, programs or workloads that it will be used for: LLM

I want to deploy a local chatbot, which requires a large-ram (my target is 24GB) Nvidia GPU. My main desktop is an ITX build with Arc A770 GPU, which is not an ideal GPU for this task and has no space for a second GPU.

So I either

(1) swap the current GPU with an RTX3090/4090/4060Ti

Because I need to run many daily tasks on my main desktop, I require some gaming performance and stability, which implies a relatively new GeForce GPU. But new large-ram GeForce GPUs are pretty expensive. Only 3090 (Ti)and 4090 have 24GB memory. 4090 is just expensive. A second-hand 3090 is about $700 now, but I do worry about its reliability considering it might have been used for mining. I've had a bad experience with second-hand card before, so I am hesitant to plug a second-hand card into my main machine.

4060ti 16GB is also on the table if the 24GB plan is not very achievable.

OR

(2) build a dedicated PC with Nvidia Tesla P40

The benefit of this is that I can use Nvidia Tesla P40, the cheapest 24GB Nvidia card. ( about $200) This card does not have video output and is very old (unreliable), but totally fine for AI usage. The issue is I need to buy a full set of CPU, MB, RAM, PSU, etc. I used 10 minutes and was able to find i7-9700+32GB ram+mb+psu+case for about $400. Is there a lot of room for price optimization? And I am also worried about the reliability of Tesla P40, a card released in 2016, nearly a decade ago. I am not a big fan of the second-hand market, and my first DIY build was Intel 10th Gen, so I have very limited experience with old hardware.

Which is better? Or is there any other plan I should look for?

jaslion · July 28, 2023

Used 3090 + psu upgrade. The performance of that tesla is crap and the vram amount wont make up for that. You also need to make a custom fan for it because ti DOES NOT have any way to cool itself. It is designed to be in a server airflow config.

Rajivrocks · July 28, 2023

Yeah I'd agree with the guy above me, go with the second hand 3090 (TI) you'll save a lot of money and the Tesla is really only an option for a server and the performance is probably worse anyway.