Which GPU for LLMs? (entry-level)

[OliOliOli]

I'm looking to upgrade my GPU, when i purchased my current one all i played was rocket league, it has served its purpose well.

I am now getting into various bits of development and want to start working with LLMs locally as they become available, currently i am looking at stable diffusion as an entry point - but as with anything once you fall down the rabbit hole, who knows where you'll end up.

 

My current system is AMD, so not ideal for the task to say the least. I figure a GPU upgrade, and if needed a new PSU would allow me to start working with these tools, but most are built around nvidia & cuda - so time to begrudgingly go back to team green.

 

Question is with nvidia hating its customers so much, and from what I took from the latest LTT video the 40 class cards are not good value for money, what IS a good purchase? Especially for LLMs rather than gaming?

Should i go for a 4070 / 4070ti ? I think my budget could stretch to £550 if i really had to, but it still feels excessive - im thinking about 40 series cause i figured power efficiency and latest generations of stuff will get me an extra year or two out of the card over buying a 30 series in the hope of it being cheaper?

Is a 4070 massively overkill or still horrible value for money?

Would a 30 series card be a better use of money or would it be just buying old power inefficient h/w that we've moved on from?

 

My current setup:

Board: Micro-Star International Co., Ltd. MAG B550 TOMAHAWK (MS-7C91) 2.0
AMD Radeon RX 5600 XT 

3.60 gigahertz AMD Ryzen 7 3700X 8-Core

48 Gb DD4 RAM

650w Seasonic PSU

 

 

[SorryBella]

10 minutes ago, OliOliOli said:

currently i am looking at stable diffusion as an entry point - but as with anything once you fall down the rabbit hole, who knows where you'll end up.

 

My current system is AMD, so not ideal for the task to say the least.

It should be fine for just straight stab diff but you do need to find one fork with openCL support.

 

For 550GBP id consider the RTX 3090 on the used market. It is competing with RTX 4080 in ML but instead of just 16GB of VRAM, you get a pistol whipping 24GB instead which would be a GIANT boon for high resolution stab diff or running older chatGPT model. But of course, youll need a PSU upgrade to 750 or 850W unit, and your power consumption isnt something to be bragged about.

[OliOliOli]

4 minutes ago, SorryBella said:

your power consumption isnt something to be bragged about.

100%!

 

Appreciate the response, this is what is nagging in the back of my head - is the 3090 really worth the added power consumption though?

Power isn't cheap in the UK

[SorryBella]

9 minutes ago, OliOliOli said:

 

Appreciate the response, this is what is nagging in the back of my head - is the 3090 really worth the added power consumption though?

Power isn't cheap in the UK

With power limit and undervolts it can be worthwhile to do and would be more efficient than what reviewers get. @igormp did it on their 3090.

For gaming its less desirable to limit its power (undervolt is fine but in my experience they dont make a dent in power consumption most of the time) but for ML, its a boon.

[igormp]

1 hour ago, SorryBella said:

But of course, youll need a PSU upgrade to 750 or 850W unit, and your power consumption isnt something to be bragged about.

I just bought an 850W... So I can run 2x3090 lol

 

Power limit each of those to 275W, 550W total + ~200W for the rest of the setup and I still have 100W or so to spare.

 

1 hour ago, OliOliOli said:

Would a 30 series card be a better use of money or would it be just buying old power inefficient h/w that we've moved on from?

They didn't turn into shit just because the 40 series became a thing haha

While the 4000 series is indeed more power efficient, by the end of the day you'll still be getting a best bang for the buck, and more perf per time with a 3090 rather than a 4080. The savings you'd get in power consumption are negligible IMO.

 

Anyhow, it still depends on how you used market looks like and if you're comfortable with it. I'd say that the best GPUs for ML are, in this order: 4090, 3090(ti), 4080, 4070ti, 3080 (either ti or just the 12gb model), 3060 12gb. For any other model with 10gb of vram or less, I'd say to just pick what does best at your gaming needs instead, you'll be limited by the vram at this point.