Which old AMD GPU should I get for Local LLMs on an eGPU?

[ImLuciano]

TLDR: I want to buy a used AMD GPU for AI. I don't know which is best to buy or how to compare cards. I am also limited to PCIe 3.0 in an external chassis

I'm currently building a new pc and I have an external GPU chassis that I want to put a card in for dedicated AI loads. I am using AMD GPUs in my pc so I would like an AMD gpu in the eGPU as well. Most of my local AI I am currently using is used for DND NPCs and keeping an output speed like an IRL conversation is important to me. I realize I can just change the AI parameters, but I want to maximize value with the card I pick. I'm unsure which card to get for a few reasons:
1. Since this is an external GPU Chassis, does that mean the speed will be too slow for simulating conversations? I used to use the eGPU for video games on a laptop and didn't have issues with the information transferred over a cable so I'm assuming it won't be an issue, but I would love to know more. 

 

2. The GPU Chassis is PCIe 3.0, I saw some reddit posts saying that cards with PCIe 4.0 or higher will work in PCIe 3.0. But since I care mostly about the compute speed, does this mean a PCIe 4.0 card will just run the same as a PCIe 3.0 card for local AIs? Is getting a PCIe 4.0 or higher card just a waste of money for me?

 

3. I have no idea how to compare AMD cards for AI. I see a lot of AMD Instinct cards on ebay for about $50-$100 (I have no problem with used cards). I don't know how these would compare to getting an old GPU for a similar price. If this was an Nvidia card, I could find graphs with Tokens/second comparisons such as 2080s and K40s. Seems a lot of Youtubers and articles just don't do the same for AMD, probably because AMD has been unusable for AI for a while. If someone can provide some context on which metric to use for best comparing AI GPUs to regular GPUs that'd be great. I've mostly been looking at this wiki for comparisons, but I don't really know which metric listed is the best for comparing price to performance. 

Asking for a lot of stuff I don't know, I appreciate your response in advance

 

[Dr. Will0hlep]

AMD is still basically unusable for AI at this time. packages either don't support AMD, or run way worse. Nvidia's CUDA cores are the way to go.

[PDifolco]

Could you be a bit more precise about budget and intended use ? Which LLM, what kind of power you need ?

And budget ?

But I agree with @will0hlep an old AMD GPU will be crap for AI

And why an eGPU ? Using a chassis makes sense for laptops, but not if you plan to build a desktop...

[YoungBlade]

1. Since the compute should all be happening on the card, an eGPU should be fine. The slow connection should only be a factor if you try running a model that exceeds the VRAM capacity of the card, but that's a bad idea in general - being an eGPU will just make it even worse.

 

2. PCIe 3.0 should be fine - again, as long as you don't run out of VRAM, this shouldn't be a big factor in compute performance.

 

3. The reason people aren't doing these comparisons with old AMD cards is that old AMD cards had terrible AI performance. Until this generation, AMD's AI compute was abysmal, and they still haven't caught up to Nvidia, not to mention that the majority of AI applications still expect CUDA. You can certainly find comparisons with the most recent AMD cards - LTT's reviews of the 9060XT and 9070 series had AI benchmarks, for instance. And there are LLMs that can work on AMD. However, you're not going to get good performance on them unless you're buying something from this generation.

 

Asking to run AI on old AMD is like asking to run AI on old Nvidia - as in, 10 series and older. Even the 1080 Ti, the best consumer Nvidia graphics card without Tensor cores, gets curb-stomped by anything from the 20 series and newer in AI performance. Old AMD does likewise, except in some ways even worse, because they lack CUDA, so some applications just refuse to work. Here's an article from Puget Systems that shows a 1080 Ti against other cards for AI - it ain't pretty.

[KidKid]

1 hour ago, ImLuciano said:

TLDR: I want to buy a used AMD GPU for AI. I don't know which is best to buy or how to compare cards. I am also limited to PCIe 3.0 in an external chassis

I'm currently building a new pc and I have an external GPU chassis that I want to put a card in for dedicated AI loads. I am using AMD GPUs in my pc so I would like an AMD gpu in the eGPU as well. Most of my local AI I am currently using is used for DND NPCs and keeping an output speed like an IRL conversation is important to me. I realize I can just change the AI parameters, but I want to maximize value with the card I pick. I'm unsure which card to get for a few reasons:
1. Since this is an external GPU Chassis, does that mean the speed will be too slow for simulating conversations? I used to use the eGPU for video games on a laptop and didn't have issues with the information transferred over a cable so I'm assuming it won't be an issue, but I would love to know more. 

 

2. The GPU Chassis is PCIe 3.0, I saw some reddit posts saying that cards with PCIe 4.0 or higher will work in PCIe 3.0. But since I care mostly about the compute speed, does this mean a PCIe 4.0 card will just run the same as a PCIe 3.0 card for local AIs? Is getting a PCIe 4.0 or higher card just a waste of money for me?

 

3. I have no idea how to compare AMD cards for AI. I see a lot of AMD Instinct cards on ebay for about $50-$100 (I have no problem with used cards). I don't know how these would compare to getting an old GPU for a similar price. If this was an Nvidia card, I could find graphs with Tokens/second comparisons such as 2080s and K40s. Seems a lot of Youtubers and articles just don't do the same for AMD, probably because AMD has been unusable for AI for a while. If someone can provide some context on which metric to use for best comparing AI GPUs to regular GPUs that'd be great. I've mostly been looking at this wiki for comparisons, but I don't really know which metric listed is the best for comparing price to performance. 

Asking for a lot of stuff I don't know, I appreciate your response in advance

 

AMD isn't good for AI unless you get a newer one (RDNA 4) but you said you want an older one.

[ImLuciano]

44 minutes ago, PDifolco said:

Could you be a bit more precise about budget and intended use ? Which LLM, what kind of power you need ?

And budget ?

But I agree with @will0hlep an old AMD GPU will be crap for AI

And why an eGPU ? Using a chassis makes sense for laptops, but not if you plan to build a desktop...

LLM is flexible, most of what I've done so far is all entry level and can be moved to different models. I plan to find which model works best once I've finished the setup. Budget is less than $200, I was planning to get an older AI GPU off ebay, but it seems like old generations aren't worth it from the replies I've seen here.
I am using an eGPU because it is what I already have. I don't have extra space in my new Case for a card specific to AI.

[ImLuciano]

34 minutes ago, YoungBlade said:

1. Since the compute should all be happening on the card, an eGPU should be fine. The slow connection should only be a factor if you try running a model that exceeds the VRAM capacity of the card, but that's a bad idea in general - being an eGPU will just make it even worse.

 

2. PCIe 3.0 should be fine - again, as long as you don't run out of VRAM, this shouldn't be a big factor in compute performance.

 

3. The reason people aren't doing these comparisons with old AMD cards is that old AMD cards had terrible AI performance. Until this generation, AMD's AI compute was abysmal, and they still haven't caught up to Nvidia, not to mention that the majority of AI applications still expect CUDA. You can certainly find comparisons with the most recent AMD cards - LTT's reviews of the 9060XT and 9070 series had AI benchmarks, for instance. And there are LLMs that can work on AMD. However, you're not going to get good performance on them unless you're buying something from this generation.

 

Asking to run AI on old AMD is like asking to run AI on old Nvidia - as in, 10 series and older. Even the 1080 Ti, the best consumer Nvidia graphics card without Tensor cores, gets curb-stomped by anything from the 20 series and newer in AI performance. Old AMD does likewise, except in some ways even worse, because they lack CUDA, so some applications just refuse to work. Here's an article from Puget Systems that shows a 1080 Ti against other cards for AI - it ain't pretty.

I was under the impression most of what AMD was lacking was on the driver side and therefore older cards would get to a usable state when newer cards were optimized. Clearly not the case. 

[Dr. Will0hlep]

3 minutes ago, ImLuciano said:

I was under the impression most of what AMD was lacking was on the driver side and therefore older cards would get to a usable state when newer cards were optimized. Clearly not the case. 

Yeah, No. AMD's older cards are lacking the hardware to be useful for AI work.

[igormp]

Get a 2060 Super if you want the cheapest GPU with 8GB and tensor cores, or go for a 3060 if you want 12GB of vram.

 

If you are ok with slower performance but want to run larger models, get a P40 or even a P100.