Two RTX 5060Ti 16GB for AI Model Training

[CheeseNg]

Has anyone tried to train a model using two RTX 5060 Ti 16GB? I plan to use them to train models using DDP or self-hosting LLMs.

 

Never try that before, is it possible??

 

If yes, I have made a list.

https://pcpartpicker.com/user/kcheeseng/saved/Z76Jt6

 

Any recommendations, or are there any issues with the performance?

[Dr. Will0hlep]

Given the number of Tensor cores, I wonder if you might be better off just getting a single RTX 5070 Ti... Almost as many Tensor cores, but should be cheaper

[Shimejii]

Is it possible? Depends on the program you use and if they can properly utilize them. Is it worth it? Absolutely not. You are far better off with one much better GPU then two mediocre ones.

 

5070ti should be cheaper to buy then two 5060tis. But this pricing is pretty abysmal for them.

[Dr. Will0hlep]

Also, just noticed, why are you going intel 12600KF when your already paying for DDR5. Surely, you should be looking at AM5?

[CheeseNg]

Yup, compared to the cores, it’s crap. But I’m thinking of getting 32 GB of VRAM to run or train a larger model. I was previously planning to get a used gpu, like 3090, but the marketplace in my country doesn’t seem trustworthy.

4 minutes ago, will0hlep said:

Given the number of Tensor cores, I wonder if you might be better off just getting a single RTX 5070 Ti... Almost as many Tensor cores, but should be cheaper

 

[Dr. Will0hlep]

2 minutes ago, CheeseNg said:

Yup, compared to the cores, it’s crap. But I’m thinking of getting 32 GB of VRAM to run or train a larger model. I was previously planning to get a used gpu, like 3090, but the marketplace in my country doesn’t seem trustworthy.

From my experiance it would be better to just let the model overrun into RAM at that point. Moving data between GPUs is not very efficent.

[CheeseNg]

5 minutes ago, will0hlep said:

Also, just noticed, why are you going intel 12600KF when your already paying for DDR5. Surely, you should be looking at AM5?

I hadn’t considered AMD. Does it work better than Intel with DDR5? The reason I’m leaning toward Intel is because I want to benefit from certain libraries, like scikit-learn.

[Dr. Will0hlep]

1 minute ago, CheeseNg said:

I hadn’t considered AMD. Does it work better than Intel with DDR5? The reason I’m leaning toward Intel is because I want to benefit from certain libraries, like scikit-learn.

There is definately stuff that dosn't work on AMD GPUs, but I'm not actually aware of any common libraries for this stuff that doesn't work on AMD CPUs.

 

From my experiance of this stuff, if you want performance, you want an AMD CPUs.

[CheeseNg]

7 minutes ago, will0hlep said:

From my experiance it would be better to just let the model overrun into RAM at that point. Moving data between GPUs is not very efficent.

Never heard about that. How does it work during model training? Does it automatically decide what to offload, or do we need to configure it manually?

[Dr. Will0hlep]

1 minute ago, CheeseNg said:

Never heard about that. How does it work during model training? Does it automatically decide what to offload, or do we need to configure it manually?

For any work I've done, it is automatic.

 

It does add latency cause RAM is slower than VRam, but you'll have similar issues with storing the model accross multiple GPUs, unless the model is specifically Parallised to work across multiple GPUs.

[igormp]

9 hours ago, CheeseNg said:

Has anyone tried to train a model using two RTX 5060 Ti 16GB? I plan to use them to train models using DDP or self-hosting LLMs.

 

Yeah, it'd work just fine. Not the fastest, but it'd still work nonetheless.

9 hours ago, will0hlep said:

Given the number of Tensor cores, I wonder if you might be better off just getting a single RTX 5070 Ti... Almost as many Tensor cores, but should be cheaper

Half the vram in total limits one quite a lot, depending on what they're doing.

9 hours ago, Shimejii said:

Is it possible? Depends on the program you use and if they can properly utilize them. Is it worth it? Absolutely not. You are far better off with one much better GPU then two mediocre ones.

 

5070ti should be cheaper to buy then two 5060tis. But this pricing is pretty abysmal for them.

See above.

8 hours ago, will0hlep said:

From my experiance it would be better to just let the model overrun into RAM at that point. Moving data between GPUs is not very efficent.

No way, it's way better to use 4x3060s than a single 16GB GPU that won't be able to fit your entire model. For example, a 3060 with its 12gb was often better than a 3080 simple because it had an extra 2gb of vram and could run larger models this way.

If you don't have enough vram to begin with, then you're SOL. Get a reasonable amount of vram that fits your model, then look into the fastest solution you can afford with this amount of memory (or more).

 

8 hours ago, CheeseNg said:

I hadn’t considered AMD. Does it work better than Intel with DDR5? The reason I’m leaning toward Intel is because I want to benefit from certain libraries, like scikit-learn.

There are no benefits in using intel with scikit. Zen 5 also has a pretty amazing AVX-512 unit, which consumer Intel CPUs lack, so its way faster at ML-related tasks.

8 hours ago, CheeseNg said:

Never heard about that. How does it work during model training? Does it automatically decide what to offload, or do we need to configure it manually?

No, it's crap, don't do that.

 

 

8 hours ago, will0hlep said:

For any work I've done, it is automatic.

 

It does add latency cause RAM is slower than VRam, but you'll have similar issues with storing the model accross multiple GPUs, unless the model is specifically Parallised to work across multiple GPUs.

I know you're trying to help, but this is actual misinformation.

Spilling your model into RAM is AWFUL, and it's not the case for training.

All models can be made to run across multiple GPUs, so that's the norm. Both for inference and training. The impact in having models across GPUs is FAR lesser than having stuff spilling into RAM, since you're just doing gradient accumulation or something of the sorts across the PCIe bus, and not full blown weights swapping or running the feed-forward within the CPU.