nvidia Used Tesla cards, are they any good?

[Aleksey]

There are some used Nvidia Tesla cards on the market at relatively low price, like $200-$300, some of them seem to have come from China. Are they OK cards? Will they work for machine learning tasks?

[Lingwendil]

Yes, but the use cases for them require the support for them specifically. They also require a forced air cooling solution to run in a desktop. I had a handful of Tesla T4 cards but sold them as they werent as useful as I had hoped for in widows, I think linux has better support.

 

For the money a used quadro P series, or Quadro RTX works well. A p5000 is great with its 16gb of VRAM.

[Aleksey]

Thanks. I use Linux so that's OK. Specifically the Tesla P40 cards with 24Gb of RAM seems to be very interesting.

[Bagzie]

14 minutes ago, Aleksey said:

Thanks. I use Linux so that's OK. Specifically the Tesla P40 cards with 24Gb of RAM seems to be very interesting.

As said above they are quite hard to get running properly in a mainstream system,  You have to bear I mind that they usually have blowymatrons cooling them which is loud.

[Aleksey]

10 hours ago, Bagzie said:

As said above they are quite hard to get running properly in a mainstream system,  You have to bear I mind that they usually have blowymatrons cooling them which is loud.

There are many adapter models that can be 3D printed. E.g. - https://www.yeggi.com/q/nvidia+tesla+p100+p40+k80+by/ An alternative would be to pay Amazon over $1,000/month. I can build an entire computer from used parts for these money.

[igormp]

19 hours ago, Aleksey said:

Thanks. I use Linux so that's OK. Specifically the Tesla P40 cards with 24Gb of RAM seems to be very interesting.

Yeah, those should be plug and play on linux (after you install the driver ofc), but since you mentioned you'll be doing ML, be aware that a 3060 will be faster than it and be able to work on the same amount of data since it has proper half precision support (which the p40 lacks).

 

Turing/volta were a big game changer for ML with their tensor cores.

[aisle9]

19 hours ago, Bagzie said:

blowymatrons

I'm using this term someday. I don't know how or why, but I will randomly use the word "blowymatron" at some point in my life.

[Aleksey]

4 hours ago, igormp said:

Yeah, those should be plug and play on linux (after you install the driver ofc), but since you mentioned you'll be doing ML, be aware that a 3060 will be faster than it and be able to work on the same amount of data since it has proper half precision support (which the p40 lacks).

 

Why? According to this data it supports half precision. According to this comparison chart, it is indeed slower, but there is a catch - 3060 only has 12GB of RAM, while P40 has 24GB, which is important for running modern AI models. Most LLM do not fit into 12GB.

[Aleksey]

Actually if we compare P40 specification and 3060 specification the former is almost 15 times faster when using half precision - 183.7 GFLOPS vs 12.74 GFLOPS. Furthermore - 3060 does not give any performance advantage for half precision. This look too good to be true. I am not even sure if the data is correct.

[Aleksey]

5 hours ago, igormp said:

which the p40 lacks

I think you mixed it up with M40. M40 does not support half precision, P40 - does.

[igormp]

1 hour ago, Aleksey said:

Why? According to this data it supports half precision

Notice how it's 64 times slower than regular fp32, while on volta/turing and newer fp16 is much faster than fp32.

1 hour ago, Aleksey said:

According to this comparison chart, it is indeed slower,

That's not a credible source for comparisons.

One practical comparison is for the stable diffusion benchmarks, notice how a 3060 is 2~3x faster than a P40:

https://vladmandic.github.io/sd-extension-system-info/pages/benchmark.html

1 hour ago, Aleksey said:

but there is a catch - 3060 only has 12GB of RAM, while P40 has 24GB, which is important for running modern AI models. Most LLM do not fit into 12GB.

Given how important quantization is, you won't benefit from it, so you will have to load fp32 models using 24gb, while you could load the exact same model with fp16 using only 12gb, or even lower precision such as int8 and even int4, using 6/3gb ov vram (or making use of that to use bigger models).

At that price point you could also buy 2x used 3060s for a total of 24gb of vram and much faster compute.

1 hour ago, Aleksey said:

Actually if we compare P40 specification and 3060 specification the former is almost 15 times faster when using half precision - 183.7 GFLOPS vs 12.74 GFLOPS. Furthermore - 3060 does not give any performance advantage for half precision. This look too good to be true. I am not even sure if the data is correct.

It is true, and you can take a look at Ampere's whitepaper, those tensor cores are magic

1 hour ago, Aleksey said:

I think you mixed it up with M40. M40 does not support half precision, P40 - does.

See explanation above. Anything older than volta is pretty much useless for modern ML unless you're really tight on budget or don't have any other options.

[Aleksey]

19 hours ago, igormp said:

Notice how it's 64 times slower than regular fp32, while on volta/turing and newer fp16 is much faster than fp32.

21 hours ago, Aleksey said:

OMG, that Gigaflops compared to Teraflops. That's bad...

 

 

[Aleksey]

On 10/7/2023 at 2:03 AM, igormp said:

Given how important quantization is, you won't benefit from it, so you will have to load fp32 models using 24gb, while you could load the exact same model with fp16 using only 12gb, or even lower precision such as int8 and even int4, using 6/3gb ov vram (or making use of that to use bigger models).

At that price point you could also buy 2x used 3060s for a total of 24gb of vram and much faster compute.

Yes, this seems to be a better option. I ditched P40 idea. It has no value. I decided to go with 3090. It has 24GB of RAM. Two 3060 may be a bit cheaper but it is slower than one 3090 and it may require some software tweaks.

[igormp]

1 hour ago, Aleksey said:

Yes, this seems to be a better option. I ditched P40 idea. It has no value. I decided to go with 3090. It has 24GB of RAM. Two 3060 may be a bit cheaper but it is slower than one 3090 and it may require some software tweaks.

As a 3090 owner who pretty much only uses it for ML, that was a nice pick haha

I'll likely pick a second 3090 next month or so