GPU Memory amount vs memory speed

[Krolic]

Hi all,

 

So, I am a graduate student in a machine learning lab. We currently have a dedicated GPU server with 8 GeForce Titan X cards (I believe the Maxwell chipset) and have been looking at getting an additional server with some new GPUs. I am the main go-to person for my PI about this stuff, as I know the most about hardware. Personally, I would love to get some of the new Titan V cards, but my PI got a quote for a setup with some Telsa P40 cards. The P40 cards have 24 GB of GDDR5 (I believe) at 346 GB/s memory bandwidth. The Titan V's have only 12 GB of graphics memory but at a bandwidth of 652.8 GB/s. My question is this: for machine learning purposes, memory is a crucial aspect of it so would these be considered equal? How does the amount of memory and memory bandwidth compare? Is half the memory but having it twice as fast equivalent for our purposes? I am usually fairly knowledge about this sort of hardware stuff, but I have no idea.

[Mira Yurizaki]

2 minutes ago, Krolic said:

My question is this: for machine learning purposes, memory is a crucial aspect of it so would these be considered equal? How does the amount of memory and memory bandwidth compare? Is half the memory but having it twice as fast equivalent for our purposes? I am usually fairly knowledge about this sort of hardware stuff, but I have no idea.

While I don't know what neural networks do with memory, I can say this: GPU memory bandwidth is not a substitute for video card memory capacity. The thing is, when you run out of VRAM on a GPU, it has to start swapping not-used-as-often data to somewhere else, usually system RAM. The thing is, it can only do this over the PCIe interface, which at PCIe 3.0 x16 is 15.6GB/s. And when memory has to swap, anything the GPU was doing has to stall so overall performance goes down.

 

So figure out what your application needs. If it eats up a lot of memory, you need more capacity over bandwidth.

[pyrojoe34]

This is highly dependent on the algorithms and datasets you use. There is no answer to this question. The bottom line is: where is the bottleneck? Do you run out of RAM when loading assets? If not then capacity is not the issue. Are your cores skipping cycles while waiting to load data from the RAM to the cache? If not then bandwidth is not the issue. Is your GPU skipping cycles while it waits for instructions from the CPU? Then your CPU is the limitation.

 

There is not good way to answer this without either running diagnostic codes during a run or just trying different configs and seeing what matters most.

 

You might also want to see if your algorithms can use mixed precision (i.e. tensor cores) which would give you a >10x improvement in FLOPS over traditional comp methods. If you can use that tech and can implement it efficiently you might want to consider Volta-based cards (Tesla V100 or Titan V) for much faster training/execution.

[KarathKasun]

Depends on workload.  Bandwidth is more important in some cases and capacity is more important in others.

 

Capacity isnt everything @M.Yurizaki, There were GT1030 class cards back in the day that came with 4gb+ of VRAM (GT430 DDR3).  The memory bus was not capable of even transferring enough data to use the memory fast enough to work faster than a CPU for CUDA work.

[Mira Yurizaki]

Just now, KarathKasun said:

Capacity isnt everything @M.Yurizaki, There were GT1030 class cards back in the day that came with 4gb+ of VRAM (GT430 DDR3).  The memory bus was not capable of even transferring enough data to use the memory fast enough to work faster than a CPU for CUDA work.

If you're going that slow, then of course. But for the purposes of OP's question, if the application eats up memory, then it's better to have capacity over bandwidth to avoid running out and needing to swap, even if the task could benefit from more bandwidth.

 

That's assuming of course the application doesn't have an option to scale how much RAM it can consume.

[Krolic]

Ok, thanks for the input. The stuff we are dealing with are massive image files (like 3D volumes for CT and MRI scans). More memory would mean we can dumb more info per batch for training. But the points about the PCIe bandwidths and other points are valid. The points about bottle necks are probably the most valid, where are they. I think the increase in memory would mean better performance, in the case where all other variables are held constant. I think the best thing I should look at are some bench marks between the Titan V and the Tesla P40. Personally, I would love the Quadro GV100 as it is volta architecture, meaning tensor cores and therefore mixed precision, but also 32 GB per card. The downside is they are hella expensive.