how do GPU cores work?

[asheenlevrai]

Hi,

 

A colleague of mine is currently running a science-related software that does some kind of "machine learning", "transfer learning with deep neural networks" and "training"... I have no idea what all this means yet...

Shame on me.

 

However, the developers say you can use it on your CPU and it will be slow, or on your Nvidia GPU (recommended 8GB of vRAM or more) and it will be much faster. And indeed it is.

 

Our systems runs on an 9900K with 32GB of 2800MHz DDR4 coupled to a GTX Titan X with 12GB of vRAM.

 

Now, when we launch the software and make it use the GPU for calculation, Task Manager tells us that only 7% of the GPU is used while ALL 12GB of the vRAM is used.

 

AFAIU our GPU has 3072 cores.

Does it uses all the cores all the time?

Can it use only some of the cores like CPUs do?

How can I know how many GPU cores are used at a given moment and at what level? In a way similar to the task manager that could show me what CPU cores run at 100% and what other cores are not used or run at lower levels...

 

Thank you very much in advance for your feedback.

 

Best,

-a-

[Parideboy]

In the gpu tab in taskmanager, you can click on the small arrows to change what is monitored. Probably atm the 3D compute load is being monitored, showing no to little load

[Jurrunio]

While CPUs have many different types of processing units inside each core (for different workloads), GPU cores are essentially just ALUs (arithmetic logical units), good at crunching math and that's it.

 

14 minutes ago, asheenlevrai said:

Does it uses all the cores all the time?

You can check core usage for GPU for that, I dont know what OS you're on but you can bet on OC tools to monitor such thing

 

14 minutes ago, asheenlevrai said:

Can it use only some of the cores like CPUs do?

It can but GPU related tasks are heavily multithreaded. Otherwise why use a GPU at all, an individual GPU core is slower than a CPU core in tasks a GPU is good at.

 

14 minutes ago, asheenlevrai said:

How can I know how many GPU cores are used at a given moment and at what level? In a way similar to the task manager that could show me what CPU cores run at 100% and what other cores are not used or run at lower levels...

atm such thing doesnt exist to us. Maybe Nvidia/AMD use such thing internally but it's not publically available.

[asheenlevrai]

15 minutes ago, Parideboy said:

In the gpu tab in taskmanager, you can click on the small arrows to change what is monitored. Probably atm the 3D compute load is being monitored, showing no to little load

We tried that.

3d show occasional spikes (rare) while copy shows a constant low level. Nothing in video encode/decode

[asheenlevrai]

7 minutes ago, Jurrunio said:

You can check core usage for GPU for that, I dont know what OS you're on but you can bet on OC tools to monitor such thing

 

 

We're using Win10.

 

How can I check GPU core usage then?

[Jurrunio]

18 minutes ago, asheenlevrai said:

We tried that.

3d show occasional spikes (rare) while copy shows a constant low level. Nothing in video encode/decode

click on "copy" or "3d" and such, you'll find way more options. Change one of them to "Cuda" and you should see the usage properly

[asheenlevrai]

3 minutes ago, Jurrunio said:

click on "copy" or "3d" and such, you'll find way more options. Change one of them to "Cuda" and you should see the usage properly

Thanks a lot!

I didn't know I could do that.

I looked at all available options to monitor ant they are all close to 0 except "Compute_0" that is at 97% ("compute_1" is at 0%).

 

I'll look into what is Compute_0 and how it is different from Compute_1

 

Thanks again

[Jurrunio]

1 hour ago, asheenlevrai said:

Thanks a lot!

I didn't know I could do that.

I looked at all available options to monitor ant they are all close to 0 except "Compute_0" that is at 97% ("compute_1" is at 0%).

 

I'll look into what is Compute_0 and how it is different from Compute_1

 

Thanks again

hmm so I guess it's not using CUDA but some other compute standard (like OpenCL and more) instead

[igormp]

10 hours ago, asheenlevrai said:

AFAIU our GPU has 3072 cores.

Does it uses all the cores all the time?

Depends, but it likely does so.

Quote

Can it use only some of the cores like CPUs do?

Yes. In GPUs, there are tons of different units with smaller units inside those (like for nvidia, GPU > GPC > TPC > SM > Individual execution units, like a cuda or tensor core), which can be used to run different "tasks".

Quote

How can I know how many GPU cores are used at a given moment and at what level? In a way similar to the task manager that could show me what CPU cores run at 100% and what other cores are not used or run at lower levels...

You really can't. There's no publicly available tool for that.

 

9 hours ago, asheenlevrai said:

We tried that.

3d show occasional spikes (rare) while copy shows a constant low level. Nothing in video encode/decode

Try using nvidia-smi, it'll provide a better usage percentage.

 

As for the title question:

Quote

how do GPU cores work?

GPU cores consist of many small units capable of only doing simple tasks, as Jurrunio said, the catch is that it can do MANY small tasks at once, something that's called SIMD (single instruction, multiple data). Those units are akin to the ones used for AVX/SSE in modern CPUs.

 

The main difference between your regular CPU is that, while a CPU can easily do many different tasks, and easily do branching and whatnot, SIMD cores can only do a single task at a time, and are really dumb, so they can't do any branching or something like that. Where they excel at is when you give them a single that can be done in parallel over huge amounts of data at once, like for example, having 1000 8bytes numbers and multiplying all of them at ONCE by a constant or even another series of numbers. A CPU would need to step number by number to do something like that.

 

Here's a link that I really like about an optimization using SIMD. Since it may be quite complicated for most people, I leave those 2 gifs here that show the difference between regular serial execution in a CPU vs parallel execution with a SIMD unit:

 

[asheenlevrai]

14 hours ago, Jurrunio said:

hmm so I guess it's not using CUDA but some other compute standard (like OpenCL and more) instead

mmhh...

 

Why do you say it's not using CUDA? Excuse my ignorance here.

According to requirements the software uses CUDA and is not compatible with AMD GPUs

[asheenlevrai]

6 hours ago, igormp said:

Depends, but it likely does so.

Yes. In GPUs, there are tons of different units with smaller units inside those (like for nvidia, GPU > GPC > TPC > SM > Individual execution units, like a cuda or tensor core), which can be used to run different "tasks".

You really can't. There's no publicly available tool for that.

 

Try using nvidia-smi, it'll provide a better usage percentage.

 

As for the title question:

GPU cores consist of many small units capable of only doing simple tasks, as Jurrunio said, the catch is that it can do MANY small tasks at once, something that's called SIMD (single instruction, multiple data). Those units are akin to the ones used for AVX/SSE in modern CPUs.

 

The main difference between your regular CPU is that, while a CPU can easily do many different tasks, and easily do branching and whatnot, SIMD cores can only do a single task at a time, and are really dumb, so they can't do any branching or something like that. Where they excel at is when you give them a single that can be done in parallel over huge amounts of data at once, like for example, having 1000 8bytes numbers and multiplying all of them at ONCE by a constant or even another series of numbers. A CPU would need to step number by number to do something like that.

 

Here's a link that I really like about an optimization using SIMD. Since it may be quite complicated for most people, I leave those 2 gifs here that show the difference between regular serial execution in a CPU vs parallel execution with a SIMD unit:

 

Thank you so much for the detailed answer :)))

[asheenlevrai]

On 8/25/2020 at 9:24 AM, asheenlevrai said:

mmhh...

 

Why do you say it's not using CUDA? Excuse my ignorance here.

According to requirements the software uses CUDA and is not compatible with AMD GPUs

bump :)

[igormp]

3 hours ago, asheenlevrai said:

bump

Do you know which software it is? If it's tensorflow, pytorch or the likes, it's using CUDA for sure since those frameworks don't support any other GPU API. It may be that the task manager is just misreporting.

 

Again, I'd recommend you to use nvidia-smi to get a better GPU usage info.

[asheenlevrai]

On 8/28/2020 at 2:27 PM, igormp said:

Do you know which software it is? If it's tensorflow, pytorch or the likes, it's using CUDA for sure since those frameworks don't support any other GPU API. It may be that the task manager is just misreporting.

 

I believe it is tensorflow. I'll look into it

 

On 8/28/2020 at 2:27 PM, igormp said:

 

Again, I'd recommend you to use nvidia-smi to get a better GPU usage info.

Thanks. Will do