Running Deep learning project is slowing sown the system

[Srihari Krishna]

I have RTX3080 GPU and i use it for my deep learning projects, but whenever i run basic machine deep learning project, my system starts to lag and even youtube videos starts to lag.

Is there anything i am missing that should have been done or is this as it is? Please help

[Fjolfrin]

Depending on your project, the GPU utilization can be really high. I've trained a simple GAN for I2IT and it would take 12GB of VRAM off the GPU.

I would suggest leaving the GPU to train so as to not cripple training time.

If you really gotta use the computer maybe try opening e.g. your browser for youtube using your integrated graphics if your CPU has. But again the CPU is also used during training.

[Bartholomew]

If using windows, "there is your problem", but im assuming you are using linux (windows and ML dont mix well, windows uses too much vram itself, limiting resources for ML and expect longer training times as diverse layers of drivers for ML are suboptimal compared to linux ones).

 

It will depend on the project, but commonly vram shortage plays a role, but also fetching and decompressing trainjng data and writing intermediate snaphots and results can "harass" cpu and storage (like with stylegan training on 512 res and up). So it will depend on the system.

 

For reference, Im using 12c/24t 3900x, 64gb mem, pcie4 nvme for ML trainingdata read and (500mb each) snapshot write, and rtx 3090 with 24gb vram. On stylegan2-ada on 512 res that does +-500-900kimg (500.000 to 900.000 terations) in 12h (depending on dataset and hyperparameters), leaving the machine completely usable for other tasks. This is with powerlimiting the 3090 to 250w using nvidia SMI (the last 100w only gives like a few pct performance with a ton of extra fan noise/wear).

 

With enough cpu power and vram youtube should be okay, even if learning says "100%" gpu usage, as it says that when both training 10million or 40million parameters; batch size 1 or 16 etc. Playing a video and doing a desktop just takes a few cuda cores, not much is needed, and its exceptionally unlikely your task and batch params cause to use exactly "all" cuda cores available.

 

[12345678]

can't you reserve some space in the gpu?

[igormp]

As others have already said, training takes up a lot of GPU utilization, and that gives you the feeling that the rest of the system is slow.

Your only option is to put up with it, or use another GPU just for display (either another cheap, dedicated one, or an iGPU).

 

On 7/4/2021 at 8:18 AM, 12345678 said:

can't you reserve some space in the gpu?

You can limit the vram usage, but not the actual core utilization.