Folding@Home occasionally owning all ram (linux)

[slick613]

I've got a headless ubuntu server with 7 nvidia gpus. It's normally for mining but I switched it over to folding last week. Twice now it's got to a point where I can't ssh in. One time I managed to get in to find load average of 24+ and got to a point where any command executed resulted in an unable to allocate memory error. 

 

Any idea what could be causing this or how to diagnose? It's a very reliable rig. Been running for years mining without a hiccup. Just keeps tripping on folding for some reason. 

[FloRolf]

Well folding is a completely different thing than mining. Maybe your graphics cards are bandwidth limited (i guess you are running pcie x1?) and then they shovel everything to system memory until it overloads. 

Just a guess tho. 

[jo___l]

What are the specs of the machine?

[Gorgon]

I have a 6 GPU rig also running headless on Ubuntu 18.04.4.

 

I had to up the RAM to 32GB as some of the newer OpenMM22 Work Units are consuming 1.5GB each and I've recently seen some people reporting some units north of 2GB.

 

Make sure you have 1 thread free per GPU to keep it fed and 1 thread for the OS and you should be able to allocate the remaining threads for CPU folding.

 

If your using PCIe3 x1 risers I'd be interested in seeing what your PPD performance per card is to see what, if any, degradation in performance your seeing.

 

I'm running with m.2 to PCIe risers so I've got x4 to all the slots and in testing only saw degradaion in performance under Windows on a RTX 2080 Super. Under Linux it seemed OK

[slick613]

4G of ram. Some celeron processor. Six PCIe risers except 1 M.2 riser. Six 1070s, 1 1070ti. 

Mining takes next to no cpu/ram. That's why it's those specs are minimal. I guess folding is much heavier on the CPU/RAM. 

The machine gets owned but if I leave it it, it comes out of it. I guess worst case, I just let it be. 

 

Here's the top output when it's "ok". No work for the 7th GPU here.

 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4188 fahclie+  39  19 33.290g 432212 110384 R  33.7 11.0  25:27.12 FahCore_22
 3622 fahclie+  39  19 33.291g 363784 105752 R  33.3  9.3  29:51.17 FahCore_22
 3629 fahclie+  39  19 33.290g 361252 106436 R  33.3  9.2  29:34.52 FahCore_22
 4609 fahclie+  39  19 33.620g 661472 118964 R  33.3 16.8  19:33.65 FahCore_22
 3636 fahclie+  39  19 33.290g 360892 105028 R  33.0  9.2  30:17.78 FahCore_22
 3649 fahclie+  39  19 33.305g 293240 105080 R  33.0  7.5  29:46.66 FahCore_22
 1538 root      20   0       0      0      0 S   0.3  0.0   0:27.85 nv_queue
 1574 root     -51   0       0      0      0 S   0.3  0.0   0:15.98 irq/130-nvidia
 3601 fahclie+  20   0 58.166g 434956   2324 S   0.3 11.1   0:40.10 FAHClient
    1 root      20   0  119660   3404   2212 S   0.0  0.1   0:04.74 systemd

 

That's 6 GPUs eating 33% CPU each. hahah. 

I'm brand new to folding. How do I find out PPD perf?

[slick613]

With such low CPU specs, would it be better to cut down how many GPUs I use? I think I only got a few hundred thousand points yesterday from 5 WU. 

FAHControl estimates 2m per day.

[jo___l]

Most definitely not enough ram. For that machine I would use minimum 16 GB but more likely 32 or 64GB.
Most likely not enough cpu to feed the gpus work.

 

Do you have any ram you can add?

[jo___l]

Can you provide a screenshot of htop?

[slick613]

4 hours ago, jo___l said:

Can you provide a screenshot of htop?

TIL htop. 

Definitely don't have that kind of ram laying around. Think CPU would still be too big of a bottleneck anyways?

 

How many GPUs do you think that's enough ram for. I should at least cut it down to that. Problem is I don't know how to gauge if folding is working alright. Is there a metric for folding performance or utility I should be looking at?

 

[jo___l]

Count on 1GB RAM per GPU.

 

Start with 1 GPU. If that works fine add another one. Repeat until failure.

 

To get information about your CPU run cat /proc/cpuinfo.

[Gorgon]

5 hours ago, slick613 said:

TIL htop. 

Definitely don't have that kind of ram laying around. Think CPU would still be too big of a bottleneck anyways?

 

How many GPUs do you think that's enough ram for. I should at least cut it down to that. Problem is I don't know how to gauge if folding is working alright. Is there a metric for folding performance or utility I should be looking at?

 

Yeah, if that's only a 2c/2t processor then it is going to be massively CPU bottle-necked as each NVidia GPU tries to exclusively lock one thread. Also I see all you RAM is consumed and all your swap so your disk is thrashing trying to task switch and I see your system load at 6.76! You normally want to see this under 1 on an interactive system.

 

It's pretty impressive that it is still running.

 

With a 2c/2t processor I'd restrict it to just 2 GPUs and you'll likely generate more points. Give that a try and get some stable results then try adding an additional GPU slot and see what happens.

 

FAHClient --send-command ppd

should show you the PPD

FAHClient --send-command slot-info

should show you the slots configured

FAHClient --send-command queue-info

should show you the PPD per slot

 

 

[slick613]

Excellent info here. Thanks!

Folding is very different than mining. I guess there's a lot of work for the CPU to move data in and out of the GPU. 

Seems to be surviving 3 GPUs without maxing out ram, but the CPU is pinned. Going to bring it down to 2 and put the rest back to their mining work. 

Too bad. I was hoping to point all 7 at this but a CPU upgrade isn't in the cards. 

 

Thanks for all the help. I learned a bunch.