Folding Community Board

[IkeaGnome]

5 minutes ago, Gorgon said:

Id check to make sure your preferences aren’t set to pause the card when the system isn’t idle or there might be an issue with the PCIe bandwidth but even running that card on an x2 or x4 it should be producing much better.

What are you using to get that graph? 

Spoiler

I'm wondering if it's a hardware config issue. About the only thing I'm thinking is pig tailed cables. I'm running this on "Oppbevaring" in my signature. 9900k, Maximus Code XI. 2070 is in the top slot, 1070 is in the bottom. The only reason I went with pigtailed cables for the GPUs is that I ran out of PCIE cables. If this seems like the issue, I'll order more and test. 

The interesting part is that it almost looks like neither gpu is being worked hard according to MSI afterburner. 67C on a 2070 seems low for folding for basically the last 10 days straight. I do have GPU fans set to full speed through most of the temperature curve, but having them stacked like they are seems like there should be higher temps. I do know when I mined with these two cards in an open chassis I was getting higher temps. 

I did just go and set F@H instances to high priority in task manager. I am using the downloaded client for Windows. 

The i9 is thermal throttling. So I'm going to play with rad settings to see if there's anything there I can do and if maybe that effects GPU PPD?

[Pezui]

2 hours ago, Gorgon said:

If your not running HfM.net you should. You might have an unstable overclock and failing Work Units. Your logs will show you but it’s quicker to just look at the HfM.net dashboard.

So how do I get information on HFM? The website gives a connection timeout 408 error if I follow the link in your signature line; and searching the URL info from your hyperlink is talking about Canadian air force retention policy, painting miniatures, or study of super nova... (all cool but) I am coming up pretty dry here. I value your feedback really highly so I am trying to follow what you are saying but I am lost. 

[Gorgon]

2 hours ago, Pezui said:

So how do I get information on HFM? The website gives a connection timeout 408 error if I follow the link in your signature line; and searching the URL info from your hyperlink is talking about Canadian air force retention policy, painting miniatures, or study of super nova... (all cool but) I am coming up pretty dry here. I value your feedback really highly so I am trying to follow what you are saying but I am lost. 

Sorry,

 

my external web server decided to go on vacation when the NVMe disk failed on my ESXi server and I haven’t had time to rebuild it yet so the site your being served is from your dns providers redirection.

 

My other sig link on remote management of Folding has a link to the HFM software which is found here on GIThub

[Gorgon]

17 hours ago, IkeaGnome said:

What are you using to get that graph?

I run a Zabbix server and a bunch of custom scripts to pull the PPD values from all my slots across all my nodes so I can look at long term trends.

 

A side benefit of things I’ve had to do at work over the years.

 

17 hours ago, IkeaGnome said:

About the only thing I'm thinking is pig tailed cables.

Shouldn’t be an issue as long as they’re at least 18 gauge and not too long.  I use dual cables when I can and singles when necessary and I’ve notice no measurable difference.

 

Are you CPU folding as well? If you are and aren’t leaving at least 2-3 threads free then that could be the issue.

 

Nvidia GPUs need one dedicated CPU thread each and if it’s a system just for folding an additional 1-2 threads should be left available for Windows.  If the systems being used for “daily” use then more threads will likely be needed.

 

On my daily driver I don’t fold but I do use 6/8 of the threads for BOINC and Windows 10 seems to work just fine on 2 threads but then the GPU is dedicated to Windows.

 

If you are CPU folding as well try disabling (Pausing) it and see if your PPD goes up on the GPUs.

 

What’s task manager show for GPU compute? It should show a steady 95-97% utilization on each GPU.  

nvidia-smi

in the Command Prompt will also show the GPU utilization, temp and power draws

[Gorgon]

5 hours ago, IkeaGnome said:

  Reveal hidden contents

 

Whoops, didn’t see that. Utilization looks good. Depending on your airflow those temps may be normal but I’d expect a bit higher.

 

Try using the Advanced Control applet instead of the web version.  Right-Click on the folding icon in your task bar and run it.  That will show your individual GPU folding stats in real-time

[IkeaGnome]

1 hour ago, Gorgon said:

Are you CPU folding as well? If you are and aren’t leaving at least 2-3 threads free then that could be the issue.

I am. I'll turn that off for testing. It looks like that was the issue. I'm watching my PPD climb fairly rapidly on Advanced control. 

I am getting 96 and 93% utilization in MSI afterburner, and 96 and 92 in command prompt, but task manager is showing 55 and 51% utilization. I'll let it sit over night and see if it was just an issue of lack of CPU cores. 

 

[Captainmarino]

11 hours ago, Pezui said:

So I complain on the internet about an issue with points. Two days later all normal. Like nothing was wrong. 14m ppd today. Just had two random 5m point days since it wanted a weekend I guess to kick the feet up. 

The same thing happened to me. I even went downstairs to check on things and LAR had it running smoothly, all my underclocks and whatnot were standard... nothing out of the ordinary. Then... poof! Right back to normal. Got me, man.

[marknd59]

I didn't have any drop in PPD but looking at the team points it must have happened to a lot of people.

 

 

[Captainmarino]

30 minutes ago, marknd59 said:

I didn't have any drop in PPD but looking at the team points it must have happened to a lot of people.

 

 

The strange thing is that it nearly perfectly mirrors my graph for those few days.

[NetoriusNick]

I think this is start of the week and lack of WU, I quite consistently see a did at the start of the week as I revert to cancer WU as there are no higher priority ones. !I think!

[NetoriusNick]

On 6/21/2021 at 9:54 AM, Kilrah said:

https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/

 

Ethereum is the main GPU-minable coin, and they're planning to move away from that. 

hummm weekend reading list I think

[Gorgon]

4 hours ago, marknd59 said:

I didn't have any drop in PPD but looking at the team points it must have happened to a lot of people.

 

 

Ahh - OK. This happens once in a while. Some of the Collection servers get stuck and points don't get awarded. Eventually they will get awarded all at once and you'll have a massive spike on that day.

[marknd59]

1 minute ago, Gorgon said:

Ahh - OK. This happens once in a while. Some of the Collection servers get stuck and points don't get awarded. Eventually they will get awarded all at once and you'll have a massive spike on that day.

That's what I've been thinking. It's happened to me before when points have gone missing for a few days.

[Gorgon]

9 hours ago, IkeaGnome said:

I am. I'll turn that off for testing. It looks like that was the issue. I'm watching my PPD climb fairly rapidly on Advanced control. 

I am getting 96 and 93% utilization in MSI afterburner, and 96 and 92 in command prompt, but task manager is showing 55 and 51% utilization. I'll let it sit over night and see if it was just an issue of lack of CPU cores.

Nice - I don't run F@H on my CPUs but spread the love and I run BOINC Word Community Grid's OperPandemics on CPU threads but it is a bit of an art balancing running Folding@Home and BOINC on the same system.

 

On a 9900K with 16 threads and two GPUs I would try a maximum on 12 threads for CPU folding which would allow for 2 for the GPUs and two for the OS. If it is a system you "Daily Drive" then I'd reduce the threads further. The nice thing is you can do this on-the-fly in the Advanced Control and not dump the Work Units (WUs) I believe.

 

Windows Task Manager can deceive you. You have to check all the different "Compute" values and find the one that matches AfterBurner/nvidia-smi.

 

This:

nvidia-smi -i 0 -l 1 --format=csv,noheader --query-gpu=temperature.gpu,power.draw,clocks.current.sm,fan.speed

can be useful for monitoring; adjusting -i 0 or -i 1 for GPU 0 or GPU 1.

 

I found on my 9900K I had to set a AVX offset of -2 on my motherboard to keep it from melting even with an EVGA 280mm CLC. Folding and BOINC tend to really push that chip but I can live with it running at 4.4GHz all-core at 62-68C on the cores and 71C on the VRM.

[WhitetailAni]

3 hours ago, Gorgon said:

Nice - I don't run F@H on my CPUs but spread the love and I run BOINC Word Community Grid's OperPandemics on CPU threads but it is a bit of an art balancing running Folding@Home and BOINC on the same system.

I do the same thing and it's quite special, though I run PrimeGrid instead of OpenPandemics. Current hardware is:
Ryzen 5 3600
GTX 690 (2 dies, 1 card)
GTX 650 Ti BOOST

The first time I did it, I forgot to disable my CPU slot for F@H (fresh install, Windows had to be reinstalled due to Hackintoshing) and didn't know how to limit my BOINC threads. I also had it installed normally so it could use GPUs.

So I had BOINC and F@H and Windows all trying to use my CPU and GPU at the same time. Instant lock and reset.

 

Booted into Safe Mode, uninstalled both, installed BOINC as a service, and then limited threads to 8. GPU problem solved.

Next, I installed F@H with BOINC not running and removed the CPU slot. CPU problem solved. No more problems!

 

With 3 GPUs (GTX 690 has 2 dies, and GTX 650 Ti BOOST) and 8 threads I get 50Kish CPD (credits per day) and 500Kish PPD.

3600 has 12 threads, so thread allocation is something like:

Spoiler

1 - BOINC

2 - BOINC

3 - BOINC

4 - BOINC

5 - BOINC

6 - BOINC

7 - BOINC

8 - BOINC

9 - F@H (GPU 1)

10 - F@H (GPU 2)

11 - F@H (GPU 3)

12 - OS

Works fine for me, though I recently discovered that I can't stop my GTX 650 Ti BOOST from Folding without causing my PC to freeze and then screens to black, making me hit the reset switch. I'm not sure if that's a sign of the GPU going bad, or not enough threads, or something else, but it doesn't matter too much since the 650 runs my second monitor, and that's the entertainment monitor. Even at full power 1080p60 YouTube doesn't stutter at all. 1080p60 Twitch does struggle a bit but turning down the power to Medium fixes that and it's smooth as a greased piece of glass on a slip-n-slide with extra soap (on my end, I am not the cause for frame drops on the streamer's end)

 

I've also noticed that my CPD seems to have picked up from 10K to 50K. I'm not sure if that's because I have 2 different PrimeGrid tasks (PPS Sieve and WW) and PPS Sieve tasks go faster (about 8-10 hours) than WW ones (about 1.5-2 days), or something else.

 

Temps for those curious are:

CPU - 74C (CPU temperature does not go past this)

GTX 690 #1 - 84-85C
GTX 690 #2 - 87-88C
GTX 650 Ti BOOST - 68-70C

 

690 #2 die is warmer as it exhausts towards the front of my case, where there isn't active airflow moving the air out. I'm going to get some small 40mm fans to put in a front bay slot right behind the GPU exhaust which should help a lot (I did a simulation by pulling off the front panel and holding an 80mm fan there, temps dropped to 84 from 88C. Repasting the card with Kryonaut and swapping the thermal pads for 12.8 W/mK pads should help too (in the mail right now).

[IkeaGnome]

8 hours ago, Gorgon said:

Nice - I don't run F@H on my CPUs but spread the love and I run BOINC Word Community Grid's OperPandemics on CPU threads but it is a bit of an art balancing running Folding@Home and BOINC on the same system.

 

On a 9900K with 16 threads and two GPUs I would try a maximum on 12 threads for CPU folding which would allow for 2 for the GPUs and two for the OS. If it is a system you "Daily Drive" then I'd reduce the threads further. The nice thing is you can do this on-the-fly in the Advanced Control and not dump the Work Units (WUs) I believe.

 

Windows Task Manager can deceive you. You have to check all the different "Compute" values and find the one that matches AfterBurner/nvidia-smi.

 

This:

nvidia-smi -i 0 -l 1 --format=csv,noheader --query-gpu=temperature.gpu,power.draw,clocks.current.sm,fan.speed

can be useful for monitoring; adjusting -i 0 or -i 1 for GPU 0 or GPU 1.

 

I found on my 9900K I had to set a AVX offset of -2 on my motherboard to keep it from melting even with an EVGA 280mm CLC. Folding and BOINC tend to really push that chip but I can live with it running at 4.4GHz all-core at 62-68C on the cores and 71C on the VRM.

I've got something messing with it for sure. My PPD dropped back down to 800k over night on both GPUs. Back to the same as before I disabled the CPU. This computer is not daily driven. 

I just bumped power limit and core clock up to make sure it is responding. 

from 69, 173.91 W, 1845 MHz, 93 %

to 71, 191.17 W, 1920 MHz, 94 %

So the GPU is responding tweaks. It does look like when ever I try to get a new WU I get errors all over the place due to configuration. Would this be because of the specific GPU combination(doesn't seem like it because they both fold separately), or just the GPUs I'm using? 

 

Spoiler

I did just change network settings from public to private to see if that'll help. Thanks for all the help so far. I just have no clue where to start troubleshooting these issues. 

For what it's worth, I am using slots 16_1 and 16_2. I've removed my HBA from the x1 slot and the only storage plugged into the motherboard currently is the 500gig Windows Boot SSD. I do have a Windows update to do, so I'll do that, as well as a BIOS and Chipset update both from this year and see if that helps at all.

Spoiler

 

[RollinLower]

what is going on?!

[marknd59]

41 minutes ago, RollinLower said:

what is going on?!

Looks like some of the lost points have turned up. Also looks like there is a shortage of covid WU's I'm getting cancer and alzheime WU's.

 

[Gorgon]

12 hours ago, IkeaGnome said:

I've got something messing with it for sure. My PPD dropped back down to 800k over night on both GPUs. Back to the same as before I disabled the CPU. This computer is not daily driven. 

I just bumped power limit and core clock up to make sure it is responding. 

from 69, 173.91 W, 1845 MHz, 93 %

to 71, 191.17 W, 1920 MHz, 94 %

So the GPU is responding tweaks. It does look like when ever I try to get a new WU I get errors all over the place due to configuration. Would this be because of the specific GPU combination(doesn't seem like it because they both fold separately), or just the GPUs I'm using?

Your on the right track. There was/is an issue overnight with the Folding at Home Servers we refer to as "The Download Bug" so most people suffered a drop in points as we couldn't get work assigned consistently, sometimes for hours on end.

In my case this graph of all my nodes' GPU power draws from overnight _SHOULD_ be flat at about 1kW with the three slots running BOINC occasionally kicking off for a short period.

 

The "Forcibly Closed Connection" messages are just from you closing the Advanced Control or Web Control applications.

 

I wouldn't worry too much about your HBA as the x1 slots in most x570 boards are all fed from the Chipset whereas the upper x16 and middle x8 in x16 slots are both fed directly from the CPU. If it's in the upper x1 slot, however, it is likely getting a little warm above the top GPU.

[Pezui]

So my pair of 3090's are sitting idle unable to get a work unit right now... but my GTX 1030 that I recently purchased just to give me more monitor outputs is rocking a 400,000 atoms size job expected to finish in 3 days. Like what? 

23:17:54:WU01:FS02:Connecting to assign1.foldingathome.org:80
23:17:54:WARNING:WU01:FS02:Failed to get assignment from 'assign1.foldingathome.org:80': No WUs available for this configuration
23:17:54:WU01:FS02:Connecting to assign2.foldingathome.org:80
23:17:55:WARNING:WU01:FS02:Failed to get assignment from 'assign2.foldingathome.org:80': No WUs available for this configuration
23:17:55:WU01:FS02:Connecting to assign3.foldingathome.org:80
23:17:55:WARNING:WU01:FS02:Failed to get assignment from 'assign3.foldingathome.org:80': No WUs available for this configuration
23:17:55:WU01:FS02:Connecting to assign4.foldingathome.org:80
23:17:56:WARNING:WU01:FS02:Failed to get assignment from 'assign4.foldingathome.org:80': No WUs available for this configuration
23:17:56:ERROR:WU01:FS02:Exception: Could not get an assignment
23:27:50:WU00:FS00:0x22:Completed 175000 out of 2500000 steps (7%)

[marknd59]

I started to have a problem getting WU's yesterday and I've been getting cancer and alzheime WU's not covid. I've set next-unit-percentage down to 97 and started up babysitter which has kicked in a couple of times.

 

EDITED to add

A quick look at the F@H forum and everyone is have a problem getting WU's.

[roxasthehunter]

 Posting this here for more visibility. If anyone has any extra cpu power they can spare for Universe at Home, we're just barely scraping by in the sprint so it any help would be appreciated.

[LazyDev]

After a little while away, I'm back! 

 

I don't mind the lack of WU's right now, as it's saving me money on the electric bill.

 

[TVwazhere]

Well at least this isnt a chart of crypto currency :^) 

[RollinLower]

3 minutes ago, TVwazhere said:

Well at least this isnt a chart of crypto currency :^)

looks simmilar enough tbh. have you seen XCH prices?