Folding Community Board

[Gorgon]

1 hour ago, Justaphf said:

Awesome level of analysis @Gorgon, thanks!  I have been running my 1070s at ~80% TPD (120W) with 200 MHz memory overclock and no core overclock and getting ~1.0-1.2m PPD each.  I never went through and did a full sweep to figure out optimal operating points like you have in your GPU profiling guide (I've been meaning to, just haven't gotten around to it).  Basically just spot checked 4 or 5 settings, found something that worked 2 years ago and haven't really touched those settings since.  For the 30 series cards I just reduced the TPD to 90% on each of them and left overclock alone after even minimal increments seemed to cause issues (they seemed more sensitive compared to the 10 series).

 

For the 30 series cards, have you ever done testing of PCIe gen 3 vs gen 4?  I didn't think it made any difference for FAH compute, but when I was getting the NVMe to PCIe risers from ADT-Link they had 4.0 for only $6 more than the 3.0 so picked those up instead.  They work fine and show up as running the 30 series cards at 4.0x4 as expected.  Though now that I think of it, wonder if gen 4 costs more power than gen 3.

Thanks,

 

First - Over-clocking - with the exception of the 1070ti where it's base +boost clock seems way too low at ~1850MHz, it is generally not worth it to try anything over 50MHz as it will not be stable long-term and one or two failed WUs after 2-3 hours will quickly negate any boost in performance. The issue seems to be when the GPUs return from a checkpoint or some other idle task with too high of an o/c they boost up to >2050Mhz briefly and crash. Running:

nvidia-smi -i <x> -lgc 0,2040

can help with this but it's just not worth the effort IMHO. So I typically just run a +50 and leave it at that.

 

Pascal and Turing will both perform most efficiently at the lowest Power-Limit and some Turing cards (2060, for one) have a ridiculously high lower power limit so locking the graphics clock will get you too a more efficient place. Somewhere between 1300-1500MHz seems to be the sweet spot for Turing and Ampere.

 

I haven't tried Gen4 as I only have the 3070Ti that supports it but I suspect it would make only marginal differences and only with possibly 3070ti but definitely 3080 and higher might benefit on Windows and likely less so on Linux as I observed on my 6-GPU build. Likely the power difference from Gen4 is negligible.

[rkv_2401]

On 7/26/2022 at 6:50 AM, Gorgon said:

Pascal can be a bit challenging but in general they are most efficient at their lowest Power-Limit setting. They'd likely be even more efficient below their lower Power-Limit but, unfortunately, unlike Turing and Ampere they're incapable of having their Graphics-Clock locked (limited to a maximum frequency that would allow them to run at a lower power than the lower Power-Limit).

 

I only have one Pascal GPU left, an EVGA 1070Ti SC. Unlike all my other cards it needs a +150MHz o/c to run at a reasonable (1900-2000MHz) stable clock at defaults and also unlike the other GPUs is also 24x7x365 stable Folding with that overclock.

You can set locked clock speeds on Pacal cards in Windows. I did this on both my old 1060 3GB and current EVGA 1070 SC. I'm not sure about Linux though.

9 hours ago, Gorgon said:

Thanks,

 

First - Over-clocking - with the exception of the 1070ti where it's base +boost clock seems way too low at ~1850MHz, it is generally not worth it to try anything over 50MHz as it will not be stable long-term and one or two failed WUs after 2-3 hours will quickly negate any boost in performance. The issue seems to be when the GPUs return from a checkpoint or some other idle task with too high of an o/c they boost up to >2050Mhz briefly and crash.

This is a case where it may help to lock clocks using Afterburner. When folding, I just set my 1070 at 1999/2012Mhz and let it stay at that speed until I change the profile. There may be a minor inefficiency at checkpoints or the end of each WU where there's no work to do but the GPU is still running at full speed, but it's a fair trade off in my eyes.

[Gorgon]

4 hours ago, rkv_2401 said:

You can set locked clock speeds on Pacal cards in Windows. I did this on both my old 1060 3GB and current EVGA 1070 SC. I'm not sure about Linux though.

This is a case where it may help to lock clocks using Afterburner. When folding, I just set my 1070 at 1999/2012Mhz and let it stay at that speed until I change the profile. There may be a minor inefficiency at checkpoints or the end of each WU where there's no work to do but the GPU is still running at full speed, but it's a fair trade off in my eyes.

nvidia-smi -i 0 -lgc 0,1440
	Setting locked GPU clocks is not supported for GPU 00000000:06:00.0.
	Treating as warning and moving on

Nope - not supported on Linux under Pascal.

 

I assume maybe your talking about creating a custom voltage/frequency curve in Afterburner/xoc etc. which is also not possible under Linux.

[BondiBlue]

Any ideas for what I should expect to get out of a basic GTX 1080? I picked one up for a good price, mostly just to mess around with it for a bit and eventually put into permanent use in a Folding@home system. The one I bought is an HP OEM card, so I don't expect it to have a great cooler, but that's not too big of a deal. 

 

What should I be expecting in terms of PPD at stock speeds? How low can you get the power draw to go before the PPD drops a lot? I'm not very familiar with Pascal cards, and the only other one I own is a GT 1030. 

 

I also picked up a Quadro M4000 (8GB single slot Maxwell 2.0 card), so I'll be trying to see what I can squeeze out of it as well. 

[Gorgon]

10 hours ago, BondiBlue said:

Any ideas for what I should expect to get out of a basic GTX 1080? I picked one up for a good price, mostly just to mess around with it for a bit and eventually put into permanent use in a Folding@home system. The one I bought is an HP OEM card, so I don't expect it to have a great cooler, but that's not too big of a deal. 

 

What should I be expecting in terms of PPD at stock speeds? How low can you get the power draw to go before the PPD drops a lot? I'm not very familiar with Pascal cards, and the only other one I own is a GT 1030. 

 

I also picked up a Quadro M4000 (8GB single slot Maxwell 2.0 card), so I'll be trying to see what I can squeeze out of it as well. 

If your planning on running the 1080 at a low power limit in the efficiency “sweet spot” then the quality of the cooler doesn’t really matter. I usually prefer EVGA SC or XC over FTW models, for example, as they’re usually less wide (2.0 slot width) and the extra $ not spent on a beefier cooler can be used for more GPUs

 

My 2070super Hybrids, run around 47-49C and the air cooled cards between 55-65C. Not a lot of cooling required at lower power-limits.

 

For Pascal run the GPU with a slight overclock (+50MHz) and at the lowest power limit supported by the card. Any Quick Return Bonus (QRB) losses will be more than offset by the savings in Power.

 

Install HfM.net (see Remote Access in my Sig). Run the card at default for a week. Drop the Power Limit to the minimum for a week and compare. You’ll be pleased with the results.

[rkv_2401]

17 hours ago, Gorgon said:

nvidia-smi -i 0 -lgc 0,1440
	Setting locked GPU clocks is not supported for GPU 00000000:06:00.0.
	Treating as warning and moving on

Nope - not supported on Linux under Pascal.

 

I assume maybe your talking about creating a custom voltage/frequency curve in Afterburner/xoc etc. which is also not possible under Linux.

Yeah, I was sure you'd already be doing this if it was possible on Linux. Just wanted to give a heads up to people who had this question and are folding on a Pascal card on Windows

[Justaphf]

Woohoo, 16bn!  Plus look at all those team LTT players running it up on page 1 of the global rankings.

[Justaphf]

On 7/26/2022 at 12:04 PM, Gorgon said:

I went through my HFM.net logs and extracted data on p18202 for the 4 GPUs and looked at my Zabbix Servers to get the Power Consumption.

GPU    Slot    TPF   Points  PPD       Power   Eff.   Adj. MSRP   Value
                                        (W)  (kPPD/W)   ($US)    (kPPD/$)
3070ti dcn04s1 01:17 434,172 4,871,747 144.8   33.6    $639.00     7.6
2070s  dcn05s0 01:43 377,230 3,164,339 106.7   29.7    $564.00     5.6
1660ti dcn05s1 03:07 280,047 1,293,905  68.6   18.9    $315.00     4.1
1070ti dcn10s0 02:53 292,204 1,459,331  89.2   16.4    $470.00     3.1

Quote

I have a Intel i7 NUC running a free ESXi 6.7 instance and have a Virtual Machine on that running Ubuntu 18.04.3 LTS that is my Zabbix Server which queries Zabbix Agents installed on my Distributed Computing Nodes. This is a bit of a Black Art but I've been doing System Management a work for over 25 years and so is second nature to me.

@GorgonDid you have a recommended guide related to setup of Zabbix, specifically for accumulating FAH/BOINC relevant metrics? I saw in the BOINC post above you mentioned it was a bit of a black art and that you did system management professionally, just wondering how much of a learning curve (for a non-systems pro) there is and what relevant information you are gathering.  Do you think it is worth setting up (as a novice) if I only have 3 boxes to monitor (after the Bifrost-style system consolidations) and that I don't really care about anything being monitored beyond the FAH relevant stats (i.e nvidia-smi query outputs)?

Edited July 29, 2022 by Justaphf
Clarification of too subjective question.

[dogwitch]

so i still not been able to get mining board working.

atm that cut 25% of my folding points.

 

[Gorgon]

On 7/28/2022 at 9:45 PM, Justaphf said:

@GorgonDid you have a recommended guide related to setup of Zabbix, specifically for accumulating FAH/BOINC relevant metrics? I saw in the BOINC post above you mentioned it was a bit of a black art and that you did system management professionally, just wondering how much of a learning curve (for a non-systems pro) there is and what relevant information you are gathering.  Do you think it is worth setting up (as a novice) if I only have 3 boxes to monitor (after the Bifrost-style system consolidations) and that I don't really care about anything being monitored beyond the FAH relevant stats (i.e nvidia-smi query outputs)?

I don't have a guide written for Zabbix as that is more of a master class in Linux System Administration and will vary significantly depending on which OS your running and the hardware on your monitored systems and especially what your interested in monitoring.

 

I would say its not worth the effort for most people but I'm a bit OCD and interested in doing the most amount of work in F@H and BOINC as efficiently as possible.

 

I need to get my Linux Folding Guide updated for Ubuntu 20.04 LTS first as that will be of more use to most people but I will share my Build Notes for Zabbix.

 

[Justaphf]

@GorgonThanks for leaving the build notes.  I'll look over them when next I get the chance, though sounds like it might be overkill for my needs.

 

On 7/30/2022 at 2:08 PM, Gorgon said:

I need to get my Linux Folding Guide updated for Ubuntu 20.04 LTS first as that will be of more use

I actually just installed from scratch 20.04.4 this evening for my second Bifrost style build.  I think I've got it down to a dozen or so CLI commands for a clean install for the purpose of just a FAH client (not counting burning the ISO to USB with Rufus and BIOS settings changes).  I do wish they would stop changing how you manage network settings every new release though.  I've given up trying to stay on top of managing that through command line, it's one of the things that's actually just easier to deal with in the desktop now.  Managing the nvidia drivers through UI also seems slightly easier now too, though I still like the control of doing most things by CLI.

[Gorgon]

21 hours ago, Justaphf said:

@GorgonThanks for leaving the build notes.  I'll look over them when next I get the chance, though sounds like it might be overkill for my needs.

 

I actually just installed from scratch 20.04.4 this evening for my second Bifrost style build.  I think I've got it down to a dozen or so CLI commands for a clean install for the purpose of just a FAH client (not counting burning the ISO to USB with Rufus and BIOS settings changes).  I do wish they would stop changing how you manage network settings every new release though.  I've given up trying to stay on top of managing that through command line, it's one of the things that's actually just easier to deal with in the desktop now.  Managing the nvidia drivers through UI also seems slightly easier now too, though I still like the control of doing most things by CLI.

Getting Zabbix working is a ton of work and getting it configured to capture relevant information is an additional pile of work.

 

I agree that the network settings are easiest done through the GUI. I usually harcode the IP, Mask, Gateway and DNS on the initial build then just manage everything through SSH via Public Keys after that.

nvidia-smi -i 0 -l 1 --format=csv,noheader --query-gpu=temperature.gpu,power.draw,clocks.gr,fan.speed

is one of my favorite commands. I have PuTTY shortcuts for all my slots so I can watch them.

[Justaphf]

3 hours ago, Gorgon said:

Getting Zabbix working is a ton of work and getting it configured to capture relevant information is an additional pile of work.

 

I agree that the network settings are easiest done through the GUI. I usually harcode the IP, Mask, Gateway and DNS on the initial build then just manage everything through SSH via Public Keys after that.

 

nvidia-smi -i 0 -l 1 --format=csv,noheader --query-gpu=temperature.gpu,power.draw,clocks.gr,fan.speed

is one of my favorite commands. I have PuTTY shortcuts for all my slots so I can watch them.

I decided I wanted to have all of it reported in HFM directly so I've started a fork to display there.  Have a hacked together proof-of-concept with basic nvidia-smi outputs being retrieved and mapped to slots. It's very much a hack job at the moment, it's display only, no recording of measurements for doing any trending/analysis and using hard coded credentials just to verify functionality.  Basically just added a SSH interface for Linux and PowerShell for Windows (though Windows I only have working local, not remote) and query based on the same polling method used to get the FAHclient stats (which I think is every 30 or 60 seconds).

nvidia-smi --query-gpu=pci.bus,fan.speed,temperature.gpu,pcie.link.gen.current,pcie.link.width.current,pcie.link.gen.max,pcie.link.width.max,pstate,power.draw,power.limit,power.default_limit,clocks.current.graphics,clocks.current.memory --format=csv,noheader,nounits

This is what I've got hooked in so far in the UI.  A number of static things there, like pcie gen and speed, that I was including to be able to quickly verify current config at a glance.

 

The SSH interface seems pretty straight forward so I'm thinking I can probably get full management of things like power limits, clock speeds, fan speeds, etc eventually.  Though my next task is to get it to use proper SSH key authentication instead of hard coded user/pw.

[BondiBlue]

I've been folding on my new GTX 1080 for about a day now, and at stock settings FAHControl is giving me an estimate of 1.1 million PPD. That's a bit lower than the 1.4 million my 1660 Ti gets, but this 1080 is much quieter, which is nice considering it's in my main PC. I won't be able to run both of these cards in my main machine at once though unless I upgrade my power supply and downgrade my SSD. The second PCIe X16 slot is disabled when the primary M.2 slot is in use. 

 

The Quadro M4000 also arrived today, and I just got it installed in a test system. It's currently working on a 10 hour WU, but FAHControl is estimating that it'll get around 450K PPD. That's comparable to my Radeon RX 570s at stock settings, but with the benefit of only needing a single slot and a 6-pin power connector. This Quadro probably won't be folding 24/7, but it'll be nice to have when I want to get a few more points. 

[IkeaGnome]

On 8/2/2022 at 5:11 PM, BondiBlue said:

The Quadro M4000 also arrived today, and I just got it installed in a test system. It's currently working on a 10 hour WU, but FAHControl is estimating that it'll get around 450K PPD.

That sounds about right. My NAS has a pair of those with no adjustments to how they run get a little under 1,000,000 a day.

[BondiBlue]

Anyone else not getting many points lately? My GTX 1080 is currently estimating only ~130K PPD, and the Quadro M4000 installed in another system is estimating ~130K PPD as well. Both cards are also not running at full load - the 1080 is only pulling 25W right now. Also, the WU that my 1080 is working on has nearly 17 hours left according to FAHControl, and that number doesn't seem to be going down. Any ideas as to what is going on? 

 

I have two other machines folding without issues right now (2x RX 570s and 1x GTX 1660 Ti), and they don't seem to have any of these weird work units. 

 

On 8/4/2022 at 12:28 PM, IkeaGnome said:

That sounds about right. My NAS has a pair of those with no adjustments to how they run get a little under 1,000,000 a day.

I used a couple Quadro M4000s for a bit of folding many years ago, and back then I remember getting around 250K-300K PPD on each one. It's nice to see that they've come down in price a fair amount over the past many months. I remember looking for one on eBay somewhere around December of last year and couldn't find a working one for less than $250 or so. I paid $150 for this one, and so far it's been mostly perfect (see above for the issues I'm having). Doesn't make too much heat either, so that's definitely nice. 

[miker07]

Has anyone else been getting WU's that are a lot lower ppd than avg for your gpu? I've tried with and without the advanced client-type tag, and I keep getting these work units that  are 1/3-1/5 normal ppd for 3080 and 3090. It's been doing it for a while, and especially with the heat it is really frustrating getting these low ppd work units, that don't seem to be running efficiently. 

[BondiBlue]

8 minutes ago, miker07 said:

Has anyone else been getting WU's that are a lot lower ppd than avg for your gpu? I've tried with and without the advanced client-type tag, and I keep getting these work units that  are 1/3-1/5 normal ppd for 3080 and 3090. It's been doing it for a while, and especially with the heat it is really frustrating getting these low ppd work units, that don't seem to be running efficiently. 

I have (see my reply above yours), but I've only been getting them on two out of five GPUs. However, one of the two affected GPUs isn't drawing much power and therefore isn't producing much heat. The other one seems to be running only slightly cooler than normal. 

[IkeaGnome]

I don't have my machine set up for remote access, so I don't know specifics on WUs. I've been all over the place lately as well though.

I don't know how hard the main machine is being hit in these dips. I usually am between 8.5 and 9m ppd.

[miker07]

3090 only getting 2.2mill on 18125 WU's, vs 7mil+ normally. Not the only horrible work unit, but a reoccurring one I've seen multiple times. Wouldn't be so bad if it was 1 every few days, but getting multiple every day anymore. May just take a break if they are going to keep pushing these horrible, inefficient WU's. They really should do better QC on wu's, considering all the free compute they get out of it. Need to work on some boinc projects anyway.

[Starman57]

Just thought I'd drop a quick msg for all of the AMD folders here.

 

I've just installed the windows Radeon drivers 22.8.1 for my 6900xt.  They fix the crappy windows performance and appear at first blush to be almost equivalent to the performance on the Linux side.  The only windows Radeon drivers that I used for folding was 21.3.1, which means it took almost 1.5 YEARS for AMD to fix the performance regression.

 

At least it's fixed.

 

Currently my Asus 6900xt is folding on 18213 is around 5.8 million to 6.0 million PPD.

[RollinLower]

anyone else suddenly doing suspiciously consistent PPD? my graph is pretty much flat!

[TVwazhere]

Nope!

[rkv_2401]

On 8/22/2022 at 5:12 AM, RollinLower said:

anyone else suddenly doing suspiciously consistent PPD? my graph is pretty much flat!

Mine is super consistent at 0PPD for the last month, RIP laptop fan

[dogwitch]

13 hours ago, TVwazhere said:

Nope!

mine a flat line!!!!

(mining mobo is dead)

plus its been almost 100 or over that this past summer. so i dont no  folding what so ever.