Folding - Beating the Heat in Summertime

[Gorgon]

Here in Eastern Ontario we have Time of Use (ToU) Electricity Rates that increase the price of Electricity during "Mid Peak" and "On Peak" times during the day.

 

If your running multiple GPUs Folding you've likely noticed the increases in your Electricity Bill and here are some hints on what you can do to keep folding during the hot months and not live in a sweat box and pay through the nose.

 

The very first change you should contemplate is reducing the GPU's Power Limit. During normal folding in the Fall, Winter and Spring months with 10-12 GPUs I run my cards about 20% above the minimum Power Limit. I find this gives me around 90% of the PPD for 30-40% less power. NVidia Pascal and Turing GPUs, like modern CPUs, tend to run close to the edge of stability and at "stock" 100% Power Targets are not very efficient. During the summer months I run the GPUs at the minimum Power Limits for an even greater decease in Power with only a slight lessening of PPD.

 

While using a GUI like the NVidia Control Panel or MSI Afterburner you can lower the power limits you can also use the NVidia System Management Interface:

nvidia-smi -i 0 -pl 90

Here we are forcing the power-limit on GPU 0 to 90W. This will work in either Windows or Linux.

 

You will also want to enable "Persistence Mode" so the Power Limit is not reset to the Default after the current task is finished. This command:

nvidia-smi -pm 1

will enable persistence mode for all GPUs. A specific GPU can be specified also using the "-i <X>" switch where "<X> is the Device ID shown in

nvidia-smi

output.

 

Scheduling Folding only during less expensive ToU periods can be accomplished but will take a bit of work to setup. Here we leverage the ability of the FAHClient program to send commands to a running "Slot" (GPU).

 

What we need to do is tell the GPU slots to "Finish" about half the average time it takes for that GPU to complete a Work Unit (WU) before the start of the ToU rate period. In my case my GPUs typically complete a WU in about 3 hours when running at minimum power limits so I schedule the "Finish" command to run at 05:30 as the ToU period here starts at 07:00. This results in the GPUs finishing their current WUs then "Pausing." When the ToU period ends you schedule a "Un-Pause" command to be sent to resume folding. In my case the ToU rates end at 19:00.

 

In Linux you could use the "cron" scheduler thus:

# use /bin/bash to run commands, instead of the default /bin/sh
SHELL=/bin/bash
# Time of Day Folding - Summer
#Mid-Peak 07:00 - 11:00
30 5 * * 1-5 /usr/bin/FAHClient --send-finish
#On-Peak 11:00-17:00
#30 9 * * 1-5 /usr/bin/FAHClient --send-finish
#Mid-Peak 17:00 - 19:00
#00 17 * * 1-5 /usr/bin/FAHClient --send-unpause
#Off-Peak 19:00-07:00
00 19 * * 1-5 /usr/bin/FAHClient --send-unpause

Windows users can utilize the Command Scheduler and the same commands:

'C:\Program Files (x86)\FAHClient\FAHClient.exe --send-finish'
'C:\Program Files (x86)\FAHClient\FAHClient.exe --send-unpause'

The result looks something like this:

Here 5 of my systems are scheduled to "Finish" at 05:30 and by 07:00 my GPU power consumption has dropped from 1kW to 300W with the last couple of Slots "Pausing" by ~07:20. Note: I am also running some BOINC tasks on the Blue and Dark Red systems accounting for the periodic power spikes.

 

And here's what it looks like over the last couple of weeks:

So you can see I run 24hours during weekends and holidays and about 12 hours (19:00-07:00) during weekdays when ToU rates are in effect.

 

[CWP]

Some time ago, I mashed this bash script together to set my nVidia cards to 75% of the default power draw:

PERCENT=0.75
for UUID in `nvidia-smi -L |cut -d: -f3 |cut -d\) -f1 |awk '{print $1}'`; do
	DEFAULT=`nvidia-smi -i $UUID -q -d POWER |grep 'Default Power' |cut -d: -f2 |awk '{print $1}'`
	MIN=`nvidia-smi -i $UUID -q -d POWER |grep 'Min Power' |cut -d: -f2 |awk '{print $1}'`
	SET=`echo $DEFAULT $MIN $PERCENT |awk '{printf "%f", ( $1 * $3 < $2 ? $2 : $1 * $3 )}'`
	nvidia-smi -i $UUID -pl $SET
done

It can handle any number of cards with varying power limits, as well as automatically capping the low end to the minimum supported power level.

 

EDIT 2023/11/09:

Recent versions of nvidia-smi 5xx.xx.xx (guessing in the 530s) changed the query output so it broke my script. Here is an updated one:

PERCENT=0.75
for UUID in `nvidia-smi -L |cut -d: -f3 |cut -d\) -f1 |awk '{print $1}'`; do
        DEFAULT=`nvidia-smi -i $UUID -q -d POWER |grep 'Default Power' |head -n1 |cut -d: -f2 |awk '{print $1}'`
        MIN=`nvidia-smi -i $UUID -q -d POWER |grep 'Min Power' |head -n1 |cut -d: -f2 |awk '{print $1}'`
        SET=`echo $DEFAULT $MIN $PERCENT |awk '{printf "%f", ( $1 * $3 < $2 ? $2 : $1 * $3 )}'`
        nvidia-smi -i $UUID -pl $SET
done

Keeping the old version here as well for the older versions of nvidia-smi, just in case people are still folding with older GPUs.