HOWTO: Time of Use Scheduling for Distributed Computing

[Gorgon]

Summary

A method for implementing scheduling Distributed Computing (DC) tasks to take advantage of Time-of-Use (ToU) Electricity rates.

Introduction

Once you start running multiple Central Processing Units (CPUs) and Graphics Processing Units (GPUs) working on Distributed Computing (DC) tasks the energy consumed can become large and Utility power bills a significant burden. Many Utility power providers offer Time of Use (ToU) rates where electricity is priced most expensively at “On-Peak” times, more expensively at “Mid-Peak” times and least expensively at “Off-Peak” times. Depending on your goals and finances it may be convenient to suspend DC tasks during more expensive ToU periods and resume them during less expensive ToU periods.

For example, my electricity provider in Eastern Ontario, Canada, defines ToU rates for summer (May 1st through October 31st) and winter (November 1st through April 30th) weekdays:

Season	Start Time	Period	Cost $/kWh
Summer	07:00	Mid-Peak	0.102
Summer	11:00	On-Peak	0.151
Summer	17:00	Mid-Peak	0.102
Summer	19:00	Off-Peak	0.074
Winter	07:00	On-Peak	0.151
Winter	11:00	Mid-Peak	0.102
Winter	17:00	On-Peak	0.151
Winter	19:00	Off-Peak	0.074

 

As can be seen the electricity rate more than doubles during the On-Peak period compared to Off-Peak and is 39% more expensive during Mid-Peak periods.

The rationale for the periods is that the “baseload” power during off-peak periods is supplied from always on-line Nuclear and Hydro-electric generation and during Peak usage more expensive generation using Natural Gas is employed. During summer the On-Peak period corresponds to the hottest period of the day and was likely established to encourage consumers to reduce set-points on Air Conditioning (A/C) to conserve electricity. The Winter On-Peak rates appear to target times when households are waking or arriving home from work and so appear to be encouraging use of high power devices such as drying machines and stoves earlier and later in the day.

BOINC

The Berkeley Open Infrastructure for Network Computing (BOINC) client supports suspending applications and no penalty is incurred for suspending then resuming a task. The BOINC Manager application includes functionality for scheduling when during the day tasks should run:

 

Note, however, that you can only schedule one period of activity per day and so may be unsuitable for more complex use cases. To accommodate needing to schedule more than one BOINC suspend/resume cycle in a day we can leverage the boinccmd application which can be used to send commands to a locally attached BOINC client or a remote client by specifying its IP Address and, optionally, a password.

 

The BOINC client must first be configured to allow Remote Procedure Calls (RPCs) on the target client, the source of the RPC call must be specified and, optionally, a password.

 

Linux/MacOS

The examples used here are for Linux (Ubuntu 22.04) but can be used on a Mac.

 

There is no command to suspend or resume all BOINC applications running on a host but the running applications can be enumerated using the tool and the results used to suspend or resume all running applications.

 

For a locally attached client the syntax is:

for url in $(boinccmd --get_project_status | sed -n 's/\s*master URL: //p'); do
  boinccmd --project ${url} suspend;
done

For a remote client the syntax is:

for url in $(boinccmd –-host <IP> [--passwd <Password>] --get_project_status | sed -n 's/\s*master URL: //p'); do
  boinccmd –-host <IP> [--passwd <Password>] --project ${url} suspend;
done

On a local host the code can be placed in a script and launched at appropriate times using the cron utility.

Create the script:

sudo nano ~/all_boinc_tou.sh

Adding:

#!/bin/bash
for url in $(/usr/bin/boinccmd --get_project_status | sed -n 's/\s*master URL: //p'); do
  /usr/bin/boinccmd --project ${url} $1;
done

Type <Ctrl>+x to exit the editor saving the file.

Make the script executable:

sudo chmod +x ~/all_boinc_tou.sh

 

Edit the root crontab:

sudo crontab –e

appending:

# use /bin/bash to run commands, instead of the default /bin/sh
SHELL=/bin/bash
# Time of day BOINC
#  Summer: May 1st - Oct 31st
#  Mid-Peak 07:00 - 11:00
00 07 * * 1-5 /root/all_boinc_tou.sh suspend
#  On-Peak 11:00-17:00
#00 11 * * 1-5 /root/all_boinc_tou.sh suspend
#  Mid-Peak 17:00 - 19:00
#00 17 * * 1-5 /root/all_boinc_tou.sh resume
#  Off-Peak 19:00-07:00
00 19 * * 1-5 /root/boinc_tou.sh resume
#
#  Winter: Nov 1st - Apr 30th
#  On-Peak 07:00 - 11:00
#00 07 * * 1-5 /root/all_boinc_tou.sh suspend
#  Mid-Peak 11:00-17:00
#00 11 * * 1-5  /root/all_boinc_tou.sh resume
#  On-Peak 17:00 - 19:00
#00 17 * * 1-5 /root/all_boinc_tou.sh suspend
#  Off-Peak 19:00-07:00
#00 19 * * 1-5 /root/all_boinc_tou.sh resume

Here we have enabled BOINC applications to run during the current Season (summer) during Off-Peak hours from 7pm until 7am weekdays.

 

Exit saving the file.

 

If we wanted to run BOINC during the Mid-Peak as well as Off-Peak during the Summer we could do so by commenting out (prepending a “#” at the start of) the Mid-Peak line and un-commenting (removing the “#”) at the start of the On-Peak line.

 

Windows

Under Windows we can use the Task Scheduler to created two Scheduled Tasks, one to Suspend BOINC Tasks and the other to Resume them.

 

Open the Task Scheduler (Start Menu | Windows Administrative Tools):

 

In the Actions Pane at the Right click on “Create Task …”

In the General Tab:

 

Give the Task a Name and Description and make sure to select “Run whether the user is logged on or not”

In the “Triggers” tab we create triggers to specify when we want the “Action” (suspend command) to run:

 

Click on the “New …” button to define a new Trigger:

 

Select “Weekly” and check “Monday” through “Friday” to have the Task run on Weekdays and set the
“Start” to the time of day that BOINC should be suspended.

 

Click “OK” to save the Trigger.

 

Create Triggers for each time when the BOINC Projects should be suspended.

 

Next we create an Action to suspend each BOINC Project we are running. These will require the Master URLs for each Project. These can be found by running in a Command Prompt:

“C:\Program Files\BOINC\boinccmd.exe” --get_project_status | find "master URL"

 

The leading “http(s)://” should be excluded.

 

Click on the “Actions” tab:

 

In the “Actions” tab click on “New” to create a new Action:

 

Entering:

Program/script: "C:\Program Files\BOINC\boinccmd.exe" including the quotes.
Add arguments (optional): add: --project <master URL> suspend

where “<master URL>” is one of the Master URLs found above using the “get_project_status” command. In this example we use:

--project einstein.phys.uwm.edu suspend

To suspend Einstein@Home.

 

Click “OK” to save the Action.

 

Add additional Actions for each Project you wish to follow this schedule.

 

Click “OK” to save the Scheduled Task. You will be prompted for your Password so the Task can be run even if you are not currently logged on to the system.

Following the same steps now create another Scheduled Task to resume the projects when desired changing the “argument” in the “Action” to:

“--project <master URL> suspend” to “--project <master URL> resume”

 

Holidays, when ToU rates do not apply. can be accommodated by editing the Triggers a few days in advance to not include the holiday and changing it back after the Holiday.

 

Folding at Home

For Folding at Home (F@H) things are a little more complicated. Folding at Home has a Quick Return Bonus (QRB) to encourage prompt processing of Work Units (WUs are like tasks in BOINC). If a WU is suspended the QRB, which is exponential in nature to reward participants with fast CPUs and GPUs, can dramatically decrease to the point where “dumping” the WU and starting another would net more points. “Dumping” is discouraged as it can interrupt the flow of work and progress of the Science being done. In informal testing pausing GPU WUs for more than a couple of hours is counter-productive.

A better strategy for longer intervals is to evaluate the average duration of WUs for the GPUs in your system and then set the tasks to “finish” at 1/2 to 2/3 the average duration before the ToU period starts. For example, if tasks on average take 3 hours to complete on your GPU and your ToU period ends at 07:00 then you could set your WUs to finish at 05:30. In this manner, on average, half the tasks would finish between 05:30 and 07:00 and the other half between 07:00 and 08:30. This can be adjusted to meet your needs as required.

The FAHClient application can be used to send commands to a locally attached F@H client or a remote client.

 

For a locally connected client the command to pause tasks is:

FAHClient --send_pause

to finish tasks is:

FAHClient --send_finish

and to resume paused or start new tasks:

FAHClient --send_unpause

For remote Clients you need to first configure Remote Access and then connect to the remote client using either the FAHClient remote console port (TCP 36333) and use a scripting language such as expect or Secure Shell (SSH) to send a remote command.

 

We will use SSH and assume that the Clients are configured to authenticate using the SSH Public Key of the user on the system initiating the command.

 

To pause tasks on a remote system we use:

ssh <hostname> ‘FAHClient --send_pause’

to finish tasks on a remote system we use:

ssh <hostname> ‘FAHClient --send_finish’

and to resume paused or start new tasks on a remote system:

ssh <hostname> ‘FAHClient --send_unpause’

Linux

On a local host the commands can be executed on the schedule using the cron utility with the commands placed in the root crontab.

 

Edit the root crontab:

sudo crontab –e

appending:

# use /bin/bash to run commands, instead of the default /bin/sh
SHELL=/bin/bash
# Time of Day Folding
#  Summer: May 1st - Oct 31st
#  Mid-Peak 07:00 - 11:00
30 5 * * 1-5 /usr/bin/FAHClient --send-finish
#00 7 * * 1-5 /usr/bin/FAHClient --send-pause
#  On-Peak  11:00-17:00
#30 9 * * 1-5 /usr/bin/FAHClient --send-finish
#00 11 * * 1-5 /usr/bin/FAHClient --send-pause
#  Mid-Peak 17:00 - 19:00
#00 17 * * 1-5 /usr/bin/FAHClient --send-unpause
#  Off-Peak 19:00-07:00
00 19 * * 1-5 /usr/bin/FAHClient --send-unpause
#
#  Winter: Nov 1st - Apr 30th
#  On-Peak  07:00 - 11:00
#30 5 * * 2-5 /usr/bin/FAHClient --send-finish
#00 7 * * 1-5 /usr/bin/FAHClient --send-pause
#  Mid-Peak 11:00 - 17:00
#00 11 * * 2-5 /usr/bin/FAHClient --send-unpause
#  On-Peak  17:00 - 19:00
#30 15 * * 2-5 /usr/bin/FAHClient --send-finish
#00 17 * * 2-5 /usr/bin/FAHClient --send-pause
#  Off-Peak 19:00 - 07:00
#00 19 * * 2-5 /usr/bin/FAHClient --send-unpause

Here we have enabled F@H tasks to run during the current Season (Summer) during Off-Peak hours from 7pm until 7am weekdays setting the running tasks to “finish” starting at 05:30. Exit saving the file.

 

If we wanted to run F@H during the Mid-Peak as well as Off-Peak during the Winter we could do so by commenting out (prepending a “#” at the start of) the Mid-Peak line and un-commenting (removing the “#”) at the start of the On-Peak line.

 

Windows

In Windows we can use the Task Scheduler as described previously to create Finish and Resume Tasks which include in the “Actions” tab:

Program/script: "C:\Program Files (x86)\FAHClient\FAHClient.exe" (including the quotes)
Add arguments (optional): --send-finish (OR –send-unpause as appropriate.)

 

Multiple GPUs

If you have multiple GPUs in a system and wish to specify differing times to Finish WUs then simple append the slot ID at the end the "finish" command to specify the GPU to finish at that time.

 

Conclusion

We have maximized our production in Distributed Computing applications (BOINC and/or Folding@Home) while minimizing our electricity usage during more expensive Time-of-Use rates.

Here you can see the results across seven systems with multiple GPUs with summer season ToU scheduler settings:

We are just running Distributed Computing applications during the least expensive ToU rates here and setting Folding@Home to “finish” Work so on average power consumption is greatly decreased when the On-Peak period starts at 07:00.

 

[Gorgon]

Reserved for further Information

[rkv_2401]

Great thread! BOINC Manager's built-in time-of-use scheduler was enough for me, thankfully.

 

Also, I used two batch files and task manager with the setup here - (https://www.reddit.com/r/EtherMining/comments/mc32qp/how_to_automatically_reapply_msi_afterburner/) to turn on and off my custom Afterburner profiles for when it starts and ends computing...

[Gorgon]

3 hours ago, rkv_2401 said:

Great thread! BOINC Manager's built-in time-of-use scheduler was enough for me, thankfully.

 

Also, I used two batch files and task manager with the setup here - (https://www.reddit.com/r/EtherMining/comments/mc32qp/how_to_automatically_reapply_msi_afterburner/) to turn on and off my custom Afterburner profiles for when it starts and ends computing...

That sub-reddit is private an inaccessible

[rkv_2401]

4 hours ago, Gorgon said:

That sub-reddit is private an inaccessible

Yeah, Reddit subs have gone offline for a couple of days or more to protest some API pricing thing. I used the cached version (https://webcache.googleusercontent.com/search?q=cache:8cOJOTgXB98J:https://www.reddit.com/r/EtherMining/comments/mc32qp/how_to_automatically_reapply_msi_afterburner/&cd=10&hl=en&ct=clnk&gl=au&client=firefox-b-d) to see the instructions.

 

Basically, creating a .bat file with the text"C:\Program Files (x86)\MSI Afterburner\MSIAfterburner.exe" -ProfileX where X is the profile you want to load, and using Task Scheduler to run that .bat file at a certain time every day.