Howto: Ubuntu 22 Folding Rig

[Gorgon]

Why Linux:

• It will run well on older hardware that can be re purposed for a dedicated folding machine.
• You can disable automatic updates.
• Though Core22 has narrowed the performance gap between Windows and Linux anecdotal evidence still suggest there is a slight performance increase Folding on Linux.
• I have found it to be more stable than Windows with months of up-time. YMMV

Why Ubuntu:

• It is well supported by a large Community and the Long Term Support (LTS) version is supported by Canonical until April 2027.

Prerequisites:

• Pretty much any 2GHz or faster processor with a minimum of 2-3GB DDR3 and a PCIe 2 x16 slot should work.
• If you’re not running DHCP on your router you’ll need an available IP Address, and to note your Subnet Mask and Gateway Address.
• NVidia GTX or RTX GPU (AMD GPUs are NOT covered here yet, but if anyone wants to contribute please do so)

OS Install (Ubuntu Desktop 22.04.2 LTS):

This is the LTS Desktop version. More adventurous types may want to consider installing the LTS Server version for a less cluttered install. There are several other guides on how to setup a Linux Folding Rig (see the References Section at the end)

          Note: Ubuntu 19 removed support for Python2 so FAHControl will not work without some effort

1. Download Ubuntu 22.04.2 LTS Desktop ISO File
2. Burn the ISO to a USB Flash Drive using Rufus or similar tool
3. Boot from the USB key and start the install
4. For US English just accept the defaults
5. Partition the whole disk as a single “Root” partition, don’t bother creating a separate swap partition.
6. Select a “Minimal” install and to “Add 3rd-Party Drivers”
7. Use the default Gnome Desktop (GDM3) for now, it can be changed later. To use “CoolBits” to Over or Under-Clock or change Fan speed control you will need to disable the newer “Wayland” X Server in favor of the classic Xorg Server (more later)

OS Configuration:

Spoiler

• Add a Shortcut to Terminal:

1. Click on the "Show Applications" Icon in lower Left Desktop
2. Type "Terminal" in the Search Box to show the Terminal Application
3. Right-Click on the Terminal Application and select "Add to Favorites" to add it to the Desktop.

• (Optional) Add OpenSSH Server package for remote access to CLI. Open a Terminal Window and type:

sudo apt install openssh-server

• Add lm-sensors (Hardware Monitoring) package (Run "sensors-detect" to configure and "sensors" to view hardware sensors)

sudo apt install lm-sensors
sudo sensors-detect
sensors

• (Optional) Add psensor Application to monitor sensors from GUI

sudo apt install psensor

 

Graphics Drivers:

Spoiler

It is no longer required to manually add the PPA Graphics Drivers Archive (Repository) as this option is now included as an option as part of the Install (“Add 3rd-Party Drivers” above).

• List the available NVidia drivers and install the most current GPU Driver (v530 at the time of writing):

sudo apt list nvidia-driver* (and pick the newest version to install)							
sudo apt install nvidia-driver-530

This should install all the additional NVidia packages required.

OpenCL:

Spoiler

Though when using an NVidia GPU you should be using CUDA to run tasks and OpenCL SHOULD only be needed for AMD GPUs it is a good practice to install OpenCL regardless.

• Install the OpenCL Development Package and CLinfo:

sudo apt install ocl-icd-opencl-dev clinfo

and restart the computer then run CLinfo:

sudo shutdown -r now
clinfo

 

Folding Client:

Spoiler

• Copy the Most recent Debian Package from foldingathome.org:

wget https://download.foldingathome.org/releases/public/release/fahclient/debian-stable-64bit/v7.6/fahclient_7.6.21_amd64.deb

Install the Package as a Service (Daemon):

sudo dpkg -i fahclient_7.6.21_amd64.deb

Specifying your Folding username, Team ID (223518), Passkey and specifying "Full Power" for folding and "Auto Start."

Optional - Install FAHControl:

To install the Folding at Home Control application locally on the system requires a bit of work as this application is written using Python 2 and Ubuntu 20.04 and later only comes with Python version 3 installed. There is a Beta FAHControl re-written to use Python3:

It will, however, need some tweaks to get it running due to changes since Python version 3.9 (Courtesy @justpoet) :

Quote

For those that grab the Python3 FAHControl package for Linux and are running as up to date as possible for Python, once it installs, you'll need to either force running under older Python 3, or go edit: /usr/lib/Python3/dist-packages/fah/util/OrderedDict.py on line 23, where it imports MutableMapping, you change:
"from collections import MutableMapping as DictMixin" to: "from collections.abc import MutableMapping as DictMixin"

You can thank Python for breaking their own python3 implementation with 3.10+ for this.  But, then after the edit it'll work correctly again.

Create a Desktop Favorite for FAHControl:

1. Click on the "Show Applications" Icon in lower Left Desktop
2. Type "FAHControl" in the Search Box to show the Folding@Home Control Application
3. Right-Click on the Terminal Application and select "Add to Favorites" to add it to the Desktop.

Configure Folding Client:

1. Configure GPU and, optionally, CPU Slots
2. Configure Advanced Options (Beta Flag, Pause on Start, Cause Preference ...)

Configuring X-Windows and GPU:

Spoiler

CoolBits

CoolBits are binary flags used by the NVidia GPU driver to enable some normally locked functionality. For Folding the bits we typically use are Bit 2 (4), which enables manual fan control and Bit 3 (8), which enables GPU Shader and Memory overclocking.

 

Since Ubuntu 19.04 Canonical has stopped using the classic Xorg  X-Server and is now configuring the OS to use a newer X-Server known as “Wayland.” Unfortunately CoolBits still requires using the Classic X-Org X-Server so we must disable Wayland to enable CoolBits.

Disable Wayland:

sudo nano /etc/gdm3/custom.conf
(changing: #WaylandEnable=false to WaylandEnable=false)
(Press <Ctrl>+X to exit the editor saving the file)

There are multiple ways to set CoolBits but by far the easiest is:

sudo nano /usr/share/X11/xorg.conf.d/10-nvidia.conf
adding: Option "Coolbits" "12"
(Press <Ctrl>+X to exit the editor saving the file)

The setting will take effect after the next time the system is restarted.

But this will only work for a single GPU in a system. For multiple GPUs just use the following to create a xorg.conf file with CoolBits enabled:

sudo nvidia-xconfig -a --cool-bits=12 --allow-empty-initial-configuration

• nvidia-settings is the utility provided with the NVidia driver package to control the GPU. Once Coolbits is enabled it can be run from X-Windows to manually control the fan and/or apply GPU and Memory overclocks. There is also a command-line version (see the Advanced Section)
• nvidia-smi is the Nvidia Systems Management Interface used to query or program the GPU. See the references section below. The most common use is to set a higher or lower power limit for the GPU:

sudo nvidia-smi -i 0 -pm 1; nvidia-smi -i 0 -pl 150

this first sets "Persistent Mode" enabling changes to the power level to persist after the current job is terminated and, next, sets the power level for GPU 0 to 150W (entering an unrealistic value here will throw a helpful error detailing the valid range of values for the referenced GPU)

Startup Script:

The above settings, once "seasoned to taste" can be stored in the Desktop Startup Script and thus be run at startup.

System Administration 101:

Spoiler

"The Terminal is your best friend" - Gorgon 2018

Fixing the System Daemon

The default system daemon (etc/init.d/FAHClient) installed with the Folding at Home client software has an error that prevents it stopping or restarting the FAHClient service.

    1. Create a patch file:

sudo nano ~/FAHClient.patch
--- FAHClient.orig      2023-06-24 21:29:12.319689099 -0400
+++ FAHClient   2023-06-24 21:30:21.321954003 -0400
@@ -121,6 +121,7 @@
     fi

     pid=$!
+    echo $pid > $PID
     if wait_for_log_change "$LOG_STATUS" $pid; then
         echo "OK"
     else
(Press <Ctrl>+X to exit the editor saving the file)

   2. Kill the running service:

sudo killall FAHClient

   3. Apply the patch:

sudo patch /etc/init.d/FAHClient ~/FAHClient.patch

   4. Reload the System Control Daemon:

sudo systemctl daemon-reload

   5. Start the FAHClient Service:

sudo systemctl start FAHClient

Starting, Stopping and Restarting Service:

sudo service FAHClient start
sudo service FAHClient stop
sudo service FAHClient restart

Package Updates: (Aptitude, dpkg, apt ...)

To upgrade the packages on your system you need to periodically run:

sudo apt update
sudo apt upgrade –y
sudo shutdown –r now
sudo apt auto-remove -y

Monitoring System Health:

Use htop, ps, sensors, ...

Monitoring GPU Performance:

Use nvidia-settings, nvidia-smi, nvtop ...

Monitoring Folding Performance:

Use FAHClient, hfm.net ...

Advanced Topics:

Spoiler

Multiple GPUs:

• In most standard mid and full-tower ATX cases with dual-fan cards the lower card will cook the upper card. I usually install the higher performing card in the lower slot so it is not breathing the hot air from the adjacent card. I have noticed that running the fans on the lower card about 10% above the stock value will result in the upper card getting cooler air and running at a lower temperature and boosting higher.
• Remote Management
• Using a Command Line Interface (CLI) Use your favorite SSH application to access the folding machine. I use PuTTY.
• You can use a Graphical User Interface (GUI) from a Remote System. See:

https://linustechtips.com/topic/990176-howto-remotely-access-your-folding-systems-part-1-fahcontrol/

Headless Operation:

In order to run headless (without a display attached to the system) you will need to either install HDMI or DVI "Dummy Plugs" to spoof a display or trick nvidia-settings utility so you can apply an overclock or manually control the fans on a GPU. The trickery involves running:

user@kvm1:~# sudo ps -ef | grep auth
root      1772  1769  0 Sep16 tty1     00:00:14 /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth /run/user/131/gdm/Xauthority -background none -noreset -keeptty -verbose 3

to find the location of the Xauthority file then explicitly point to this file when invoking nvidia-smi from the Command-Line Interface (CLI):

sudo DISPLAY=:0 XAUTHORITY=/run/user/131/gdm/Xauthority nvidia-settings -a /GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=80

Here we are changing the Fan Control to Manual and then setting the first Fan to 80%

To monitor a GPU in a separate SSH session type:

gordonb@kvm1:~$ nvidia-smi -i 0 -l 1 --format=csv,noheader --query-gpu=temperature.gpu,power.draw,clocks.current.sm,fan.speed
63, 89.82 W, 2025 MHz, 80 %
63, 89.47 W, 2025 MHz, 80 %
63, 93.58 W, 2025 MHz, 81 %
63, 93.72 W, 2025 MHz, 81 %
63, 97.35 W, 2025 MHz, 80 %
63, 89.27 W, 2025 MHz, 81 %
63, 93.68 W, 2025 MHz, 81 %
...

to display the GPU Temperature, Power Draw, Shader Clock and, fan speed, on one line once a second.

Running BOINC and FaH:

Dual-Booting Linux & Windows:

1. Install Windows 1st
2. Disable Fast Boot
3. Shrink Partition
4. Install Linux in free space

References:

Spoiler

Build Guides:

• Chris21010's at EVGAform's Guide to Multi GPU on Linux Mint
• Puget Systems' Dr. Donald Kinghorn: The Best Way To Install Ubuntu 18.04 with NVIDIA Drivers and any Desktop Flavor
• TenneseeTony's GUIDE: Linux Mint 18.3 Cinnamon with Nvidia GPU(s)

Folding Client Install:

• Folding@Home: Alternative Downloads
• Folding@Home: Manual Installation (Advanced)
• iceflatline's Install and Configure Folding@home on Ubuntu Server

Proprietary GPU Drivers:

• PPA Graphics Drivers Team

NVidia Graphics Drivers:

• NVidia Graphics Driver README
• nvidia-smi documentation
• NVidia Management Library Reference Manual
• NVML API Reference Guide
• NVidia Tools for Managing GPU Clusters
• NVidia CUDA Toolkit Documentation
• ArchLinux Wiki: NVidia Tips & Tricks
• NVidia Headless Server

Ubuntu Documentation:

Hardware Monitoring:

• NVidia: FAN Control Ubuntu 18.04

IT87 Kernel Module (All Gigabyte, and some Asus, Motherboards):

Kernel.org: IT87 Kernel Module Default Supported Chips

Phoronix: IT87 Linux Driver For Supporting Many Motherboard Sensors Is Facing Death

Leve1Techs: I patched the IT87 (Temperature monitoring) driver, need some help testing it

GitHub: Read-Only Archive of Guenter Roeck's Experimental IT87 Driver

 

Ubuntu 18.04 LTS Install Guide here

[Gorgon]

Reserved for Updates

[justpoet]

Probably worth just putting in the edits I mentioned for FAHControl, since linking to the comment doesn't always actually go to that post.

[Gorgon]

2 hours ago, justpoet said:

Probably worth just putting in the edits I mentioned for FAHControl, since linking to the comment doesn't always actually go to that post.

Thanks - I'm just running through the instructions on a new Testbed. You interested in contributing a section on Installing Modern AMD Drivers for Folding? Or expanding how to actually setup FAHControl from the Beta version?

[justpoet]

2 hours ago, Gorgon said:

Thanks - I'm just running through the instructions on a new Testbed. You interested in contributing a section on Installing Modern AMD Drivers for Folding? Or expanding how to actually setup FAHControl from the Beta version?

There's really nothing more than that on the "Beta", since it isn't being developed further as V7 is done.  FAH is being rewritten to use everything in the web interface and not have a python FAHControl in V8, which FAH considers the real beta (and I haven't played with it yet):

https://foldingathome.org/foldinghome-v8-client-guide/?lng=en