RX5700XT on Linux - Works!

[Gorgon]

It's Alive!

 

Following instructions from a kind user on FoldingForm I managed to get my 2 new RX5700XTs working on Ubuntu 18.04.4.

 

First you need to be running the 18.04.4 with the Hardware Enablement (HWE) so you have the 5.3 Kernal.

Next you need the Adreneline 20.10 Preview Drivers

Unpack them:

sudo tar -Jxvf amdgpu-pro-20.10-1028677-ubuntu-18.04.tar.xz

change to the directory:

cd amdgpu-pro-20.10-1028677-ubuntu-18.04

install the drivers:

./amdgpu-pro-install --headless --opencl=pal

Then add the unprivileged fahclient user to the video group:

sudo usermod -a -G video fahclient

Replace the systemd service file so that fahclient loads properly:

sudo nano /etc/systemd/system/FAHClient.service

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target

Reboot:

sudo shutdown -r now

 

If you get No GPUs found errors then add the GPUs.txt file

sudo wget http://fah-web.stanford.edu/file-releases/public/GPUs.txt -P /var/lib/fahclient/

If you get OpenCL index errors you should install the Dev package:

sudo apt install ocl-icd-opencl-dev

and you might need to remove then re-install the FAHClient packages or just set the GPU and OpenCL indexes manually to 0 and 0 for the first card and 1 and 1 for the second etc.

[Gorgon]

@justpoet @Metallus97 This!

[HaRdLy007]

How do I make sure I am running the 18.04.4 with the Hardware Enablement (HWE)?
 

18 hours ago, Gorgon said:

It's Alive!

 

Following instructions from a kind user on FoldingForm I managed to get my 2 new RX5700XTs working on Ubuntu 18.04.4.

 

First you need to be running the 18.04.4 with the Hardware Enablement (HWE) so you have the 5.3 Kernal.

Next you need the Adreneline 20.10 Preview Drivers

Unpack them:

sudo tar -Jxvf amdgpu-pro-20.10-1028677-ubuntu-18.04.tar.xz

 

change to the directory:

cd amdgpu-pro-20.10-1028677-ubuntu-18.04

 

install the drivers:

./amdgpu-pro-install --headless --opencl=pal

 

Then add the unprivileged fahclient user to the video group:

sudo usermod -a -G video fahclient

 

Replace the systemd service file so that fahclient loads properly:

sudo nano /etc/systemd/system/FAHClient.service

 

 

[Unit]
Description=Folding@Home Client
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/FAHClient /etc/fahclient/config.xml --pid-file=/var/run/fahclient/fahclient.pid --daemon
PIDFile=/var/run/fahclient/fahclient.pid
User=fahclient
RuntimeDirectory=fahclient
WorkingDirectory=~

[Install]
WantedBy=multi-user.target

 

Reboot:

sudo shutdown -r now

 

 

If you get No GPUs found errors then add the GPUs.txt file

 

wget https://fah-web.stanford.edu/file-releases/public/GPUs.txt
sudo mv GPUs.txt /var/lib/fahclient/

 

 

If you get OpenCL index errors you should install the Dev package:

sudo apt install ocl-icd-opencl-dev

 

 

and you might need to remove then re-install the FAHClient packages or just set the GPU and OpenCL indexes manually to 0 and 0 for the first card and 1 and 1 for the second etc.

 

[CWP]

46 minutes ago, HaRdLy007 said:

How do I make sure I am running the 18.04.4 with the Hardware Enablement (HWE)?
 

 

I think there is some explanation and instructions over on the Ubuntu website: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

[HaRdLy007]

14 minutes ago, CWP said:

I think there is some explanation and instructions over on the Ubuntu website: https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Thanks, much appreciated

[Gorgon]

1 hour ago, HaRdLy007 said:

How do I make sure I am running the 18.04.4 with the Hardware Enablement (HWE)?
 

 

The standard Desktop install should do this by default

[BlueJedi]

21 hours ago, Gorgon said:

Then add the unprivileged fahclient user to the video group:

sudo usermod -a -G video fahclient

I've run into a few cases where this wasn't enough to give fahclient permissions to run openCL calls. Not entirely sure what the issue is. Only solution I've found is to run FAHClient as root, which obviously isn't ideal.

[Gorgon]

OK. The system's completed 5 GPU WUs per card so far today and will be done the 6th in a couple of hours. So I've set the system to "Finish" and will next try upgrading to a 5.5 kernel as AMD has apparently made some significant improvements in OpenCL processing.

[Gorgon]

Well the Kernel 5.5 attempt was a dismal failure. It appears the preview drivers are built against 5.3 and dont work under 5.5 (DKMS errors). Which kind of sucks and I couldn't be bothered at this point to try to uninstall the preview drives, upgrade the kernel then try a re-install of the preview drivers.

 

So I've spent part of the day getting the GPUs instrumented under Zabbix so I can start seeing the long-term trends.

 

So far I'm seeing averages of 999,431 and 1,003,163PPD

[CWP]

I have never needed DKMS for my RX 550s to work properly. If you can deal with it not fully 3D-capable in the GUI, or if you installed the server edition of Ubuntu, I would suggest skipping DKMS altogether (add --no-dkms to the AMDGPU installer).

[HaRdLy007]

@Gorgon

 

Installed 18.04.4 and followed your directions and got my darned RX480 folding in Linux!!!

 

Thanks for your great guide!

[Gorgon]

24 minutes ago, CWP said:

I have never needed DKMS for my RX 550s to work properly. If you can deal with it not fully 3D-capable in the GUI, or if you installed the server edition of Ubuntu, I would suggest skipping DKMS altogether (add --no-dkms to the AMDGPU installer).

Unfortunately I need DKMS for the it87 Super I/O controller module so lm-sensors can report on the hardware sensors on the motherboard

[CWP]

Sorry I was not clear: The --no-dkms is only for the AMD installer to instruct it not to install the kernel modules for the AMD cards. It should not affect any DKMS modules that you need for other parts of the computer.

[Gorgon]

21 hours ago, CWP said:

Sorry I was not clear: The --no-dkms is only for the AMD installer to instruct it not to install the kernel modules for the AMD cards. It should not affect any DKMS modules that you need for other parts of the computer.

Ahh - OK. Thanks. I might give that a try.

[Gorgon]

On 4/2/2020 at 5:33 PM, Gorgon said:

Well the Kernel 5.5 attempt was a dismal failure. It appears the preview drivers are built against 5.3 and dont work under 5.5 (DKMS errors). Which kind of sucks and I couldn't be bothered at this point to try to uninstall the preview drives, upgrade the kernel then try a re-install of the preview drivers.

 

So I've spent part of the day getting the GPUs instrumented under Zabbix so I can start seeing the long-term trends.

 

So far I'm seeing averages of 999,431 and 1,003,163PPD

Got the 5.5 kernel to work.

 

Had to:

1. uninstall AMD drivers
2. install new kernel
3. reboot to load it
4. reinstall AMD drivers with --no-dkms flag
5. reboot

Folding resumed running WUs at checkpoints but no discernible change in production (PPD)

 

So unless there are improvements in the AMD Drivers or GROMACS code Turing (RTX2060 Super) out-produces (1.4MPPD vs 980kPPD) Navi at a similar price point.

[Gorgon]

2 hours ago, Gorgon said:

Got the 5.5 kernel to work.

 

Had to:

1. uninstall AMD drivers
2. install new kernel
3. reboot to load it
4. reinstall AMD drivers with --no-dkms flag
5. reboot

Folding resumed running WUs at checkpoints but no discernible change in production (PPD)

 

So unless there are improvements in the AMD Drivers or GROMACS code Turing (RTX2060 Super) out-produces (1.4MPPD vs 980kPPD) Navi at a similar price point.

So that was a failure. When the GPU slots finished the resumed WUs they immediately erred out on the next WUs.

 

Reverted to old kernel with the AMD driver with the --no-dkms flag but again on obtaining the next WU the GPU slots erred out.

 

Reverted to the AMD Driver without the --no-dkms flag and things appear to be working.

[Gorgon]

Things have been running for a bit and I've got the cards instrumented in Zabbix so time for some eye candy.

 

Points Per Day:

Spoiler

 

Temperature, Power and Fan Speed %

Spoiler

 

Graphic Clock

Spoiler

 

Vgfx

Spoiler

 

Note that both cards are running stock other than forcing fans speed to a fixed value to keep temperatures in the mid 70s.

 

I may play with under-volting with GTK-Wattman at a later date but want to get a good baseline first

[CWP]

On 4/4/2020 at 4:54 PM, Gorgon said:

So that was a failure. When the GPU slots finished the resumed WUs they immediately erred out on the next WUs.

 

Reverted to old kernel with the AMD driver with the --no-dkms flag but again on obtaining the next WU the GPU slots erred out.

 

Reverted to the AMD Driver without the --no-dkms flag and things appear to be working.

Hmm... I was not expecting DKMS as necessary for error-free folding. Would you mind sharing what the error message is and/or the projects that failed? My puny 550s were not happy with any work units that are larger than 110370 atoms (errors immediately with "Error invoking kernel sortShortList: clEnqueueNDRangeKernel (-5)"). I am just wondering if installing DKMS version of the drivers would solve my plight as well.

[Gorgon]

1 hour ago, CWP said:

Hmm... I was not expecting DKMS as necessary for error-free folding. Would you mind sharing what the error message is and/or the projects that failed? My puny 550s were not happy with any work units that are larger than 110370 atoms (errors immediately with "Error invoking kernel sortShortList: clEnqueueNDRangeKernel (-5)"). I am just wondering if installing DKMS version of the drivers would solve my plight as well.

All I have in my notes is:

Quote

After other GPU slot finished WU had new WU load and fail with "Bad Kinetic Energy" then the next started failing with "Particle Coordinate is nan"

 

[Gorgon]

Just a quick update. Figured out how to under-volt and overclock the 5700XTs under Linux.

 

As I did not want to test them in production folding and there is no Linux version of FAHBench bult for Core22 and the 5700XTs and Navi in general will not work on Core21 I used the AMD drivers under windows to find a stable undervolt (1150mV @ 2064MHz for the Gigabyte Card and 1151mV @2059 MHz for the PowerColor Red Dragon). Tested their stability using TimeSpy and that there was a bit of headroom left by testing 1125mV as well.

 

Then under Linux set the power-play tables with the help of this document:

cd /sys/class/drm/card0/device/hwmon/hwmon3/device/
echo "manual" > power_dpm_force_performance_level
echo "vc 2 2064 1150" > pp_od_clk_voltage
echo "c" > pp_od_clk_voltage

cd /sys/class/drm/card1/device/hwmon/hwmon4/device/
echo "manual" > power_dpm_force_performance_level
echo "vc 2 2059 1151" > pp_od_clk_voltage
echo "c" > pp_od_clk_voltage

Also set custom manual fans for the cards:

Absolute Paths:
GPU0_PATH=/sys/devices/pci0000:00/0000:00:03.1/0000:07:00.0/0000:08:00.0/0000:09:00.0/hwmon/hwmon3
GPU1_PATH=/sys/devices/pci0000:00/0000:00:03.2/0000:0a:00.0/0000:0b:00.0/0000:0c:00.0/hwmon/hwmon4

This is the upper card (GPU 0 - Gigabyte) and so needs more fan
echo 1 > /sys/class/drm/card0/device/hwmon/hwmon3/pwm1_enable
echo 176 > /sys/class/drm/card0/device/hwmon/hwmon3/pwm1

This is the lower Card (GPU 1 - PowerColor)
echo 1 > /sys/class/drm/card1/device/hwmon/hwmon4/pwm1_enable
echo 128 > /sys/class/drm/card1/device/hwmon/hwmon4/pwm1

To restore Automatic Control
echo 2 > /sys/class/drm/card0/device/hwmon/hwmon3/pwm1_enable
echo 2 > /sys/class/drm/card1/device/hwmon/hwmon4/pwm1_enable

To run full-speed:
echo 0 > /sys/class/drm/card0/device/hwmon/hwmon3/pwm1_enable
echo 0 > /sys/class/drm/card1/device/hwmon/hwmon4/pwm1_enable

[djotter]

Thanks for the write up! @Gorgon

 

I have got this working on Linux Mint 19.3 Cinnamon with the 5.3.0-46-generic kernel, running an AMD RX580. Only a single Core22 work unit done so far today as they seem to be few and far between at the moment.

 

Only change I made was "--opencl=legacy" flag on the driver install.

 

One note, how do I set the FAHClient.service to start on boot?

 

Thanks!

[CWP]

3 hours ago, djotter said:

Thanks for the write up! @Gorgon

 

I have got this working on Linux Mint 19.3 Cinnamon with the 5.3.0-46-generic kernel, running an AMD RX580. Only a single Core22 work unit done so far today as they seem to be few and far between at the moment.

 

Only change I made was "--opencl=legacy" flag on the driver install.

 

One note, how do I set the FAHClient.service to start on boot?

 

Thanks!

You can use:

systemctl enable FAHClient.service

to allow it to start on boot.

[Gorgon]

9 hours ago, djotter said:

Thanks for the write up! @Gorgon

 

I have got this working on Linux Mint 19.3 Cinnamon with the 5.3.0-46-generic kernel, running an AMD RX580. Only a single Core22 work unit done so far today as they seem to be few and far between at the moment.

 

Only change I made was "--opencl=legacy" flag on the driver install.

 

One note, how do I set the FAHClient.service to start on boot?

 

Thanks!

It should do that as part of the FAHclient on Install if you told it to do that. I'm surprised if it didn't on Mint as it is Debian-Based.

[djotter]

2 hours ago, Gorgon said:

It should do that as part of the FAHclient on Install if you told it to do that. I'm surprised if it didn't on Mint as it is Debian-Based.

It is probably more that I somehow disabled auto-startup while messing around with trying to get it to work. Yours wasn't the first guide I tried

[Gorgon]

1 hour ago, djotter said:

It is probably more that I somehow disabled auto-startup while messing around with trying to get it to work. Yours wasn't the first guide I tried

Glad to hear its working.

 

I'll clean up my notes and add an AMD GPU section to my Linux Folding HOWTO.

 

Did you try any under-volting/over-clocking or manual fan control?