TUTORIAL FREE PPD Cloud GPU Tesla P100/V100 F@H! FINAL edition V2

Metallus97 · October 10, 2019

---UPDATE IN PROGRESS---

First of all A HUGHE THANKS TO @Gorgon, @proptied and @sazrocks who helped me figure some stuf out and contributed some tip, tricks and commands to this guide!

Another HUGE THANKS to @jctappel67 and @WhyKickAmooCow. Those guys did the advanced part of the guide. This will be a lot more involved and require more fiddling around with the configuration. However I highly recommend doing it if you have the time! @WhyKickAmooCow has written some custom scripts to set up the instances for you as a preemtible one. That way you will save 50% of your money. Normally those preemtible instances don't run 24/7 and get shut down by google after some time (depending on demand). BUT with his awesome work you will be able to gracefully stop and directly restart the machines. Everything done automatically!

So go over here for that:

Continue here for the basic version using just (more expensive) normal VMs. The rest about this guide beneath the installation and setting up of things stays the same for both normal and preemtible instances!

Basics and Introduction

I am using Google Cloud Engine and other services sometimes for my work as it includes MUCH Data science etc. Setting up an account there and getting a Debian (Linux) VM going with F@H at full blast on a big Nvidia Tesla is pretty easy and can be done in like 30min. We will start with google since its the easiest, but ill include a more general approach you can use on every Debian server.

Basic Infos

Google clouds virtual machines are paid by usage time. Sign up for the free trial will give you 365 days of time and a free budget of $300 to spend on computing time. Since its a payment system by the hour you cant loose much after the free budget is over. Simply calculate how much time you have and deactivate the machines accordingly.

In this guide I will show how to create some machines which max out the power but burn through the 300€ relative slowly. I calculated the prices. Best PPD for PPD (PerformancePerDollar for PonitsPerDay seems to be a Nvidia Tesla P100 GPU. It pulls about 1,4mPPD and is a little more than 1$/h of usage time. Therefore you can get a little more than a week of free big PPD from your account. Also nobody stops you from creating more than one account. BUT I am responsible for nothing!

Therefore this is a nice, easy and free way of boosting your PPD!

RIIIGHT so letzzzze go ~~MARIO~~

(The screenshots will be German, because I didn’t want to reset my whole google accounts language, but ill mark + tanslate everything so its easy to follow)

1. You need a Google Account. You can use your existing one or create a new one (multiple are also possible, but I am NOT responsible for anything :P)

https://cloud.google.com/ sing in, give them your Card (there they write about not charging you until you’re done with the free 300$ budget. If you have google pay you can use that). Watch out to create a PERSONAL account!!

Once your in at the dashboard: Click on compute engine, then you have to wait a few minutes for google cloud to activate some stuff... (see screenshot, everything will be in spoilers form now on to keep this already big post as clean as possible)

Spoiler
The free plan does not include GPU Compute. Trying to create a slot with a GPU will result in an error. Therefore: At the top of the page there should be a banner with a gift box. Click on upgrade and upgrade your account to a payed one. Dont fear you will only pay when going over the free budget. As mentioned above, simply be smart and terminate the machines in time. If you do this you wont have to pay anything .
The payed account was created to have the opportunity to apply for a (or several) GPU slots. Support has to unlock them for you. But this works every time. Requesting a single or maybe 2 GPUs is peanuts for them. To request GPU quota go to: Top left main menu: IAM & Adminstration -> quotas. Search for Max GPU allowed here:

Spoiler

You have to deselect everything and then only select GPUs (all regions) leave all the other boxes as they are. Now check the box with "Compute Engine API GPUs (all Regions)" at the top of the page click "edit quota". You now have to give them your name email and phonennumer (I used 0218347348 or something but suggest taking your real one. They are trustworthy and that way you wont have complications in getting approved). You also have to tell them what you plan to to. I wrote something like "I need it for a month for a Folding event. If performance is nice ill consider letting my work run on your servers (I actually do this right now). Then send the request.
It can take 1 or 2 business days to get approved. BUT for me on several instances it only took like a few hours. You will get Email confirmation!
When aprovede go back to. main menu-> compute engine.
Create a new VM-Instance
Spoiler
Leave every setting as is unless I tell you to change it
We need a standard machine with a general purpose dual core. Also a Nvidia Tesla P100. To add it click CPU platform and GPU. Also chose the newest CPU arch you have available. Changing CPU arch should not cost more, but watch it!
There are V100s available. They are faster but in terms of price to performance P100 wins and still gets you a nice 1,4mPPD (V100 is 1,7mPPD but 1,7$/h compared to 1,07$/h for P100)
The cheapest ones are available in the Regions beginning with US. Not in every region there is availability for what we want. simply try weather you can select it or not. Also watch pricing. Some places are more expensive!
Should look like this and beneath this the CPU type and GPU selection:

Spoiler
In terms of OS we will save some time and hassle by selecting a preconfigured Image with Nvidia CUDA (that's where F@H is running on)
Choose "Deep Learning Image: Base m36 (with CUDA 10.0)" Yes I know we dont do Deep learning but CUDA and a preconfigured graphics driver is niiiiiice!'

Spoiler
Leave the Memory option as is. We do not need a SSD and they are more expensive

Back at the „VM-Instances“ view: wait a minute for your VM to be initialized and booted
Info: You can connect to the VM via SSH, so you Putty guys: go!

But for our purpose the simplest way is to connect with the build in tools. So click on SSH besides the machine informations.
after you're connected (takes a minute) it will i meditaly ask you to install Nvidia drivers -> answer with y enter. This takes a moment.
reboot the box with
```
sudo reboot
```
SSH will disengage. Wait 2min then click reconnect. Give it time!
Now you’re in Debian (terminal Version). No need to configure anything more! To be able to copy and paste go to settings at the top of the window „Clipboard settings“ and activate „copy with ctrl…“

Spoiler
To see if driver install was ok: Do a quick
```
nvidia-smi
```
P100 showing up? -> nice!
Now we basically only need the FAHClient (FAHControl is not needed and unless since we don’t have graphical output :P)
Everything is taken from: https://foldingathome.org/support/faq/installation-guides/linux/manual-installation-advanced/
First run (input it with copy and paste)

wget https://download.foldingathome.org/releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb

Thats downloading the client, then

sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb

To install FAH.  A configuration will pop up and ask for Username and Team Nr. and Passkey  -> you know what to do 223518 FTW!

Set that baby to full thrust!
ENABLE autostart!!!! You will get some errors now, but you can ignore them
Give it 3min, for real!
Now we need to tell the client to use our GPU also. Therefore we will need to edit the configuration file with
```
sudo nano /etc/fahclient/config.xml
```
To close this editor: crtl+X
If you open that and it has a bunch of comments everywhere (you'll see what I mean the first time you do it), FAHClient hasn't finished started. Wait 30 seconds and try again. Repeat as necessary.

Once the comments are gone you have to edit it to:

config>
  <!-- Client Control -->
  <fold-anon v='true'/>
  <!-- Folding Slot Configuration -->
  <gpu v='true'/>
  <!-- Slot Control -->
  <power v='full'/>
  <!-- User Information -->
  <passkey v='yourpasskey_should_already_be_here'/>
  <team v='223518'/>
  <user v='your_user_should_already_be_here'/>
  <!-- Folding Slots -->
  <slot id='0' type='CPU'/>
  <slot id='1' type='GPU'/>
</config>

What you have to change should be:
-GPU to true
-create the GPU slot. Watch out to have CPU on slot 0 and GPU on 1!

Exit via Ctrl+X and the choose Y to save but dont change anything on how or where it is saved just pressing enter when it asks for the location of where to save to.
To apply this new config reboot once more, you know how to do that. Otherwise -> Step 17
after being logged back in give it some more time to spinn everything up. Folding should be active now.
You can check via the following commands:
```
nvidia-smi
```
should look like this:

Spoiler

there you see the Folding process pegging the GPU
also run:
```
top
```
This is sort of a task manager where you should see the folding slots doing their work.
You can see the log file via
```
tail /var/lib/fahclient/log.txt
```
this shows the last lines of the log. Usually enough to judge if everything is fine. To see everything replace top with nano.
Cool! Everything done, to get out of there at the main prompt:
```
exit
```

The tips and tricks form here on can be used on the preemtible and the normal instances

How to do a quick PPD check and other checks

Main Terminal:

FAHClient --send-command <command>

Commands:

ppd -> gives you the current overall produced PPD (you know PPD is not accurate in the first hour or so)
configured -> if output is "True" your client is configured with username team etc
slot-info -> gives you information on wich devices its running
queue-info -> verry detailed info about the wokrunits being done at the moment
help -> everything you can do and some you probably shouldn’t do.

Remote Monitoring (Client or HFM)

In Google Cloud Platform

Go to VPC Network>Firewall Rules>Create Firewall rule
Set the network to default, target to "All instances on this network"
Source Ip ranges to 0.0.0.0/0 [This matches all IPs], protocols and ports to tcp port 36331.
[Nonstandard port to try to prevent portscanning bots from easily guessing that FAH is running]
In the VM, edit the FAH config.xml to include

<allow v='127.0.0.1 My.Public.ip.Address.at.home'/>
<command-port v='36331'/>
<password v='My_Super_Secret_Password'/>

Do note that the password is sent in plaintext, so don't use anything that you particularly care about. This is again just to mainly keep out bots.

5. restart FAH with this command:

FAHClient restart

(If this dent work do "sudo /etc/init.d/FAHClient restart"

6. Add the client to FAHControl or HFM.Net as you normally would, making sure to use the external IP address of the VM (shown on the compute engine homepage) as the IP and also making sure to set the port to 36331 (or whatever it was that you set it to). Also make sure to enter your password. If everything went well you should see the slots show up in FAHControl just as if FAH were running on your local system.

Other server providers and installation with CUDA and driver not being preinstalled

My top recommendation is CherryServers. They offer a 2xGTX1080 Box for 0,38$/h including 30$ of free budget! https://www.cherryservers.com

Plaining on paying and going over the free budget? Thats your cheapest option! Every Ubuntu 18.06 LTS server with a decent Nvidia GPUs will work with this guide. AMD is possible but not really there in the GPU compute server industry. Therefore my tutorial with CUDA:

Create yourself an account in the top right click Log in -> create account -> you know how to do this . As with the google service you will have to give them your card. But its NOT charged unless you tell them to. I also got a window promoting me to set up a automatic top up, but you can close that and work with the free test budget. Later payment is even possible with crypto etc.
Once your in the portal go to servers -> bare metal (this time we want full power and nothing virtualized) and pick the E3-1240LV5 with 2xGTX 1080, show in pic:
Spoiler
Go on customize and at the bottom choose the OS Ubuntu 18.04. Then add to chart and leave everything as is!

3. Go to your chart and turn the credit top up slider all to the left. That way you wont be charged beforehand. The server will run as long as you have balance on the account.

4. WAIT... mine took 30min to be finished. You will get email notifications. ATTENTION: once you got the notice that it is running, you're paying it.

5. When the server is running: On your servers overview click on the server and note the password displayed there. Also not the public IP.

6. SSH into dat B. SERVER (WTF is SSH? -> see this post:

7. The following steps will be similar to what I have described in top.

BUT here we will have to make a few steps before downloading and installing FAH... we need the graphics drivers and cuda support to run properly
First update your system
```
sudo apt update 
sudo apt upgrade
```
We will now install MUCH more than we need, but this is the simplest solution it can take 10-15min to finish the install, but tinkering with everything else needed would be SOO much more effort.

therefore follow this step by step and IN ORDER: (For the Linux PROS: Yes I know... but due to the server only being used for F@H and the extra stuf not having any impact on F@H this is the best solution. And yes I know I am installing nearly a full gnome + window server etc) Source is Nvidias official stuff: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804
```
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-1-local-10.1.243-418.87.00/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda
```
Reboot! IMPORTANT!
Continue with Step 19. form above. Make sure both GTX10808 are showing up. Also make sure to create two GPU slots on your config. XML
Slot 0 CPU
Slot 1 GPU
Slot 2 GPU
Don't forget the reboot and you're done

K, but now my free credits are gone and I dont want to pay for more compute time

Simply delete the machine in the overview. That way there wont be any extra cost. You now can close the GCloud account altogether if you wish to.

But bro... I only have a CPU server with 3849cores, folding possible?

YES! In that case you dont have to check for GPU drivers and simply run some modern form of Debian on the server and install F@H as shown. Editing the config is not needed. It should automatically detect the cores and start on them

Other providers to try are:

Amazon
MS Azure
IBM Cloud (Hint from @Mihle: Free credits will last about 4,2 days or so with a single P100, you have to specifically allow ssh and allow outbound connections.) ATTENTION: Some users reported being blocked/banned there for folding with he free credits)
Oracle Cloud
You know any more cheap GPU server rentals?

My results so far:

At the time of writing the original version of this guide I had:
1xRadeon ViII + 1x R9 290X in private system pulling close to 2mPPD

6x Tesla P100 in GCloud pulling 1,4mPPD for each one

2xGTX1080 on Cherry pulling 1,2mPPD together

I think its worth it!

As always: Tips and suggestions? Plz. tell me in the comments!

Happy cloud Folding,

Metallus

TVwazhere · October 10, 2019

I was wondering how you were jumping up in points so fast!

Damn, guess I gotta rent a server. @leadeater got any spare CPU's? :^)

Spakes · October 10, 2019

Woah, seems like a joke about LTT team buying GPUs purely for folding now extends even further... Maybe rent a supercomputer next time? ( ͡° ͜ʖ ͡°)

BTW, when it comes to remote monitoring, did you consider VNC and some of these web pages?

http://fahmm.net/

Metallus97 · October 10, 2019

1 hour ago, Spakes said:

Woah, seems like a joke about LTT team buying GPUs purely for folding now extends even further... Maybe rent a supercomputer next time? ( ͡° ͜ʖ ͡°)

BTW, when it comes to remote monitoring, did you consider VNC and some of these web pages?

http://fahmm.net/

Yeh @Gorgons guide doesnt work for me.

his configurations on the server side ar correct, but it seems like an firewall or something issue on my side...

Still figuring it out. But ill take a look at the website! THX

Oh and I bought my VII (used) over a 5700xt for folding and still decent gaming perf

Gorgon · October 10, 2019

1 hour ago, Metallus97 said:

Yeh @Gorgons guide doesnt work for me. Still figuring it our. But ill take a look at the website! THX

Oh and I bought my VII (used) over a 5700xt for folding and still decent gaming perf

With Windows 9 times out of 10 a reboot will fix it. The other times it is usually Windows Firewall being confused.

Metallus97 · October 10, 2019

19 minutes ago, Gorgon said:

With Windows 9 times out of 10 a reboot will fix it. The other times it is usually Windows Firewall being confused.

I have tried but no success I guess its about my router or provider... have to check later today

Mihle · October 13, 2019

This guide worked! Sadly on cherry servers I used 19 of the 30 free euro on it not working.
At this point I just think you older guide just didnt work because I tried it multiple times without working. This one worked tho.

Mihle · October 13, 2019

I just tried to log on Azure to see.... didnt find GPUs at first (dont know if I am blind or if its not available for free users) so upgraded to pay as you go, and at least to my eyes it then seems like you loose the free credits you had.... If I am not blind.

Metallus97 · October 13, 2019

52 minutes ago, Mihle said:

I just tried to log on Azure to see.... didnt find GPUs at first (dont know if I am blind or if its not available for free users) so upgraded to pay as you go, and at least to my eyes it then seems like you loose the free credits you had.... If I am not blind.

They have GPU options, but you have to request them trough support. Just google azure GPU Server. But I am no expert at azure. They are just recommendations.

53 minutes ago, Mihle said:

I just tried to log on Azure to see.... didnt find GPUs at first (dont know if I am blind or if its not available for free users) so upgraded to pay as you go, and at least to my eyes it then seems like you loose the free credits you had.... If I am not blind.

happy to hear. The original version of my old old pat did not work. The version up there now should. Although is is more complicated

Mihle · October 13, 2019

IBM cloud you do more similar to the Cherry one to do it because have to install stuff.

Free credits will last about 4,2 days or so with a single P100
If anyone else try to use IBM cloud, you have to specifically allow ssh and allow outbound connections.

Metallus97 · October 13, 2019

10 minutes ago, Mihle said:

IBM cloud you do more similar to the Cherry one to do it because have to install stuff.

Free credits will last about 4,2 days or so with a single P100
If anyone else try to use IBM cloud, you have to specifically allow ssh and allow outbound connections.

will put that hint in the tutorial! THX dude

Mihle · October 13, 2019

3 minutes ago, Metallus97 said:

will put that hint in the tutorial! THX dude

Tho, you do have to remember to stop it as the google one

Metallus97 · October 13, 2019

40 minutes ago, Mihle said:

Tho, you do have to remember to stop it as the google one

For sure

Stormseeker9 · October 14, 2019

So how many can we apply for, as in your tutorial I see 1, but at the end you mentioned you have 6 running?

Did you apply for 6 on one account? @Metallus97

Metallus97 · October 14, 2019

1 hour ago, Stormseeker9 said:

So how many can we apply for, as in your tutorial I see 1, but at the end you mentioned you have 6 running?

Did you apply for 6 on one account? @Metallus97

Actually I have 3 accounts

but be careful and I am not responsible for anything!

also the Dual 1080 is from cherry servers, I pay for that now.

Stormseeker9 · October 14, 2019

53 minutes ago, Metallus97 said:

Actually I have 3 accounts

but be careful and I am not responsible for anything!

also the Dual 1080 is from cherry servers, I pay for that now.

Yeah gotta reso the cherry ones. Was gone over the weekend so cancelled it all.

i did request 6... See how that goes

did tel them it’s for folding —> medical research so let’s hope. Even if they say we’ll give you 1 I’ll be happy

Mihle · October 14, 2019

I just said folding and might consider continue with it.

I have two P100 at Google and one P100 at IBM currently, the cherry one did run out this night. Only one account each, because I would need more than one card for more and not comfortable with using other people's

Usually do about 200-350k on my desktop but now:

Have gone up 100 LTT team all time rank soon.

IBM will run out on Thursday and Google will run out on Friday/Saturday if I have calculated correctly.

Will probably try to set up Oracle cloud one day too.

Metallus97 · October 14, 2019

41 minutes ago, Mihle said:

I just said folding and might consider continue with it.

I have two P100 at Google and one P100 at IBM currently, the cherry one did run out this night. Only one account each, because I would need more than one card for more and not comfortable with using other people's

Usually do about 200-350k on my desktop but now:

Have gone up 100 LTT team all time rank soon.

IBM will run out on Thursday and Google will run out on Friday/Saturday if I have calculated correctly.

Will probably try to set up Oracle cloud one day too.

NICE! Thats what I love to see!

But: I have no Idea weather or not GPUs are included in Oracle free and how much budget you get there

Metallus97 · October 14, 2019

2 hours ago, Stormseeker9 said:

Yeah gotta reso the cherry ones. Was gone over the weekend so cancelled it all.

i did request 6... See how that goes

did tel them it’s for folding —> medical research so let’s hope. Even if they say we’ll give you 1 I’ll be happy

lol you requested 6 P100? DAYM BRO! But I think this will burn through the $$ pretty quick! Be sure to make a machine with enough vCOres to support those GPUs. Idd get an 8 core for this. Or distribute them on different VMs. Just calculate whats cheaper

Stormseeker9 · October 14, 2019

19 minutes ago, Metallus97 said:

lol you requested 6 P100? DAYM BRO! But I think this will burn through the $$ pretty quick! Be sure to make a machine with enough vCOres to support those GPUs. Idd get an 8 core for this. Or distribute them on different VMs. Just calculate whats cheaper

Well I just requested 6 GPU's.. didn't specify model lol...

Oh well I'll see what they respond. Keep you updated

Mihle · October 14, 2019

24 minutes ago, Metallus97 said:

NICE! Thats what I love to see!

But: I have no Idea weather or not GPUs are included in Oracle free and how much budget you get there

I will check it out some day at least.

Stormseeker9 · October 14, 2019

They have been granted !

I’ll start looking during this week what I can do here

Mihle · October 15, 2019

IBM suspended my account...... Lol.

Spoiler

Automatically created a support ticket, so I will probably get an email more at some point probably. If I click th "here" or try to go to the website I get:

Spoiler

So looks like you can't use the free credits freely as it seems you can with Google or cherry..... Or something.

No more points from their servers on me I guess. Got to use it for one day tho.

Stormseeker9 · October 15, 2019

nevermind.

Got one running with 1 GPU.

Will setup the second one or just let this one VM run on its own.

@Metallus97 we can close the SSH window right? it will continue to fold ?

EDIT;

I know its folding but seems like a low wattage use ?

Metallus97 · October 15, 2019

11 hours ago, Stormseeker9 said:

nevermind.

Got one running with 1 GPU.

Will setup the second one or just let this one VM run on its own.

@Metallus97 we can close the SSH window right? it will continue to fold ?

EDIT;

I know its folding but seems like a low wattage use ?

Yes you can close it

70W seems low, but this depends on WU and can jump a Little mine also only use like 60% TDP

Simply keep and eye on PPD and that will tell you how it’s doing.

verry happy to hear it works!

Sign In

TUTORIAL FREE PPD Cloud GPU Tesla P100/V100 F@H! FINAL edition V2

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites