Tutorial: Free PPD! F@H on Google clouds free account PRESUMED 80kPPD

[Metallus97]

!!!GO HERE FOR THE UPDATED VERSION!!!! 

 

 

 

 

Those who fist want some numbers in order to see of its worth it-> it IS check my latest Post here!

Basics and Introduction

 

With the Folding month going on and competition being massive I decided to think about ways of maxing my PPD besides upgrading and OCing my own Hardware (wich I also do)

 

I am using Google Cloud sometimes for my work as it includes MUCH Data science etc. Setting up an account there and getting a Debian (Linux) VM going with F@H at full blast is pretty easy and can be done in like 30min. I know that with sharing this Ill loose the advantage, BUT competition is nice... BUT the cause is more important! I think and hope that's why we are participating in the F@H month anyway.

Basic Infos: Google clouds virtual machines are payed by usage time. Sining up for the free trial will give you 365 days of time and a free budget of 300$ to spend on computing time. After this your Card WONT be charged. You will get notified and if you do not do anything the machines will simply shut down. Therefore this is pretty save concerning the $$$ :).

In this guide we will create some machines wich max out the power but burn through the 300€  relative slowly.

Therefore this is a nice easy and free way of boosting your PPD ! 
 

RIIIGHT so letzzzze go MARIO

 

(The screenshots will be German, because I didn’t want to reset my whole google accounts language, but ill mark + tanslate everything so its easy to follow)

 

1. You need a Google Account. You can use your existing one or create a new one (multiple are also possible, but I am NOT responsible for anything :P)

https://cloud.google.com/ sing in, give them your Card (there they write about not charging you when you’re done with the free 300$ budget. If you have google pay you can use that).  Watch out to create a PERSONAL account!!

2. Once your in at the dashboard: Click on compute engine, then you have to wait a few minutes for google cloud to activate some stuff... (see screenshot, everything will be in spoilers
   Spoiler

   

    

3. Create a new VM Instance
   Spoiler

   

4. Google Cloud free is limited to 12 CPU cores per user. Therefore we have to create two machines. One with 8 and one with 4 cores. Creating one with 16 will result in an „over-quota“ error. Now we will create the 8 core machine, but the process for the 4core guy is the same, I´ll come back to that
5. Leave every setting as is unless I tell you to change it
6. We need a computing optimized machine. The cheapest ones are available in Region us-central and then on Zone us-central-c  (-b should work to, but I haven’t tried, price is the same)
7. Choose computing optimized
8. Machine type: c2-standard-8 (I know the others are BA, but the free plan wont allow them ;(
9. LEAVE everything else as is! Click Create on the bottom of the page.
   Spoiler

   

    

10. Back at the „VM-Instances“ view: wait a minute for your VM to be initialized and booted
    
    Info: You can connect to the VM via SSH, so you Putty guys: go!

    But for our purpose the simplest way is to connect with the build in tools.

11. Click on SSH and wait (copy and paste sometimes only works on Chrome… who would have thought xD. Also install this plugin "SSL for google Cloud" to be able to copy and paste in the browser SSL client https://chrome.google.com/webstore/detail/ssh-for-google-cloud-plat/ojilllmhjhibplnppnamldakhpmdnibd ). Plz do this or you will have to type some long links and your passkey for F@H :P)
    EDIT: Now it works even with Safari on my Mac... guess you should try and then see if you need Chrome/the Plugin 
12.  
    Spoiler

    

     

13. Now you’re in Debian (terminal Version). No need to configure anything, we directly download FAH! To be able to copy and paste go to settings at the „Clipboard settings“ and activate „copy with ctrl…“
    Spoiler

    

14. Now we basically only need the FAHClient (FAHControl is not needed and unless since we don’t have graphical output :P)
    Everything is taken from: https://foldingathome.org/support/faq/installation-guides/linux/manual-installation-advanced/
     

15. First run (input it with copy and paste) 

    wget https://download.foldingathome.org/releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb 

    Thats downloading the client, then 

    sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb

    To install FAH

16. A configuration will pop up and ask for Username and Team Nr. -> you know what to do 223518 FTW!

17. ATTENTION HERE: Coppy+Pasting my passkey did NOT work for me! It showed only blank space but with he cursor at the end of the line. This does NOT work! Sadly you have to type it… and plz. Tripple check it!!!!
    EDDIT: On some browsers and machines it works, on some not... well just check the the key is properly inserted before clicking ok.
18. Set that baby to full thrust!
19. DISABLE autostart!!!! You will get some errors now, but you can ignore them

Being back at the command prompt:

• Optional: The F@H installer complains about the dependency "bzip2". For me everything runs somooth without it, but if you want to install it to be super save: Very simple; its only entering

  sudo apt install bzip2

   

20. Normally the Folding process will stop when you disengage the SSH Connection 

To prevent this we will run screen. Thats sort of a terminal inside the terminal. It will persist even when logged out of SSH.

screen

21. To run F@H Input:

FAHClient

You will get even more error annnnd  were done installing!

 

22. BUT DONT CLOSE THE WINDOW NOW!

Disengage form the screen with pressing ctrl+A+D. This will bring you back to the normal Terminal. 

to see your screen sessions type

screen -ls

That should show one session with the number 1. Thats where our folding is happening

to reconnect

screen -r numberOfTheSession

to exit form the MAIN terminal AFTER exiting screen

exit

 

To create a machine with the 4 cores you have left over: Click on the + thingy on the top of the overview and repeat form point 5, but now with a "c2-standard-4" configuration. The rest of the installation is exactly the same!

 

Anny suggestions, feedback or things I can improve in this tutorial? Please let me know!

A HUGE THANKS TO: @Gorgon and  @proptiedwho helped me figure some stuf out and contributed some tricks and commands to this guide!

-To do: Remote Monitoring; exact configuration for Linux Servers... I have to get it working for myself first...-

 

I want more PPD! (be careful to watch your free balance!) 

Spoiler

It is possible to upgrade your account to a payed one. That way you're allowed to use 24cores instead of 12. Also you then can apply for as much cores and also GPUs (they have Tesla V100s :O) as you want. I guess they wont approve much, but they guaranteed me a T4  

 

1. At the top of your VMs page is a banner wich tells you that you are on the free plan and using your free credits. There is a button to upgrade to a full payed plan. This is also free and done with 2 clicks. BUT when you have used up your gifted $$ you will be charged for every hour you use their servers.So be careful! IF you are and maybe calculate a bit nothing can happen. When a server its terminated ti will directly be deleted and not charge you anymore.
2. Having done this you can directly start to delete your old 4core guy and create a new 16core boy. Same procedure but this time you chose 16core high perf in us central c or d (those are the cheapest ones). Rest of config is exactly the same. Due to already having a 8 core you now have maxed out the available 16+8=24 cores.

 

I WNAT EVEN MORE!

 

For even more you need to apply for more quota!

1. In the Cloud platform go to IAM & Adminstration -> quotas. Search for Max GPU allowed here: and tell them why you need it and what you're planing to do. Suggestion: ask for ONE  
2. Same goes for raising your CPU limit. But I suggest choosing GPU bcs. FAAAASTERRR!
3. It can take 1 or 2 business days to get approved
4. If you are create a new VM, chose standard (we dont need a super fast CPU now) but then add a T4 in the mix. I haven't jet tried a V100 (they are expensive and I am not sure about PPD) 
5.  Chose UBUNTU 18.04 (on Debian the Nvidia drivers wont install the way we need them to)
6. 

 

Useful Tips and Tricks

 

Remote Monitoring by @Gorgon

 

 

 

How to monitor if everything is running smoothly.
 

1. Wait 5min, go to the VM-Instances box, click Infos and Monitoring, check if the CPU is pegged. If it is: Congrats! Its running! Should look like this:
2. I have noticed this to update VERRRRy slowly and be kind of inaccurate. The direct way of doing things is to SSH back in the server and running 

   top
   

   thats a "taskmanager" to get out: Ctrl+C

3. Spoiler

    

   

 

 

How to check the log files once the SSH connection was closed.

1. Log in to google cloud on the left side of your dashboard go to compute engine, now you're back at your overview 
2. click SSH
3. Once connected to the server  go on your screen where the folding is running to see the live action. be careful to exit with Ctrl+A+D NOT WITH ctrl+c!!
4. With this configuration there wont be a log.txt. Dont fear its ok  as long as you see it on "screen" command and under "top" command you see the usage everything is fine.

 

How to change the configuration file (in case of error in passkey etc...)

on the main prompt:

Edit your config file with

sudo nano /etc/fahclient/config.xml

then there should be a line with 

<passkey v='xxxxxxxxx'/>

(WITH both  ' )

Correct it, save the file and restart the FAHClient  

to restart it go on the screen where it runs and kill it ctrl+c. then restart it on this same screen 

 

How to check CPU usage right at the server, in the main prompt

top

to geht out of there: crtl+c

 

This is a very early and ruff version of this post and I am working on it to improve it!

Anny questions, suggestions etc. Please let me know

Happy folding,

Metallus

 

Note: Stepps 15 to the end and the tips sections should basically work on every Debian/u until etc. server  

 

 

How to do a quick PPD check

Main Terminal:

FAHClient --send-command <command>

Commands: 

• ppd -> gives you the current overall produced PPD
• configured -> if output is "True" your client is configured with username team etc
• slot-info -> gives you information on wich devices its running
• queue-info -> verry detailed info about the wokrunits being done at the moment 
• help -> everything you can do and some you probably shouldn’t do.

 

 

Here is proof of the first successful work unit calculated on my GCloud:

 

Spoiler

 

[vanished]

Fascinating, I'm shocked by much they offer for free!  I guess they're hoping and expecting most people will get hooked and come back for more

Looking forward to seeing the conclusion and how much power you can get out of this

and PS "saving your progress" is smart, this editor is not good for long works in my experience...

[Metallus97]

10 minutes ago, Ryan_Vickers said:

Fascinating, I'm shocked by much they offer for free!  I guess they're hoping and expecting most people will get hooked and come back for more

Looking forward to seeing the conclusion and how much power you can get out of this

and PS "saving your progress" is smart, this editor is not good for long works in my experience...

yehhh.... I used word to write it, but anyway I wanted this to be saved. 

For statistics we will have to wait a few days.

But sure its pretty awesome. the first WU finished with estimated 6345 points and did that in 2:30, Now some VERRY VEERERRY ruff math puts that in the ballpark of 60k PPD... but we have to see. oh and this was only the 8core guy. Then there are the other 4 cores you get. around 90kPPD would be pretty boss for free.. but we will have to wait and see

[vanished]

2 minutes ago, Metallus97 said:

yehhh.... I used word to write it, but anyway I wanted this to be saved. 

For statistics we will have to wait a few days.

But sure its pretty awesome. the first WU finished with estimated 6345 points and did that in 2:30 Sommer VEERERRY ruff math puts that in the ballpark of 60k PPD... but we have to see. oh and this was only the 8core guy. Then there are the other 4 cores you get. around 90kPPD would be pretty boss for free.. but we will have to wait and see

I'm getting the sense that this is CPU only though?  That's slightly disappointing but still, you can't complain for free

[Metallus97]

21 minutes ago, Ryan_Vickers said:

I'm getting the sense that this is CPU only though?  That's slightly disappointing but still, you can't complain for free

sure but look at the numbers for free that's not bad. And you can make as manny accounts as you wish  just like LTT did with the Drive storage.

They also have Tesla K80,P4, T4, V100... but that will burn through the 300$ pretty quick. IF there is interest I can try to get that working also and post an other thread (much of the configuration is the same)

Oh and I am not sure how well F@H runs on those cards anyway...

[TheBean]

im surprised this thread isnt blowing up. also, is there a way to view the status of these machines through the client on my main machine?

 

EDit: I messed up the passkey on the 8 core machine. how can i fix that?

Edited October 5, 2019 by Saksham

[Metallus97]

5 hours ago, Saksham said:

im surprised this thread isnt blowing up. also, is there a way to view the status of these machines through the client on my main machine?

 

EDit: I messed up the passkey on the 8 core machine. how can i fix that?

Yes, you should be able to connect to them via FHAControl on your PC. DOnt have the time today, but ill add this to the tutorial tomorrow!

 

Messed up passkey:

Edit your config file with

sudo nano /etc/fahclient/config.xml

then there should be a line with 

<passkey v='xxxxxxxxx'/>

(WITH both  ' )

Correct it, save the file and restart the FAHClient  

[TheBean]

11 hours ago, Metallus97 said:

Yes, you should be able to connect to them via FHAControl on your PC. DOnt have the time today, but ill add this to the tutorial tomorrow!

could you help me now please? its been a while. i can see from the fah stats, that they are registering with their servers and have completed some WUs, but i want to be able to have more control over them through the fahcontrol app

[TheBean]

@Metallus97 still waiting...... 

[Metallus97]

14 hours ago, Saksham said:

@Metallus97 still waiting...... 

Chill, I have to work and sleep to!

 

Right, I checked it... its hard to do because this requires a TONNN of configuration and installation of packages on your server. The only way to get your PPD into an other FAHControl is running FAHControl on the server. This is super hard bcs this server was never intended to ruin a UI application. Then there is also a heap of configuring to do with networking to get the connection.

I recommend you to simply install a SSH client (or on Linux and Mac OS it is already present in the terminal) and configuring ist to quick connect to your servers. then check the logs or do a simple 

top

to see how the CPUs are doing.

 

However G Cloud includes tons of APIs wich allow you to monitor every bit of your server, but this is also WAAAY to micvc for our purpose here. And a quick check showed me, that access to this APIs is limited by the free trail plan  

[proptied]

Thanks for this! Got it working on GCP and controlling with HFM.NET.

 

[TheBean]

16 minutes ago, proptied said:

Thanks for this! Got it working on GCP and controlling with HFM.NET.

 

how did you manage to do this? can you explain the steps?

[proptied]

1 minute ago, Saksham said:

how did you manage to do this? can you explain the steps?

I'm not a FAH expert, but to summarize:

In the VM: Add HTTP Server to (FAHClient) config.xml and set to allow all IPs (mostly for convenience). Also added password.

In GCP: Added firewall rule to allow just my IP to access port 36330.

In HFM.NET/FAHControl: Connect to the "External IP" that GCP has listed for the VM.

In life: Win.

 

If you'd like something more detailed, I can do that as well. This may get you started though.

[TheBean]

2 minutes ago, proptied said:

In the VM: Add HTTP Server to (FAHClient) config.xml and set to allow all IPs (mostly for convenience). Also added password.

In GCP: Added firewall rule to allow just my IP to access port 36330.

im in a dorm room at college. idk if my ip is static or dynamic. will this affect this steps?

idk what you mean by add the http server to the config.xml

[TheBean]

21 minutes ago, proptied said:

I'm not a FAH expert, but to summarize:

In the VM: Add HTTP Server to (FAHClient) config.xml and set to allow all IPs (mostly for convenience). Also added password.

In GCP: Added firewall rule to allow just my IP to access port 36330.

In HFM.NET/FAHControl: Connect to the "External IP" that GCP has listed for the VM.

In life: Win.

 

If you'd like something more detailed, I can do that as well. This may get you started though.

 

I have made these rules in the gcp firewall. is this right? I cannot connect to it still. what else do I need to do?

 

 

 

 

[proptied]

20 minutes ago, Saksham said:

im in a dorm room at college. idk if my ip is static or dynamic. will this affect this steps?

idk what you mean by add the http server to the config.xml

If you notice you can no longer connect, you will just have to go back into GCP and update your IP: so no big deal.

 

Here's an example of my config.xml for the FAHClient:

<config>
  <!-- Folding Slot Configuration -->
  <gpu v='false'/>

  <!-- Slot Control -->
  <power v='full'/>

  <!-- User Information -->
  <passkey v='mypasskeyhere'/>
  <team v='223518'/>
  <user v='myuserhere'/>

  <!-- HTTP Server -->
  <allow v='0/0'/>
  <password v='myhttppasshere'/>

  <!-- Folding Slots -->
  <slot id='0' type='CPU'/>
</config>

 

[TheBean]

7 minutes ago, proptied said:

<!-- HTTP Server --> <allow v='0/0'/> <password v='myhttppasshere'/>

what does this actually do? im a newbie to servers and cloud computing etc. 

[proptied]

Just now, Saksham said:

what does this actually do? im a newbie to servers and cloud computing etc. 

That's giving access to other "controllers", such as FAHControl or HFM.NET. The '0/0' is just saying "allow all IPs to connect to me". Not secure, but the firewall rules you added in GCP protect it. The password is entered in your "controller" (see highlighted yellow):

 

HFM.NET:

 

FAHControl:

[TheBean]

25 minutes ago, proptied said:

That's giving access to other "controllers", such as FAHControl or HFM.NET. The '0/0' is just saying "allow all IPs to connect to me". Not secure, but the firewall rules you added in GCP protect it. The password is entered in your "controller" (see highlighted yellow):

thanks. it works now. @Metallus97 you should add this to the OP

[Metallus97]

1 hour ago, proptied said:

That's giving access to other "controllers", such as FAHControl or HFM.NET. The '0/0' is just saying "allow all IPs to connect to me". Not secure, but the firewall rules you added in GCP protect it. The password is entered in your "controller" (see highlighted yellow):

 

-snip-

Daymn! Nice am I allowed to add this to the OP, WITH a mention and thanks to you?

[proptied]

8 minutes ago, Metallus97 said:

Daymn! Nice am I allowed to add this to the OP, WITH a mention and thanks to you?

Absolutely not a problem.

 

Works on Azure too ($200 free credits).. though they only allow 4 CPUs total.

[Metallus97]

1 hour ago, proptied said:

Absolutely not a problem.

 

Works on Azure too ($200 free credits).. though they only allow 4 CPUs total.

Thank you sir! Ill update the post and set up a Azure ASAP

Cherry servers offers a 80 Core. But with the free credits you can only run it for a few hours. Although its really funny to see 

4X Xeon E5-4650V2 folding at full blast

They also have a 4core BUT 2xGTX1080 server.. but its always sold out

 

[Metallus97]

@proptied I cant get it to work.. can you explain how you got the firewall stuf to work properly?

I cant even get my other LAN Clients to show up in HFM/The client ok that works now, but internet connection still does not

[Stormseeker9]

@Metallus97 so I get this error after that 0.0.0.0 error.

 

Copy for some reason wont work, so had to make screenshot sorry.

 

 

[Metallus97]

1 minute ago, Stormseeker9 said:

@Metallus97 so I get this error after that 0.0.0.0 error.

 

Copy for some reason wont work, so had to make screenshot sorry.

 

did you try to update the system before installing F@H?