LTT Folding Team's Emergency Response to Covid-19

[enterprise-psi]

I thought i had it fixed

 

Spoiler

10:54:16:WU01:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
10:54:16:WU01:FS01:0x22:Following exception occured: Particle coordinate is nan
10:54:43:WU01:FS01:0x22:Bad State detected... attempting to resume from last good checkpoint. Is your system overclocked?
10:54:43:WU01:FS01:0x22:Following exception occured: Particle coordinate is nan
10:54:43:WU01:FS01:0x22:ERROR:114: Max Retries Reached
10:54:43:WU01:FS01:0x22:Saving result file ..\logfile_01.txt
10:54:43:WU01:FS01:0x22:Saving result file badstate-0.xml
10:54:43:WU01:FS01:0x22:Saving result file badstate-1.xml
10:54:43:WU01:FS01:0x22:Saving result file badstate-2.xml
10:54:43:WU01:FS01:0x22:Saving result file checkpointState.xml
10:54:43:WU01:FS01:0x22:Saving result file checkpt.crc
10:54:43:WU01:FS01:0x22:Saving result file positions.xtc
10:54:43:WU01:FS01:0x22:Saving result file science.log
10:54:43:WU01:FS01:0x22:Folding@home Core Shutdown: BAD_WORK_UNIT
10:54:43:WARNING:WU01:FS01:FahCore returned: BAD_WORK_UNIT (114 = 0x72)
10:54:43:WU01:FS01:Sending unit results: id:01 state:SEND error:FAULTY project:14541 run:0 clone:1498 gen:25 core:0x22 unit:0x0000001d0d5262775e7d2522e2e0f3db

 

SO far gtx 970 = zero issues, gtx 1070ti = zero issues , teslas = zero issues, rx580=fuuuuuuuuuuuuuuuuu

[Alex Atkin UK]

1 minute ago, WaldorfMan said:

I'm taking my 2009-ish Dell Latitude with a Celeron out of the loop today.  Usually takes 14-20 hours for one WU.  Simply not worth the effort.  However, haven't had any hiccups with it at all this whole time. 

I'm currently running this off a Fedora Live USB which didn't look like it was going to work at first, got a filesystem error.  So completing its first WU I was quite pleased.

Now if I can figure out how to use my free Google credit.   What configuration are people using for that?

[bafo_ah]

26 minutes ago, Alex Atkin UK said:

I'm currently running this off a Fedora Live USB which didn't look like it was going to work at first, got a filesystem error.  So completing its first WU I was quite pleased.

Now if I can figure out how to use my free Google credit.   What configuration are people using for that?

Tesla T4 on us-central1 / west1 region (you need to request quota)

Use the cheapest 4 vCPU + 6-8GB RAM (not sure, I use 9GB)

add 4 T4 (if you lucky enough)

 

[ShortRouter]

1 hour ago, ItchyItch said:

I see folding like body building... You can only go so far all natural before having to hit the roids...  i.e. before using the cloud VMs with Tesla GPUs 

There are also GTX / RTX servers available in the cloud.

[Alex Atkin UK]

10 minutes ago, bafo_ah said:

Tesla T4 on us-central1 / west1 region (you need to request quota)

Use the cheapest 4 vCPU + 6-8GB RAM (not sure, I use 9GB)

add 4 T4 (if you lucky enough)

 

Is the OS preconfigured to use the GPUs?

[WaldorfMan]

Microsoft has released a PowerShell script that allows you to quickly start donating your CPU cycles to the Folding@Home project by deploying the program in the Windows 10 Sandbox.

 

https://www.bleepingcomputer.com/news/microsoft/microsoft-script-installs-folding-home-in-windows-10s-sandbox/

[Alex Atkin UK]

Seems a moot point, I'm getting 0 GPU availability at Google Cloud.

 

Its annoying you can't just say "I want this hardware config, I don't care where".

[bafo_ah]

Just now, Alex Atkin UK said:

d GPU is greyed out,

 

Just now, Alex Atkin UK said:

Add GPU is greyed out, have they run out of capacity?  I did notice that trying to allocate a high CPU count it says there are no CPUs available.  LOL

You have "upgraded" your account? So they can bill you when you run out free credit, be careful tho, be sure to remove it in time!

 

T4 only available on several region... just try different region, I use us-central1 and us-west1

I using centos8 - 40GB (a little bit overkill I think, but It need more than 10GB - CUDA driver is 5GB)

 

Here the script on centOS - hope you're lucky enough, I only encounter problem once in my 10 account so far (I'm running out credit card ) - paste line by line, if you encounter error, just search for help with google... Google is your friend (remember - you just sold your soul for them)

Spoiler

sudo dnf install -y epel-release
sudo dnf install -y nano wget telnet fail2ban fail2ban-systemd
# install fail2ban for security? ddos prevention? just to lazy to change ssh port?
sudo systemctl enable fail2ban
sudo nano /etc/fail2ban/jail.local
# paste this line :
[DEFAULT]
# Ban hosts for one day!:
bantime = 86400

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true

# ctrl+x for exit and then y then enter

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd # to check status, you should see ssh jail here

sudo dnf install -y kernel
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
sudo dnf install -y http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-repo-rhel8-10.1.243-1.x86_64.rpm
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf clean all
sudo dnf install -y cuda
nvidia-smi # you should see your GPU here, if not, you screwed

# fah im coming

wget https://download.foldingathome.org/releases/public/release/fahclient/centos-5.3-64bit/v7.4/fahclient-7.4.4-1.x86_64.rpm
sudo rpm -i --nodeps fahclient-7.4.4-1.x86_64.rpm

telnet localhost 36330
slot-info
options user=usernamehere passkey=yourpasskeyhere team=223518 client-type=beta max-packet-size=big power=full gpu=true
exit

sudo /etc/init.d/FAHClient stop
# wait a moment, cross your finger... then proceed
sudo /etc/init.d/FAHClient start

telnet localhost 36330
slot-info
# you should see your GPU here, and we can delete CPU slot
slot-delete 0
always_on
unpause
exit

tail -f /var/lib/fahclient/log.txt

 

[deepbogger]

 

On 3/13/2020 at 10:31 PM, TVwazhere said:

Damn, guess I wont be playing as much Doom: Eternal Borderlands 3 on Steam as I thought GOTTA FOLD BAYBEEEE

 

[Alex Atkin UK]

3 minutes ago, bafo_ah said:

 

You have "upgraded" your account? So they can bill you when you run out free credit, be careful tho, be sure to remove it in time!

 

T4 only available on several region... just try different region, I use us-central1 and us-west1

I using centos8 - 40GB (a little bit overkill I think, but It need more than 10GB - CUDA driver is 5GB)

 

Here the script on centOS - hope you're lucky enough, I only encounter problem once in my 10 account so far (I'm running out credit card ) - paste line by line, if you encounter error, just search for help with google... Google is your friend (remember - you just sold your soul for them)

  Reveal hidden contents

sudo dnf install -y epel-release
sudo dnf install -y nano wget telnet fail2ban fail2ban-systemd
# install fail2ban for security? ddos prevention? just to lazy to change ssh port?
sudo systemctl enable fail2ban
sudo nano /etc/fail2ban/jail.local
# paste this line :
[DEFAULT]
# Ban hosts for one day!:
bantime = 86400

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true

# ctrl+x for exit and then y then enter

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd # to check status, you should see ssh jail here

sudo dnf install -y kernel
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
sudo dnf install -y http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-repo-rhel8-10.1.243-1.x86_64.rpm
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf clean all
sudo dnf install -y cuda
nvidia-smi # you should see your GPU here, if not, you screwed

# fah im coming

wget https://download.foldingathome.org/releases/public/release/fahclient/centos-5.3-64bit/v7.4/fahclient-7.4.4-1.x86_64.rpm
sudo rpm -i --nodeps fahclient-7.4.4-1.x86_64.rpm

telnet localhost 36330
slot-info
options user=usernamehere passkey=yourpasskeyhere team=223518 client-type=beta max-packet-size=big power=full gpu=true
exit

sudo /etc/init.d/FAHClient stop
# wait a moment, cross your finger... then proceed
sudo /etc/init.d/FAHClient start

telnet localhost 36330
slot-info
# you should see your GPU here, and we can delete CPU slot
slot-delete 0
always_on
unpause
exit

tail -f /var/lib/fahclient/log.txt

 

Could that be why if I choose a none-compute platform and it DOES let me pick a GPU, its saying 0 are available?  Or are just too many people hammering their service right now?

 

Its all seeming like too much of a hassle right now, particularly as I'd be screwed if they billed me and I've had no sleep.

[cbigfoot]

so apparently one of my puters will only send to 

140.163.4.231 and it gets to either .99% or 2% or .5% and looses connection why cant i send to a real server like linus1 lol

[bafo_ah]

8 minutes ago, Alex Atkin UK said:

Could that be why if I choose a none-compute platform and it DOES let me pick a GPU, its saying 0 are available?  Or are just too many people hammering their service right now?

Have you "request quota change"

click on hamburger menu (three strip menu on top left) -> IAM & Admin -> Quotas

*you need to enter to Compute Engine -> VM Instance first so they can configured your environment or something for first time, it take a time before you can create your first VM

set filter (click None first, so It clear the selection - so many quotas here)

set filter

Service -> compute engine API

Metric -> GPUs (all region) + NVIDIA T4 GPUs

Location -> Global + yourdesiredlocation

 

then you need to edit your quotas, send request to google team, and they will decided if you can do this (I request 4GPU and they approved it)

 

Then you can try to create instance again (Compute Engine -> VM Instance)

[Alex Atkin UK]

2 minutes ago, bafo_ah said:

Have you "request quota change"

click on hamburger menu (three strip menu on top left) -> IAM & Admin -> Quotas

set filter (click None first, so It clear the selection - so many quotas here)

set filter

Service -> compute engine API

Metric -> GPUs (all region) + NVIDIA T4 GPUs

Location -> Global + yourdesiredlocation

 

then you need to edit your quotas, send request to google team, and they will decided if you can do this (I request 4GPU and they approved it)

 

Then you can try to create instance again (Compute Engine -> VM Instance)

Yeah I'm REALLY not going to do that, I'm far too likely to forget and get charged.

[Spotty]

7 minutes ago, cbigfoot said:

so apparently one of my puters will only send to 

140.163.4.231 and it gets to either .99% or 2% or .5% and looses connection why cant i send to a real server like linus1 lol

The server was unable to receive uploads yesterday due to running out of storage space. It seems that the storage issue has been fixed but there's still issues with the server, perhaps caused by everyone trying to upload all the completed work units from the past day or two all at once...
The Server Stats page lists an error with collection server on that server. https://apps.foldingathome.org/serverstats

All you can do is wait until they fix it. Keep an eye on the server status page and if it's showing online without errors you might need to restart your client to get it to attempt uploading again.

[bafo_ah]

1 minute ago, Alex Atkin UK said:

Yeah I'm REALLY not going to do that, I'm far too likely to forget and get charged.

Yeah, safety first... You can use CPU fold then (I forget the max CPU amount they allowed on free tier, its like 24CPU or something - but because its vCPU FAHClient only "see" half of them)

 

Enough to get 100k PPD (if lucky enough to get steady wu)

[Kilrah]

30 minutes ago, WaldorfMan said:

the Windows 10 Sandbox

Does this work now? Last I tried they still had a bug with non-English languages.. need to have a look again

[Alex Atkin UK]

6 minutes ago, bafo_ah said:

Yeah, safety first... You can use CPU fold then (I forget the max CPU amount they allowed on free tier, its like 24CPU or something - but because its vCPU FAHClient only "see" half of them)

 

Enough to get 100k PPD (if lucky enough to get steady wu)

Yeah I was struggling to get CPUs too, it said 0 availability even when I selected 16.  They make it WAY too hard to just setup a VM when it honestly doesn't matter WHERE its located.

[WaldorfMan]

500!

(edited - dammit )

 

[Atreiumontoya]

Team	LinusTechTips_Team
Score	24,210,584.......so close!!!
WUs	572

[ShortRouter]

1 hour ago, bafo_ah said:

Tesla T4 on us-central1 / west1 region (you need to request quota)

Use the cheapest 4 vCPU + 6-8GB RAM (not sure, I use 9GB)

add 4 T4 (if you lucky enough)

 

What's your PPD on this config? Btw I tried everything, 4x T4 is impossible to get atm so I got 2x P100.

[LordKitsune]

NOOOOOO! I have to power down to save this fan from blowing up. Damn thing woke me up with the scream of a banshee

[ShortRouter]

5 minutes ago, LordKitsune said:

NOOOOOO! I have to power down to save this fan from blowing up. Damn thing woke me up with the scream of a banshee

If it's a case fan just unplug it while PC is running. If you want shutdown, make sure you pause folding so the WU doesn't get corrupted.

[Gustavehomme01]

1 hour ago, bafo_ah said:

Tesla T4 on us-central1 / west1 region (you need to request quota)

Use the cheapest 4 vCPU + 6-8GB RAM (not sure, I use 9GB)

add 4 T4 (if you lucky enough)

 

Yep i just traded in my 2 P100 for this same setup. You get a little more PPD at half the price. Just wish i would have noticed before spent 36 hours on the P100 theoretical lost of  5-6 M points in that same 36 hours... 

 

Old setup 2 P100 = 1.2-1.7M PPD each total 2.4-3.4M PPD

New setup 8 T4= 800-900k PPD each total 6.4-7.2M PPD

Almost same cost.

[HaRdLy007]

3 minutes ago, Gustavehomme01 said:

Yep i just traded in my 2 P100 for this same setup. You get a little more PPD at half the price. Just wish i would have noticed before spent 36 hours on the P100 theoretical lost of  5-6 M points in that same 36 hours... 

 

Old setup 2 P100 = 1.2-1.7M PPD each total 2.4-3.4M PPD

New setup 8 T4= 800-900k PPD each total 6.4-7.2M PPD

Almost same cost.

Sorry to interject, but how do you request a quota increase? I have a 1 GPU limit right now and would like to increase it to at least 4.

[Zberg]

10 hours ago, Macaw2000 said:

 

 

 

A little video while we wait for the blog update. I added more rows so tagging @catawalks

@Zberg @BlackAda @danielocdh @wi3si @LAR_Systems @n0xlf @bafo_ah @Unilevers @Ayren @walderston @Den-Fi and about 15 others...

 

Desperately clawing to stay on the bottom of the graph and not fall off the page!