LTT Folding Team's Emergency Response to Covid-19

[bafo_ah]

This cloudy thing... remind me of this...

Spoiler

 

And now they fold

Spoiler

 

[Alex Atkin UK]

13 hours ago, bafo_ah said:

 

You have "upgraded" your account? So they can bill you when you run out free credit, be careful tho, be sure to remove it in time!

 

T4 only available on several region... just try different region, I use us-central1 and us-west1

I using centos8 - 40GB (a little bit overkill I think, but It need more than 10GB - CUDA driver is 5GB)

 

Here the script on centOS - hope you're lucky enough, I only encounter problem once in my 10 account so far (I'm running out credit card ) - paste line by line, if you encounter error, just search for help with google... Google is your friend (remember - you just sold your soul for them)

  Reveal hidden contents

sudo dnf install -y epel-release
sudo dnf install -y nano wget telnet fail2ban fail2ban-systemd
# install fail2ban for security? ddos prevention? just to lazy to change ssh port?
sudo systemctl enable fail2ban
sudo nano /etc/fail2ban/jail.local
# paste this line :
[DEFAULT]
# Ban hosts for one day!:
bantime = 86400

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true

# ctrl+x for exit and then y then enter

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd # to check status, you should see ssh jail here

sudo dnf install -y kernel
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
sudo dnf install -y http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-repo-rhel8-10.1.243-1.x86_64.rpm
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf clean all
sudo dnf install -y cuda
nvidia-smi # you should see your GPU here, if not, you screwed

# fah im coming

wget https://download.foldingathome.org/releases/public/release/fahclient/centos-5.3-64bit/v7.4/fahclient-7.4.4-1.x86_64.rpm
sudo rpm -i --nodeps fahclient-7.4.4-1.x86_64.rpm

telnet localhost 36330
slot-info
options user=usernamehere passkey=yourpasskeyhere team=223518 client-type=beta max-packet-size=big power=full gpu=true
exit

sudo /etc/init.d/FAHClient stop
# wait a moment, cross your finger... then proceed
sudo /etc/init.d/FAHClient start

telnet localhost 36330
slot-info
# you should see your GPU here, and we can delete CPU slot
slot-delete 0
always_on
unpause
exit

tail -f /var/lib/fahclient/log.txt

 

Just realised I completely forgot to thank you for these instructions.  So, thanks, it no doubt saved me lots of hassle getting the right packages installed.

[bafo_ah]

1 minute ago, Alex Atkin UK said:

Just realised I completely forgot to thank you for these instructions.  So, thanks, it no doubt saved me lots of hassle getting the right packages installed.

You're welcome, just glad to bring another people to fold on the cloud... but DON'T FORGET to delete your instance on time, or else google will charge you!

[Alex Atkin UK]

1 minute ago, bafo_ah said:

You're welcome, just glad to bring another people to fold on the cloud... but DON'T FORGET to delete your instance on time, or else google will charge you!

I'm still waiting to see when the site decides to give me an estimated cost.  I set a budget alert, is that enough or will I need to stop early to be absolutely sure?

[LFreeman]

1 minute ago, Alex Atkin UK said:

I'm still waiting to see when the site decides to give me an estimated cost.  I set a budget alert, is that enough or will I need to stop early to be absolutely sure?

I'd stop early just in case

[LordKitsune]

6 minutes ago, Alex Atkin UK said:

I'm still waiting to see when the site decides to give me an estimated cost.  I set a budget alert, is that enough or will I need to stop early to be absolutely sure?

Companies tend to play by the rules "If you forget it's your fault and we aren't obligated to give you a refund" playbook

[LFreeman]

I hope @GOTSpectrum is ok. I haven't seen him all day.

[bafo_ah]

6 minutes ago, Alex Atkin UK said:

I'm still waiting to see when the site decides to give me an estimated cost.  I set a budget alert, is that enough or will I need to stop early to be absolutely sure?

Stop Early I guess... just check your remaining credit daily (Billing section) - usually I make an estimate in percentage - so if I spent <20% daily, it will least 5 days

Spoiler

Remember you need to FINISH all wu slot before terminate instance (use command finish on fahclient telnet commandline)

[n0xlf]

Local news did a story on NCAR in Colorado using their supercomputer for COVID-19:

https://www.9news.com/video/tech/science/supercomputer-called-cheyenne-is-helping-crunch-covid-19-numbers/73-653e6803-ad6d-4d7b-a449-13d6f4e4b872

 

5 petaflops is cool and all (and way more power efficient than us), but doesn't touch what we're doing!

[cbigfoot]

6 minutes ago, LFreeman said:

I hope @GOTSpectrum is ok. I haven't seen him all day.

I think the cat may have killed him (/Jk i hope) @GOTSpectrum didn't update the blog either so my guess is the cat was overly pissed...

[LFreeman]

I wonder if @leadeater knows? They seem to communicate often.

[Jarsky]

Looks like theres an issue with this work server not accepting connections for WU's. A couple of others on Reddit and places having a problem with this server too. 

 

Name:    plfah1-1.mskcc.org
Address:  140.163.4.231

 

01:51:08:WU00:FS01:Connecting to 140.163.4.231:8080
01:51:30:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
01:51:30:WU00:FS01:Connecting to 140.163.4.231:80
01:51:31:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
01:51:51:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 140.163.4.231:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

[cbigfoot]

6 minutes ago, Jarsky said:

Looks like theres an issue with this work server not accepting connections for WU's. A couple of others on Reddit and places having a problem with this server too. 

 

Name:    plfah1-1.mskcc.org
Address:  140.163.4.231

 

01:51:08:WU00:FS01:Connecting to 140.163.4.231:8080
01:51:30:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
01:51:30:WU00:FS01:Connecting to 140.163.4.231:80
01:51:31:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
01:51:51:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 140.163.4.231:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

both of their servers are down...i have 1 job that sent to them somehow today or it timed out and got purged...and one more waiting.   peevs me off it was work 64k now 12k and likely base if it ever sends...if they cant maintain their servers they should be pulled. there is NO EXCUSE for it to be down this long. 

[bafo_ah]

9 minutes ago, Jarsky said:

Looks like theres an issue with this work server not accepting connections for WU's. A couple of others on Reddit and places having a problem with this server too. 

 

Name:    plfah1-1.mskcc.org
Address:  140.163.4.231

 

01:51:08:WU00:FS01:Connecting to 140.163.4.231:8080
01:51:30:WARNING:WU00:FS01:WorkServer connection failed on port 8080 trying 80
01:51:30:WU00:FS01:Connecting to 140.163.4.231:80
01:51:31:WU01:FS01:0x22:Completed 340000 out of 1000000 steps (34%)
01:51:51:WARNING:WU00:FS01:Exception: Failed to send results to work server: Failed to connect to 140.163.4.231:80: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Yeah, nothing we can do about that, but assignment server will assign different collection server on next try. Yesterday I have 1 wu keep try to upload while the next wu 70% completed

[GrockleTD]

2 minutes ago, cbigfoot said:

both of their servers are down...i have 1 job that sent to them somehow today or it timed out and got purged...and one more waiting.   peevs me off it was work 64k now 12k and likely base if it ever sends...if they cant maintain their servers they should be pulled. there is NO EXCUSE for it to be down this long. 

i mean, unless they aren't ALLOWED to go to the servers location to fix whatever is wrong

[Spotty]

Looks like Default doesn't want to be overtaken. 1 billion points more than LTT so far today (by 9pm update).

 

[justpoet]

22 minutes ago, LFreeman said:

I wonder if @leadeater knows? They seem to communicate often.

@GOTSpectrum is probably just CATaTonic...busy pleasing the cat overlord after a large alcoholic tonic.

[cbigfoot]

1 minute ago, bafo_ah said:

Yeah, nothing we can do about that, but assignment server will assign different collection server on next try. Yesterday I have 1 wu keep try to upload while the next wu 70% completed

well(on mine) this is one of his projects and my guess its hard set to go back to only his server

[cbigfoot]

3 minutes ago, Spotty said:

Looks like Default doesn't want to be overtaken. 1 billion points more than LTT so far today (by 9pm update).

 

well if they where to take the bad servers off line we would be a bit higher...but then again so might they...

[Jarsky]

5 minutes ago, cbigfoot said:

both of their servers are down...i have 1 job that sent to them somehow today or it timed out and got purged...and one more waiting.   peevs me off it was work 64k now 12k and likely base if it ever sends...if they cant maintain their servers they should be pulled. there is NO EXCUSE for it to be down this long. 

Yeah just annoying because i had 1 WU timeout, and still have a WU left thats gone from 130k down to 42k and dropping. It's due to timeout in 2 hours. 

[Cabanaj]

Well... AMD and 14541 at it again!

After two crash in the last 24h,  I egt a 3 hour run that gives a wooping 54k.
Now I know it is bad practice to flush a unit from a project,a nd I have never done it, but this one is starting to tempt me...
The low points I can deal with if it the work done is usefull for science , so be it.

But the crashes...make me less patient. Well Okay, it' s true, I hate being cold, and theses days, MY PC is my heating source so there, that's why I'm grumpy.

[cbigfoot]

11 minutes ago, Cabanaj said:

Well... AMD and 14541 at it again!

After two crash in the last 24h,  I egt a 3 hour run that gives a wooping 54k.
Now I know it is bad practice to flush a unit from a project,a nd I have never done it, but this one is starting to tempt me...
The low points I can deal with if it the work done is usefull for science , so be it.

But the crashes...make me less patient. Well Okay, it' s true, I hate being cold, and theses days, MY PC is my heating source so there, that's why I'm grumpy.

yeah it crashed again in my rx580 checked my logs cpu was pegged at 100% at time of failure so dropped 1 more core from cpu lol...well i guess i will go to bed @GOTSpectrum is testing us to see howmany hild out...

[HaRdLy007]

Changed my Cloud instance from 1 V100 to 4 T4's

 

 

[Cabanaj]

4 hours ago, cbigfoot said:

yeah it crashed again in my rx580 checked my logs cpu was pegged at 100% at time of failure so dropped 1 more core from cpu lol

Thanks for the advise will do. But at least on crash (in the middle of the night) was with CPU off (paused after Finishing unit). 
I try to do 2-3 CPU units in a day, because of the release of WU for SARS-COV-2 main spiral protease a few days ago.

[dogwitch]

going to be honest here. compare to everyone. else. you people are lucky... still having wu to get a issue. a bit less ... but not by much