LTT Folding Team's Emergency Response to Covid-19

[Gustavehomme01]

 

 

 

 

Spoiler

1 minute ago, jctappel67 said:

Your picture seems to be missing, but yeah. $300/month vs $700

 

Any ideal on how often they go down? I'm guessing there's a way to streamline the software install and configuration.. 

[jctappel67]

Just now, Gustavehomme01 said:

Spoiler

 

 

 

 

  Reveal hidden contents

 

Any ideal on how often they go down? I'm guessing there's a way to streamline the software install and configuration.. 

No clue, this is all new to me

[justpoet]

Well…WUs were hard to come by overnight, and more than half my systems were idle this morning, with a few remaining so that still can't get any.

[jctappel67]

Just now, justpoet said:

Well…WUs were hard to come by overnight, and more than half my systems were idle this morning, with a few remaining so that still can't get any.

It's that whole 'needing sleep' thing. That's really killed my production

[Wh0_Am_1]

22 hours ago, LordKitsune said:

How do you know which tab is which? Even tabbing through that with hotkeys has to be exhausting

Opera GX has several features some of which Chrome has recently started to copy. where you can press this symbol

  

and get the tabs listed out for you like this

 

or there is a search function for if you know what tab you are looking for but lost it. 

To use it you select this search button

  

press "Tab" and search for your tab. 

And if you accidentally create a duplicate, you just mouse over it and Opera will light it up. 

the final one that has recently made it into to chrome is the "preview".

[LFreeman]

9 minutes ago, Metallus97 said:

I am the guy who wrote the guide about GCE folding. I did not know about this instance type back then. Sadly I don’t have free credit left and am pretty financially constrained right now (dude to friggin corona).

 

Therefore I cant test this instance type and modify the guide.

annyone wanna help with that?

Shoot me a PM  

FYI Thanks for the guide. So far google has let me use 3 accounts. Sadly I think they caught on.

[Wh0_Am_1]

  

19 hours ago, Favebook said:

You ain't seen nothing yet (but yeah,it was pinning my RAM to 100% (Ram used about 12 out of total 16GB)...

 

 

You have me beat by 200 tabs...

 

19 hours ago, Favebook said:

 (but yeah,it was pinning my RAM to 100% (Ram used about 12 out of total 16GB)... I am lucky that I found this extension, otherwise, my tabs would destroy my memory due to every day usage at 100%:

 

Please, install OneTab, thank me later

Well, in my case, when I open so many tabs, it's not that hard as I got Acer x34, but I usually know where my tabs are approximately and CTRL+Tab (or CTRL+SHIFT+TAB) is great for going through them. And having fast CPU helps a lot.

Opera GX has a RAM saving feature so I am fine. If I ever (rarely) feel like my 32GB of RAM is threatened I can just use the RAM limiter to put them to sleep. (I just restarted my system so most of my tabs are not loaded in yet)  

 

[HaRdLy007]

39 minutes ago, bafo_ah said:

Hope you using centos, paste line by line, its all setup, or you can use this https://cloud.google.com/compute/docs/gpus/install-drivers-gpu

 

  Hide contents

sudo dnf install -y epel-release
sudo dnf install -y nano wget telnet fail2ban fail2ban-systemd
# install fail2ban for security? ddos prevention? just to lazy to change ssh port?
sudo systemctl enable fail2ban
sudo nano /etc/fail2ban/jail.local
# paste this line :
[DEFAULT]
# Ban hosts for one day!:
bantime = 86400

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true

# ctrl+x for exit and then y then enter

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd

sudo dnf install -y kernel
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
sudo dnf install -y http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-repo-rhel8-10.1.243-1.x86_64.rpm
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf clean all
sudo dnf install -y cuda
nvidia-smi # you should see your GPU here, if not, you screwed

wget https://download.foldingathome.org/releases/public/release/fahclient/centos-5.3-64bit/v7.4/fahclient-7.4.4-1.x86_64.rpm
sudo rpm -i --nodeps fahclient-7.4.4-1.x86_64.rpm

telnet localhost 36330
slot-info
options user=usernamehere passkey=yourpasskeyhere team=223518 client-type=beta max-packet-size=big power=full gpu=true
exit

sudo /etc/init.d/FAHClient stop
# wait a moment, cross your finger... then proceed
sudo /etc/init.d/FAHClient start

telnet localhost 36330
slot-info
# you should see your GPU here, and we can delete CPU slot
slot-delete 0
always_on
unpause
exit

tail -f /var/lib/fahclient/log.txt

 

 

I'm stuck at the telnet part. Not sure how to go about it.

[Shlouski]

Managed to get a P4 and P100 working on google cloud, hopefully this will make up for the 3 power cuts I had this morning and many failed WU's.

[Legatus Navium]

I know there are going to be issues, but I still have WU's completed but will not send and now its CPU and GPU WU's I have rebooted the PC's and no effect I have now lost over 300k in points from projects just sitting there. Here is a image to prove my point. This is from my main PC, I have  GPU WU on my laptop that has been stuck since yesterday and it was a 110k project now down to 27k from point loss due to not sending. I know I have lost those WU's and that is what sucks here. 

[bimmerman]

50 minutes ago, Zando Bob said:

They're terrible for a lot of server use, they're XOC boards. The SR-2 can run up to the X5690, 6c/12t Westmere-EP chips from mid 2011. There's much better boards (usually SuperMicro) ones for server use, that are also more usable form factors too. You can ask @bimmerman about the woes of trying to get virtualization and such going on the SR-2. 

SR-X is eh. It's got some issues with RAM but it'd be fine for a server board. But you do have to deal with the HPTX form factor, very few cases currently sold actually support it well. 

yuuuuuuuuuuuuuuuuuuup.

 

SR2 is great at some things. Virtualization is horribly broken due to the PCIe multiplexers, and without overclocking these chips are slooooooooooooooooow.

 

My SR2 with 2x X5675 (2x 6c/12t) at stock has been pounding out 50k PPD. The 2x R9 290s are each doing ~300k.

 

In comparison, My R5 3600 (6c/12t) at stock is doing 115k PPD and my 1080ti is doing 1360k PPD.

 

My power bill this month is gonna be greaaaaaaaaat.

[LordKitsune]

34 minutes ago, justpoet said:

Well…WUs were hard to come by overnight, and more than half my systems were idle this morning, with a few remaining so that still can't get any.

I was kind of having that issue. My GPU would try 10-20 times over night to get a WU

 

Advanced WUs (The thing people have been talking about a lot these last few pages) has pretty well stopped my having downtime.

 

I didn't change when it starts asking for WUs, just what types it requests. I've also noticed I'm getting larger WUs, with a majority of them being worth 100k points or more up from the 60k average I was getting with Normal WUs (though this I feel is more of a luck thing than my settings)

[LordKitsune]

6 minutes ago, Orecal said:

I know there are going to be issues, but I still have WU's completed but will not send and now its CPU and GPU WU's I have rebooted the PC's and no effect I have now lost over 300k in points from projects just sitting there. Here is a image to prove my point. This is from my main PC, I have  GPU WU on my laptop that has been stuck since yesterday and it was a 110k project now down to 27k from point loss due to not sending. I know I have lost those WU's and that is what sucks here. 

Have you paused your slots and restarted the client by first killing the process in task manager then reopening the client?

 

Or perhaps you could try a different internet connection. If your phone supports usb tethering, you can plug your phone into a USB port and share Data with your computer.

[Kilrah]

Argh, second time today that a GPU slot does nothing for an hour+ becasue a download is initiated but stuck

[ConceptOW]

1 hour ago, HaRdLy007 said:

Ubuntu 18.04 not Centos

Try this.. Credits to  @Metallus97

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.243-418.87.00_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-1-local-10.1.243-418.87.00/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

[Tomatodude95]

1 hour ago, bafo_ah said:

Hope you using centos, paste line by line, its all setup, or you can use this https://cloud.google.com/compute/docs/gpus/install-drivers-gpu

 

  Reveal hidden contents

sudo dnf install -y epel-release
sudo dnf install -y nano wget telnet fail2ban fail2ban-systemd
# install fail2ban for security? ddos prevention? just to lazy to change ssh port?
sudo systemctl enable fail2ban
sudo nano /etc/fail2ban/jail.local
# paste this line :
[DEFAULT]
# Ban hosts for one day!:
bantime = 86400

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true

# ctrl+x for exit and then y then enter

sudo systemctl restart fail2ban
sudo fail2ban-client status sshd

sudo dnf install -y kernel
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r)
sudo dnf install -y http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-repo-rhel8-10.1.243-1.x86_64.rpm
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo dnf clean all
sudo dnf install -y cuda
nvidia-smi # you should see your GPU here, if not, you screwed

wget https://download.foldingathome.org/releases/public/release/fahclient/centos-5.3-64bit/v7.4/fahclient-7.4.4-1.x86_64.rpm
sudo rpm -i --nodeps fahclient-7.4.4-1.x86_64.rpm

telnet localhost 36330
slot-info
options user=usernamehere passkey=yourpasskeyhere team=223518 client-type=beta max-packet-size=big power=full gpu=true
exit

sudo /etc/init.d/FAHClient stop
# wait a moment, cross your finger... then proceed
sudo /etc/init.d/FAHClient start

telnet localhost 36330
slot-info
# you should see your GPU here, and we can delete CPU slot
slot-delete 0
always_on
unpause
exit

tail -f /var/lib/fahclient/log.txt

 

If you just use one of their Deep Learning Linux, you will be prompted if you wanna install the Nvidia drivers on the first boot and it does everything automatically. Then you just need to download the .deb file and install it. I think this is much simpler.

[Tomatodude95]

2 hours ago, jctappel67 said:

I figured it out! Thanks to @Macaw2000's advice. I am deleting the old 4 VMs as they finish the current WUs and replacing them with 6 preemptible VMs! 

 

Gonna be a pain in the A$$ to manage for these last 2 days, but it'll be worth it!

 

Hopefully what remains of my credit can last that long as well

 

1 hour ago, Macaw2000 said:

Nice! Folding is an embarrassingly good application for this. You have a non-mission critical workload that is compute intensive. AWS calls it spot, Google calls it preemptible, and Azure calls it Low-priority but all essentially the same thing. Yeah it can be a pain to manage if you don't build some automation but you are on the right track.

Yes, it would be however sadly you can't use your Free Credits here. It's one of the limitations, they don't want you to use your credits too efficiently.
https://cloud.google.com/compute/docs/instances/preemptible

[Atreiumontoya]

25 Million reached!!!

[Tomatodude95]

5 hours ago, Alex Atkin UK said:

Seems a moot point, I'm getting 0 GPU availability at Google Cloud.

 

Its annoying you can't just say "I want this hardware config, I don't care where".

 

5 hours ago, Alex Atkin UK said:

Yeah I was struggling to get CPUs too, it said 0 availability even when I selected 16.  They make it WAY too hard to just setup a VM when it honestly doesn't matter WHERE its located.

Interesting I haven't had any problems getting both 2x P100 and 2x ( 4x T4 ), which I'm switching to at the moment. Never had a single request denied because of capacity, only had to make sure I had the correct quotas in the correct area. I got all of mine at US-west-1, so Oregon.

[jctappel67]

2 minutes ago, Tomatodude95 said:

 

Yes, it would be however sadly you can't use your Free Credits here. It's one of the limitations, they don't want you to use your credits too efficiently.
https://cloud.google.com/compute/docs/instances/preemptible

I think if you upgrade to the Paid account, like you need to to request VMs, they remove that restriction. Will keep y'all updated.

[Tomatodude95]

1 minute ago, jctappel67 said:

I think if you upgrade to the Paid account, like you need to to request VMs, they remove that restriction. Will keep y'all updated.

Yes, please keep us up to date, really want to know as well. However, I don't think it will work, so I'm not gonna risk it by trying it out. To me the Statement is pretty clear, but I could of course be completely wrong.

[ConceptOW]

2 hours ago, Macaw2000 said:

Nice! Folding is an embarrassingly good application for this. You have a non-mission critical workload that is compute intensive. AWS calls it spot, Google calls it preemptible, and Azure calls it Low-priority but all essentially the same thing. Yeah it can be a pain to manage if you don't build some automation but you are on the right track.

I use AWS myself. I have it setup with Auto Scaling Groups where it would delete the instance when the server can't find any WUs by monitoring hw usage through CloudWatch and use my custom AMI with Folding autostart on boot and all my configuration already set. I haven't had any time to figure out how to make it start automatically and at what interval.

[jctappel67]

Just now, Tomatodude95 said:

Yes, please keep us up to date, really want to know as well. However, I don't think it will work, so I'm not gonna risk it by trying it out. To me the Statement is pretty clear, but I could of course be completely wrong.

It's the whole 'Free Tier' part. I looked it up, and I'm pretty sure that once you've upgraded to 'Paid Tier' that limitation is removed.

 

Here's a link to Google's page on the 'Free Tier': https://cloud.google.com/free/docs/gcp-free-tier#how-to-upgrade

[ConceptOW]

1 minute ago, jctappel67 said:

It's the whole 'Free Tier' part. I looked it up, and I'm pretty sure that once you've upgraded to 'Paid Tier' that limitation is removed.

 

Here's a link to Google's page on the 'Free Tier': https://cloud.google.com/free/docs/gcp-free-tier#how-to-upgrade

I'm pretty sure they would just start charging your card instead of your credit.

[LFreeman]

1 minute ago, jctappel67 said:

It's the whole 'Free Tier' part. I looked it up, and I'm pretty sure that once you've upgraded to 'Paid Tier' that limitation is removed.

 

Here's a link to Google's page on the 'Free Tier': https://cloud.google.com/free/docs/gcp-free-tier#how-to-upgrade

@jctappel67 is right you need to go to paid account. don't forget to set a budget. I forgot to on my first one. The budget will stop everything at what you set it at.