LTT Folding Team's Emergency Response to Covid-19

[ShortRouter]

1 minute ago, Costcotoiletpaper said:

Has anyone noticed that maybe AWS is folding and not telling anyone?

Essentially, this user is the #1 producer and 3x anonymous. 

https://folding.extremeoverclocking.com/team_summary.php?s=&t=238068 ;)

[J1mjam]

19 minutes ago, Macaw2000 said:

That repo describes the config in the container so you'd have to fork the container and put in your own config.

 

Easier though might be to just have it copy the config from an S3 bucket using curl on instance launch. When you launch an EC2 under advanced you can put some code.

 

Edit @J1mjam here's an example of what you could put in the that code box when you launch. This would install the FAH client, install the NVIDIA tools, and copy in your config.

apt-get update

apt-get install -y curl ocl-icd-opencl-dev nvidia-opencl-dev

rm -rf /var/lib/apt/lists/*

curl --silent --fail -o /fahclient.deb https://download.foldingathome.org/releases/public/release/fahclient/debian-testing-64bit/${BASE}/fahclient_${VERSION}_amd64.deb \

dpkg -x ./fahclient.deb / \

rm /fahclient.deb

curl -o output.file https://J1mjamsbucket.s3-us-west-2.amazonaws.com/config.xml

 

Thanks for that.  So would the script go in the AWS UI somewhere once deployed?  Or would I be aiming to change the CDK scripts to run that.  

As far as I can tell, once deployed, the spot requests are created instantly, and the VMs spun up.  Sorry, I've done a ton of work in Lambdas but very little work with EC2!

[Knuschberkeks]

I was just wondering: I don't have the most balling gaming rig, but what I have is a storage unit full of older hardware. Is it a good idea to run F@H on that? Can it even finish WUs in time? I'm talking about stuff like early to mid 2000s macs (both PowerPC and Intel, most of them with a GPU), a top of the line 2005 Gaming Laptop (Dell XPS M170), an old PC with I think a Geforce GT520. Anything retro that doesn't have a GPU is probably not worth it (modern CPUs barely are as it stands).

Is it worth my time to go and install some kind of linux on all of these?

[SlurmWorm]

I'm trying to participate in folding and just reached 1M over the last week -- I'd like to dedicate that 1M to whoever the kind admin on the LTT mindcraft server who donated 40K to my son(11) who, after spending a lot effort making a town, woke up to its erasure because neither of us knew *towns* had to pay tax ... we're building another town at the moment (he just completed a chicken farm where the chickens fall into a hole and a piston pushes the chicken into a stove, gets cooked, then gets put into a chest via hoppers. This stuff is hilarious).

 

Anyhow...

 

I understand that there are bottlenecks and hardware constraints with F@H trying to ramp up to the onslaught of participants.

 

However, I have a 16/32 AMD 3950X cpu sitting here going unused overnight (I have no idea why I'm not getting jobs from 2am to 9am but that's what's happening).

 

I'm also getting very small jobs of ~5000 points which the machine chews through in a few minutes.  Then it sits there waiting for another small job.  The machine finally got it's hands on a 40,000 point job and was done in an hour.

 

So the question is this -- does the client (cli only) have an option to *insist* on large cpu intensive jobs?  I've tried spamming the client by restarting it constantly until it receives a job but my patience with that wore off rather quickly.

Edited March 30, 2020 by SlurmWorm
UPDATE: System is running Ubuntu 19.10

[Plexas]

18 minutes ago, LordKitsune said:

What's your time until next attempt at grabbing a WU?

 

If it's above 1 hour, pause the offending slot(s) and resume them. Or you can try restarting the client, restarting the computer, deleting and restarting slots (both of mine were hung this morning and only got them to attempt reconnecting again by killing the slots)

Im working from home so i pretty much babysit the client. It finally picked up a GPU WU, after some 5 hours of nothing.

[Butwhy292]

10mil is the goal for the end of the day!

[cbigfoot]

3 minutes ago, SlurmWorm said:

I'm trying to participate in folding and just reached 1M over the last week -- I'd like to dedicate that 1M to whoever the kind admin on the LTT mindcraft server who donated 40K to my son(11) who, after spending a lot effort making a town, woke up to its erasure because neither of us knew *towns* had to pay tax ... we're building another town at the moment (he just completed a chicken farm where the chickens fall into a hole and a piston pushes the chicken into a stove, gets cooked, then gets put into a chest via hoppers. This stuff is hilarious).

 

Anyhow...

 

I understand that there are bottlenecks and hardware constraints with F@H trying to ramp up to the onslaught of participants.

 

However, I have a 16/32 AMD 3950X cpu sitting here going unused overnight (I have no idea why I'm not getting jobs from 2am to 9am but that's what's happening).

 

I'm also getting very small jobs of ~5000 points which the machine chews through in a few minutes.  Then it sits there waiting for another small job.  The machine finally got it's hands on a 40,000 point job and was done in an hour.

 

So the question is this -- does the client (cli only) have an option to *insist* on large cpu intensive jobs?  I've tried spamming the client by restarting it constantly until it receives a job but my patience with that wore off rather quickly.

add under advanced Client-type  advanced it keeps my 3900 fed non stop

[Macaw2000]

17 minutes ago, J1mjam said:

Thanks for that.  So would the script go in the AWS UI somewhere once deployed?  Or would I be aiming to change the CDK scripts to run that.  

As far as I can tell, once deployed, the spot requests are created instantly, and the VMs spun up.  Sorry, I've done a ton of work in Lambdas but very little work with EC2!

Yeah if you're doing it by hand you can 1) put in a script to run once the instance is created where it says "user data"

 

Or 2) just ssh into the instance and run those commands. I'd do #2 first just so you can see how things work then you can spin up other instances after.

 

Edit: I wouldn't mess with the CDK until you get happy with EC2, spot, etc. Lambda - way cool!

[hydrochloric]

Spoiler

Hmm, I have one GPU consistently pumping out bad work units...  That Vega has always been a pain, time to try declocking it slightly.

 

How often do all of you get failed WUs?  Last night was the first one I didn't have a machine crash on me, but all of them produced at least one bad WU on the GPU.  One is way worse than the others, but...

[Atreiumontoya]

Got both my gaming systems up and folding now my secondary 3700X and 1070TI are chugging away along with my 3950 and 2080TI, Should make some great strides willing they keep me in WU's

[Unilevers]

2 minutes ago, hydrochloric said:

  Reveal hidden contents

Hmm, I have one GPU consistently pumping out bad work units...  That Vega has always been a pain, time to try declocking it slightly.

 

How often do all of you get failed WUs?  Last night was the first one I didn't have a machine crash on me, but all of them produced at least one bad WU on the GPU.  One is way worse than the others, but...

one of my 2080Ti's decides it wants to have a bad time and will throw 2 or 3 bad WU in a row then smarten up and be fine for the rest of the day.  No idea why.

[J1mjam]

2 minutes ago, Macaw2000 said:

Yeah if you're doing it by hand you can 1) put in a script to run once the instance is created where it says "user data"

 

Or 2) just ssh into the instance and run those commands. I'd do #2 first just so you can see how things work then you can spin up other instances after.

 

Edit: I wouldn't mess with the CDK until you get happy with EC2, spot, etc. Lambda - way cool!

Gotcha, you're talking about just spinning up a spot instance manually for now! I'll give it a go!

 

Thanks!

[SlurmWorm]

5 minutes ago, cbigfoot said:

add under advanced Client-type  advanced it keeps my 3900 fed non stop

I tried installing fahcontrol but it requires python-gnome2 and Ubuntu 19.10 doesn't seem to link to it anymore.

 

sudo apt-get install -y python-gnome2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package python-gnome2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python-gnome2' has no installation candidate

 

Running this then requires that I use the Synaptic Package Manager to fix broken packages.  So I was hoping the cli could take in these parameters rather than through the fahcontrol gui.

[J1mjam]

NOOOOOOO! My Desktop just blue screened!   Lost 60% of two work units, they wouldn't restart. (What's the point of checkpoints anyways?)

 

Its been running fine for 48 hours.

 

Can I have an F in chat

[hydrochloric]

1 minute ago, Unilevers said:

one of my 2080Ti's decides it wants to have a bad time and will throw 2 or 3 bad WU in a row then smarten up and be fine for the rest of the day.  No idea why.

So far all of them have done one, which I chalk up to bad luck.  It does seem like the initial ramp-up on the start of the folding is the worst time for stability, once they've started I've been able to tweak the clocks and they stay happy, but as soon as a new WU starts at those clocks it gets pissy.

 

I declocked the vega a touch, but left the voltages alone.  Hopefully that reduces how often it throws bad WUs, given it's one of my higher PPD cards.  That compute lol.

[Gorgon]

7 minutes ago, hydrochloric said:

  Reveal hidden contents

Hmm, I have one GPU consistently pumping out bad work units...  That Vega has always been a pain, time to try declocking it slightly.

 

How often do all of you get failed WUs?  Last night was the first one I didn't have a machine crash on me, but all of them produced at least one bad WU on the GPU.  One is way worse than the others, but...

12 Failed out of 6893 WUs so far this year. Most of those were instability when testing Core22 in Beta due to overclocks on Turing and 3 due to a Dodgy Beta WU the other day.

 

Yes, Consistent WU failures usually mean a too high overclock or hardware problems

[Kenpachi1985]

We have broken 8k comments for this topic now. Has there ever been a "not offtopic" topic that made it so far?

[hydrochloric]

1 minute ago, Gorgon said:

12 Failed out of 6893 WUs so far this year. Most of those were self-inflicted and 3 due to a Dodgy Beta WU the other day.

 

Yes, Consistent WU failures usually mean a too high overclock or hardware problems

Nuts thing is, I removed all the OC's any of them had a few days ago after it was failure after failure, so everything is factory.  That said, 2/3 are 3rd party OC parts (AirBoost and the Dual-X) so I can try pulling them back to normal AMD specs.

[Guest]

4 minutes ago, SlurmWorm said:

I tried installing fahcontrol but it requires python-gnome2 and Ubuntu 19.10 doesn't seem to link to it anymore.

 

sudo apt-get install -y python-gnome2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package python-gnome2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python-gnome2' has no installation candidate

 

Running this then requires that I use the Synaptic Package Manager to fix broken packages.  So I was hoping the cli could take in these parameters rather than through the fahcontrol gui.

Do you have any Windows PCs? I just manage all of my linux machines from control on 1 Windows PC.

[Guest]

 

Added another GPU. 

I'm not being competitive, I just was bored.

[Macaw2000]

8 minutes ago, J1mjam said:

Gotcha, you're talking about just spinning up a spot instance manually for now! I'll give it a go!

 

Thanks!

 

Yeah was just thinking maybe start manually, then try CloudFormation if you wan to automate, then use CDK if you want to get fancy

[cbigfoot]

12 minutes ago, hydrochloric said:

  Reveal hidden contents

Hmm, I have one GPU consistently pumping out bad work units...  That Vega has always been a pain, time to try declocking it slightly.

 

How often do all of you get failed WUs?  Last night was the first one I didn't have a machine crash on me, but all of them produced at least one bad WU on the GPU.  One is way worse than the others, but...

My 5700xt is 12good and 0 bad 

sorry no vega but yeah no OC is better with F@H

[Gorgon]

4 minutes ago, hydrochloric said:

So far all of them have done one, which I chalk up to bad luck.  It does seem like the initial ramp-up on the start of the folding is the worst time for stability, once they've started I've been able to tweak the clocks and they stay happy, but as soon as a new WU starts at those clocks it gets pissy.

 

I declocked the vega a touch, but left the voltages alone.  Hopefully that reduces how often it throws bad WUs, given it's one of my higher PPD cards.  That compute lol.

I've seen this type of behaviour on NVidia when power-limiting cards really low with a high clock offset to compensate. When the WU returns from a check-point where the CPU is busy but the GPU idle and the clocks ramp up because the card is cooler it boosts higher natively and the offset sends it into unstable territory.

 

With some Core22 WUs throwing large amounts of power I've just given up on overclocking except a 1070Ti I have that for some reason runs at 1800MHz natively. That one gets +100MHz and runs solid.

 

In the old (Core 21) days it could take months to dial in a safe overclock. Essentially you had to see it run for a month or two with no WU errors and during a competition was not the time to do it. With Core22 that's all gone out the window.

 

I just run stock as failing a WU at 80-90% due to an o/c will wipe any minor % increase from the overclock.

[Gorgon]

7 minutes ago, hydrochloric said:

Nuts thing is, I removed all the OC's any of them had a few days ago after it was failure after failure, so everything is factory.  That said, 2/3 are 3rd party OC parts (AirBoost and the Dual-X) so I can try pulling them back to normal AMD specs.

I had to do that (negative offset) with my Turing Cards when Beta Testing Core22. Pascal worked just fine but Turing needed -2-300. They seem to have since fixed that.

[hydrochloric]

 

1 minute ago, Gorgon said:

I had to do that (negative offset) with my Turing Cards when Beta Testing Core22. Pascal worked just fine but Turing needed -2-300. They seem to have since fixed that.

For some reason the radeon settings is only letting me go max -2% on core clock though.

 

The 5700XT is running totally factory, as is the 7970.  I'd feel bad pulling that one below 1GHz lol.