LTT Folding Team's Emergency Response to Covid-19

[Gorgon]

1 hour ago, Inkertus said:

Quick question:  I have a Ryzen 3700x with 12 threads I can spare.  Do you recommend leaving it at the 12 w/1 slot or 6 w/2 slots?

one slot with a higher thread count will produce more points than the same thread count split over two slots because of the Quick Return Bonus (QRB) which gives a disproportianetly larger bonus for Work Units (WUs) returned faster.

 

Having said that on Windows make sure you leave at least one thread free for the OS or it will be very unhappy.

 

The only reason for going with 2 CPU slots is if you get screwed by the "multiple of large primes" bug. In my case I have a 2700x driving 2 GPUs under Linux so 14 threads left over. But 14 is 2 x7 and 7 is considered a large prime so it drops it down to 13 which is also a large prime so it drops it down to 12. So I setup an 8-thread slot and a 6-thread slot as I didn't want to leave any performance on the table

[J1mjam]

42 minutes ago, Shigeru said:

Surpassed the 50k barrier, yay for me XD

How come your points are that low from a total of 31 WU?  Are you not getting the Bonus(QRB)?

[Macaw2000]

5 hours ago, J1mjam said:

Hey guys, 

 

Has anyone had any luck running this stuff in AWS?  I have looked into using spot instances, but I am concerned about the possibility of then having those instances removed and losing the WU. 

 

Of course.  I want to help out more, by donating a little more to the resources but I'm not sure I can really afford the on demand prices for some of the GPU boxes, as they are 4 times the price of the spot instance. 

I run on g4dn.xlarge spot instances running inside containers. Yes, I'll lose the WU if I scale them down because a price trigger happens but I've set it up so the price would have to spike pretty good. 

[Shigeru]

Just now, J1mjam said:

How come your points are that low from a total of 31 WU?  Are you not getting the Bonus(QRB)?

I think what I have, but aside of my poor GT 1030 all the other WUs are CPU ones and I started with that account in March 28

[GalaxyNetworks]

1 hour ago, J1mjam said:

So I got everything working, but only on a Promo k80 instance.  But its better than nothing.  I had to convert the Free Trial to a pay as you go.  It means Ill be charged, but only once i use the free £150.  The free trial is limited to only 4 vCPUs, and all the GPU boxes are 6+. 

 

So now I have a 4 core (E5-2690 v3) slot and a K80 slot going too!  Its only about 30% of the performance of my 1080ti, but every little helps!

Everything counts! I'm also using the promo instances.(2x NC12 Promo) both doing CPU and GPU work. And yes! They are far from powerful. 

[Zberg]

My 2070 back online!  I shall rise from the ashes!

[J1mjam]

2 minutes ago, Macaw2000 said:

I run on g4dn.xlarge spot instances running inside containers. Yes, I'll lose the WU if I scale them down because a price trigger happens but I've set it up so the price would have to spike pretty good. 

It's not just the price spike thats the problem though, is it?  Spot instances, by design, can be spun down by AWS whenever they want, due to demand from pay as you go customers?  Or am I misunderstanding that?

 

1 minute ago, Shigeru said:

I think what I have, but aside of my poor GT 1030 all the other WUs are CPU ones and I started with that account in March 28

Have you set up the Passkey correctly, yea?  I guess the CPU WU would explain the low score, but Id have thought that 20+ WU would have got you a little more by now!

[n0xlf]

9 hours ago, pred1tor83 said:

 

@n0xlf I am glad to hear you are feeling better, how long did it take you to recover?

I'm on day 17 and still have a bit of a cough.  The bad stuff stopped about 5 days ago.

[J1mjam]

2 minutes ago, GalaxyNetworks said:

Everything counts! I'm also using the promo instances.(2x NC12 Promo) both doing CPU and GPU work. And yes! They are far from powerful. 

I'm waiting for my quota increase to go through, so i can spin up a NC12,  I'm stuck on a NC6 right now, a I can only use 10 vCPUs.

[Shigeru]

Just now, J1mjam said:

It's not just the price spike thats the problem though, is it?  Spot instances, by design, can be spun down by AWS whenever they want, due to demand from pay as you go customers?  Or am I misunderstanding that?

 

Have you set up the Passkey correctly, yea?  I guess the CPU WU would explain the low score, but Id have thought that 20+ WU would have got you a little more by now!

Yep! The passkey matchs in all machines, but the other factor is what my CPUs are old models, and A8-5600k, Phenom II X4 and one intel i5 2310 (testing several old processors what I have, maybe the E2160 could be good, but the rest are nahhh)

[Zberg]

 

Fool-proof protection plan is operational

[Macaw2000]

3 minutes ago, J1mjam said:

It's not just the price spike thats the problem though, is it?  Spot instances, by design, can be spun down by AWS whenever they want, due to demand from pay as you go customers?  Or am I misunderstanding that?

Yeah you are correct. It generally doesn't happen though as long as there is plenty of capacity in the availability zone. You piqued my curiosity though so I just filtered the logs on my cluster of 10 containers and indeed:

 

FS01:0x22:Folding@home Core Shutdown: INTERRUPTED 

FS01:0x22:Caught signal SIGINT(2) on PID 753

 

That means a spot instance got nuked. But, considering it's been around 10 spot instance running over four days, I can live with it.

[J1mjam]

1 minute ago, Macaw2000 said:

Yeah you are correct. It generally doesn't happen though as long as there is plenty of capacity in the availability zone. You piqued my curiosity though so I just filtered the logs on my cluster of 10 containers and indeed:

 

FS01:0x22:Folding@home Core Shutdown: INTERRUPTED 

FS01:0x22:Caught signal SIGINT(2) on PID 753

 

That means a spot instance got nuked. But, considering it's been around 10 spot instance running over four days, I can live with it.

Interesting!  Obviously, dumped WU have a few concerns.  If you drop too many, you lose your QRB.  And of course, there's the issue with the lost work, that someone else will have to pick up after the fact.  

 

When the spot instances die, do they just get deallocated, or are their VHD destroyed too.  i.e. is there any way of retrieving the work unit, or at least to upload and inform the server that's it's borked so it doesn't sit until the timeout in order to reassign it to someone else. If you follow me.

[DelphinusVyse]

Came back from getting something to eat to find I had a full set of WUs, Looks like I'll be breaking 4 million soon!

 

Spoiler

 

[Macaw2000]

2 minutes ago, J1mjam said:

Interesting!  Obviously, dumped WU have a few concerns.  If you drop too many, you lose your QRB.  And of course, there's the issue with the lost work, that someone else will have to pick up after the fact.  

 

When the spot instances die, do they just get deallocated, or are their VHD destroyed too.  i.e. is there any way of retrieving the work unit, or at least to upload and inform the server that's it's borked so it doesn't sit until the timeout in order to reassign it to someone else. If you follow me.

 

Yeah it's just a flag: https://aws.amazon.com/premiumsupport/knowledge-center/ami-preserve-ebs-spot/

 

I'm forked from this repo FYI: https://github.com/raykrueger/FoldingOnECS

 

It's a fancier way to fold than just standing up a bunch of servers. Let's me be more "elastic" but it does risk dropping a WU. It looks like I may drop one of out every 300 WU.

[n0xlf]

7 minutes ago, J1mjam said:

When the spot instances die, do they just get deallocated, or are their VHD destroyed too.  i.e. is there any way of retrieving the work unit, or at least to upload and inform the server that's it's borked so it doesn't sit until the timeout in order to reassign it to someone else. If you follow me.

Both options are available (destroy entirely or keep).  Keeping the instance (storage) is new for 2020, so it wasn't that way before.  There are multiple ways to bring the same instance back, so you could do it without losing the WU progress.

[efka112]

10 mil points reached 

[GOTSpectrum]

Just now, efka112 said:

10 mil points reached 

NICE!

[J1mjam]

9 minutes ago, Macaw2000 said:

 

Yeah it's just a flag: https://aws.amazon.com/premiumsupport/knowledge-center/ami-preserve-ebs-spot/

 

I'm forked from this repo FYI: https://github.com/raykrueger/FoldingOnECS

 

It's a fancier way to fold than just standing up a bunch of servers. Let's me be more "elastic" but it does risk dropping a WU. It looks like I may drop one of out every 300 WU.

 

5 minutes ago, n0xlf said:

Both options are available (destroy entirely or keep).  Keeping the instance (storage) is new for 2020, so it wasn't that way before.  There are multiple ways to bring the same instance back, so you could do it without losing the WU progress.

Are you guys able to let me know how you set it up?

 

I looked into raykruegers repo, but I couldnt get it to work as intended.  I find the networks/firewalls easier to configure in Azure.  I also couldnt find a way for the ECS instance to configure the client to not fold as anon with the basic config.xml, without having to manually go and edit the file for each ec2 instance that it created. 

 

Any tips are welcome!

[Macaw2000]

5 minutes ago, J1mjam said:

 

Are you guys able to let me know how you set it up?

 

I looked into raykruegers repo, but I couldnt get it to work as intended.  I find the networks/firewalls easier to configure in Azure.  I also couldnt find a way for the ECS instance to configure the client to not fold as anon with the basic config.xml, without having to manually go and edit the file for each ec2 instance that it created. 

 

Any tips are welcome!

That repo describes the config in the container so you'd have to fork the container and put in your own config.

 

Easier though might be to just have it copy the config from an S3 bucket using curl on instance launch. When you launch an EC2 under advanced you can put some code.

 

Edit @J1mjam here's an example of what you could put in the that code box when you launch. This would install the FAH client, install the NVIDIA tools, and copy in your config.

apt-get update

apt-get install -y curl ocl-icd-opencl-dev nvidia-opencl-dev

rm -rf /var/lib/apt/lists/*

curl --silent --fail -o /fahclient.deb https://download.foldingathome.org/releases/public/release/fahclient/debian-testing-64bit/${BASE}/fahclient_${VERSION}_amd64.deb \

dpkg -x ./fahclient.deb / \

rm /fahclient.deb

curl -o output.file https://J1mjamsbucket.s3-us-west-2.amazonaws.com/config.xml

 

[Plexas]

This a tough event, almost 0 work today  

[Dutch_Master]

32 minutes ago, Zberg said:

<snip>

 

Fool-proof protection plan is operational

Don't worry. Expect the Universe to produce a better fool very soon!

[LordKitsune]

5 minutes ago, Plexas said:

This a tough event, almost 0 work today  

What's your time until next attempt at grabbing a WU?

 

If it's above 1 hour, pause the offending slot(s) and resume them. Or you can try restarting the client, restarting the computer, deleting and restarting slots (both of mine were hung this morning and only got them to attempt reconnecting again by killing the slots)

[HaRdLy007]

Is it possible to use a free AWS account to fold with?

[Costcotoiletpaper]

Has anyone noticed that maybe AWS is folding and not telling anyone?

Essentially, this user is the #1 producer and 3x anonymous.