LTT Folding Team's Emergency Response to Covid-19

[greblos]

1 hour ago, Bitter said:

I was considering buying a couple 3GB P106-90's because they're super cheap.

https://www.techpowerup.com/gpu-specs/zotac-p106-090.b4832

That card exactly, was going to buy 2 of them and run them both in one box. At 75W they're pretty power efficient but the flops look a little low. I know mining cards are usually crippled with a slower PCIe version, does that hamper folding?

Nothing about these should negatively affect folding performance over something like a GTX 1050 Ti.  They appear to utilize all 16 lanes and are PCIe 3.0, but PCIe bandwidth isn't a huge factor in folding AFAIK, just as it isn't in mining.  I don't know if these can be fully bus powered as a 1050/Ti can.  I suspect they can, but have the 6-pin plug for mining rigs that only provide an x1 electrical interface to the cards.

 

EDIT: I can't figure out definitively if the Zotac cards have any of the bus crippling that seems to be present with the no-name and possibly Gigabyte variants.  PCIe 3 x4 or x8 shouldn't drop PPD too bad, but dropping down to less than PCIe 1 x16 or PCIe 2 x8 might pose a problem.

[danielocdh]

 

1 hour ago, SNReloaded said:

Wish I had had this a week ago! I've had so many times where I come back to my computer & it wasn't doing any folding (for possibly multiple hours at a time).

Thank you for this. : ) I now have this set as a startup program on my computer!

same here but it took me a while to come up with the idea, stop being busy and then code it

 

2 hours ago, grahamjrg said:

Hey there, I'm not really familiar with python or code in general. How would I be able to put this code to use on my pc? I hate seeing the two slots I have waiting on assignments for hours on end

you need to edit/add hosts/ips in the first few lines (and add a password if you use one) save it as babysitter.py in your desktop, you can use notepad just remember to select "save as type": "All files (*.*)" so notepad doesn't add ".txt" to the name.

for the python part install python 3 with default options (latest version on their site is ok) https://www.python.org/downloads/

run the script by double clicking babysitter.py in your desktop

when you want to stop the script just close its window

if you find any problem feel free to reply or write on my feed/dm

 

It is relatively easy but keep in mind that python scripts(and most other types of scripts) just like any program can have functions to do almost anything on your pc/user, so always be cautious with any code you want to run.

[enterprise-psi]

[GalaxyNetworks]

31 minutes ago, enterprise-psi said:

 I think I finally got the points I was missing.

[enterprise-psi]

21 minutes ago, GalaxyNetworks said:

 I think I finally got the points I was missing.

[MrFinn741]

2 hours ago, Bitter said:

Can mining cards be used for folding?

Yes, at least I have been using a p104-100, mining 1070, for two years now. But mining rigs are not optimal, F@H needs something like a 4xPcie link too keep the GPU fed with data, mining rigs usually only have a 1x link.

[therebelious1]

Morning All, 

I'm getting nice and consistent WUs now- fantastic. 

 

However i'm suck trying to upload a completed WU for a couple of days now. 

 

Log Below- repeated dozens of times over the last few days. 

08:07:49:WU00:FS01:Sending unit results: id:00 state:SEND error:NO_ERROR project:11742 run:0 clone:2808 gen:30 core:0x22 unit:0x000000338ca304f15e6999c8eca0594f
08:07:49:WU00:FS01:Uploading 12.57MiB to 140.163.4.241
08:07:49:WU00:FS01:Connecting to 140.163.4.241:8080
08:07:55:WU00:FS01:Upload 26.84%
08:08:01:WU00:FS01:Upload 52.69%
08:08:07:WU00:FS01:Upload 80.02%
08:08:11:WU00:FS01:Upload complete
08:08:11:WU00:FS01:Server responded PLEASE_WAIT (464)
08:08:11:WARNING:WU00:FS01:Failed to send results, will try again later

I know there have been a bunch of these type of issues mentioned previously in this thread, but i can't find a solution. 

 

Any recommendations what to do next? Just wait for it to expire?

 

Cheers, 

G

[enterprise-psi]

I don't get why the gtx970 gets huge juicy 70MB+ WUs , while the 1070ti and tesla get tiny 11MB ones , even though they are configured the same .

[Valentyn]

2 hours ago, GalaxyNetworks said:

 I think I finally got the points I was missing.

 

Mine are only missing in the sense they went to PCMR and another team.

[BlueSpartan]

11 hours ago, danielocdh said:

babysitter python script

It will automatically scan all the slots and pause+unpause slots that are "Waiting On: WS Assignment" and have too high "Next Attempt"

Need to set host(s) and password(only if you use a password)

  Hide contents

 

################################################################################
##                                  options                                   ##
################################################################################
hosts = [ #quoted strings, hosts or IPs separated by comma
  'localhost',
  '192.168.0.123',
]
hostsPassword = '' #quoted string, if the host(s) don't use a password just leave it as: ''

restartLimit = 10 * 60 #in seconds, pause+unpause if next attempt to get WU is this or more
checkEvery = 2 * 60 #in seconds, do a check for all hosts every this seconds

tConTimeout = 15 #in seconds, connection timeout
tReadTimeout = 10 #in seconds, read timeout
testMode = False # if set to True: checkEvery=6 and restartLimit=0 but won't actually pause+unpause slots

################################################################################
##                                    code                                    ##
################################################################################
import json
import re
import telnetlib
import time
import datetime

if testMode:
    restartLimit = 0
    checkEvery = 6
countEvery = 1 #seconds, have to be a factor of checkEvery, default: 1
countEveryDec = max(0, str(countEvery)[::-1].find('.'))
countEveryDecStr = f'{{:.{countEveryDec}f}}'
def remSeconds(seconds):
    if seconds > 0:
        if (seconds * 10000) % (countEvery * 10000) == 0:
            secondsP = countEveryDecStr.format(seconds)
            pr(f'Next check in {secondsP} seconds', same=True)
        time.sleep(countEvery)
        seconds = round((seconds - countEvery) * 10000) / 10000
        remSeconds(seconds)

prLastLen = 0
prLastSame = False
def pr(t, indent=0, same=False, overPrev=False):
    global prLastLen, prLastSame
    if not overPrev and not same and prLastSame:
        prLastLen = 0
        print('')
    t = str(t)
    toPrint = ('  ' * indent) + t
    tLen = len(toPrint)
    print(toPrint + (' ' * max(0, prLastLen - tLen)), end='\r')
    prLastSame = same
    prLastLen = tLen
    if not same:
        print('')
        prLastLen = 0

def checkKeep():
    while (True):
        checkAll()
        remSeconds(checkEvery)

def checkAll():
    for host in hosts: check(host)
    now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    pr(f'check complete at {now}', 0, False, True)


tEnd = ['\n> '.encode('utf-8'), '\n---\n'.encode('utf-8')]
def readResult(expected, expectedResult=''):
    index = expected[0]
    readB = expected[2]
    read = readB.decode('utf-8')
    #noting
    if index < 0 or read == '': return [False, 'nothing was read']
    #expected result
    if expectedResult:
        endWith = tEnd[index].decode('utf-8')
        readStrip = read[0:-len(endWith)].strip()
        if (readStrip != expectedResult):
            return [False, f'{readB}']
    #PyON->json
    match = re.search('\nPyON (\d+) ([-_a-zA-Z\d]+)\n(.*)\n---\n', read, re.DOTALL)
    ###print('');print('');print('');print(index);print(match);print(read);print(readB);print('');
    if match:
        version = match.group(1)
        if version != '1': raise Exception('Response data version does not match')
        data = match.group(3)
        #to json
        data = re.sub('(:\s)?False', r'\1false', data)
        data = re.sub('(:\s)?True', r'\1true', data)
        data = re.sub('(:\s)?None', r'\1null', data)
        data = json.loads(data)
        return [True, data]
    #auth error
    match = re.search('\nERROR: unknown command or variable', read, re.DOTALL)
    if match:
        raise Exception('error sending command, wrong password?')
    #return read
    return [True, read]
def tnCreate(host):
    tn = telnetlib.Telnet(host, 36330, tConTimeout)
    readResult(tn.expect(tEnd, tReadTimeout))
    return tn

def sendCmd(tn, cmd, par=''):
    if cmd == 'auth':
        if hostsPassword:
            cmdStr = f'auth {hostsPassword}';
            tn.write(f'{cmdStr}\n'.encode('utf-8'))
            res = readResult(tn.expect(tEnd, tReadTimeout), 'OK')
            if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
            return res[1]
        return True
    elif cmd == 'exit':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'slot-info' or cmd == 'queue-info':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'get-info-and-restart':
        queueData = sendCmd(tn, 'queue-info')
        slotData = sendCmd(tn, 'slot-info')
        ###
        #if type(queueData) == str: print('');print('');print('');print(queueData);print(queueData.encode('utf-8'));print('');
        #if type(slotData) == str: print('');print('');print('');print(slotData);print(slotData.encode('utf-8'));print('');
        restarted = []
        for slot in slotData:
            isStillRunning = False
            queueDl = False
            for queue in queueData:
                if queue['slot'] == slot['id']:
                    if queue['state'] == 'RUNNING': isStillRunning = True
                    if queue['state'] == 'DOWNLOAD': queueDl = queue
            if not isStillRunning and queueDl and queueDl['waitingon'] == 'WS Assignment':
                match = re.match('\s?(\d+ days?)?\s?(\d+ hours?)?\s?(\d+ mins?)?\s?([\d.]+ secs?)?', queueDl['nextattempt'])
                if match:
                    seconds = 0
                    if match.group(1): seconds += int(re.sub('[^\d.]', '', match.group(1))) * 3600 * 24
                    if match.group(2): seconds += int(re.sub('[^\d.]', '', match.group(2))) * 3600
                    if match.group(3): seconds += int(re.sub('[^\d.]', '', match.group(3))) * 60
                    if match.group(4): seconds += round(float(re.sub('[^\d.]', '', match.group(4))) * 1)
                    if seconds >= restartLimit:
                        if not testMode:
                          sendCmd(tn, 'pause', queueDl['slot'])
                          time.sleep(1)
                          sendCmd(tn, 'unpause', queueDl['slot'])
                        restarted.append([queueDl['slot'], queueDl['nextattempt']])
                else: raise Exception(f'Error with {cmd}, parsing queue nextattempt:{queueDl["nextattempt"]}')
        return restarted
    elif par and (cmd == 'pause' or cmd == 'unpause'):
        cmdStr = f'{cmd} {par}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    else : return False

def check(host):
    st = time.time()
    pr(f'checking {host}', 1, True)
    try:
        tn = tnCreate(host)
        sendCmd(tn, 'auth')
        restarted = sendCmd(tn, 'get-info-and-restart')
        if len(restarted):
            pr(f'{host}: restarted {len(restarted)} slot{"s" if len(restarted) > 1 else ""}: ' + ', '.join(map(lambda item: '' + (' with '.join(item)), restarted)), 1, False, True)
        sendCmd(tn, 'exit')
        ed = time.time()
        time.sleep(max(0, 1 - (ed - st)))
    except Exception as err:
        pr(f'{host} error: {err}', 1, False, True)

checkKeep()

 

 

Looks like this when running:

 

It access the clients API in a similar(much simpler) way that FAHControl or HFM.NET do

It won't restart slots that are still running and trying to download a new WU at the same time.

Tested on python 3 on windows (3.8.2)and ubuntu (3.6.9)

Let me know if you find any issues, it was stable for me after a few hours

 

What IP's need to be added? I tried my local host IP and got "telnet closed/timed out" ETC.

[Psedog]

18 hours ago, Psedog said:

I'm pretty sure that's it as well, but if I can increase the odds of success. I just uploaded a successful project to them, but it has failed a few before. It likes to fail when I'm not actively using the computer as well. We'll see. 

Welp, shutting down the CPU folding didn't change the fact that 14541 doesn't like AMD cards. Turning CPU folding back on, lol.

 

So disheartening to see 93% then fail

[ffbr]

2 minutes ago, Psedog said:

Welp, shutting down the CPU folding didn't change the fact that 14541 doesn't like AMD cards. Turning CPU folding back on, lol.

 

So disheartening to see 93% then fail

Yep had my first recorder encounter with the dreaded 14541 today.... nice it failed at 20% 

[marknd59]

I got HFM.NET setup last night. The results of the of my AMD cards Vs Project 14541 so far -

 

RX580  -  2 wins    3 losses

 

RX590  -  2 wins    1 lost

[cbigfoot]

12 minutes ago, marknd59 said:

I got HFM.NET setup last night. The results of the of my AMD cards Vs Project 14541 so far -

 

RX580  -  2 wins    3 losses

 

RX590  -  2 wins    1 lost

howmany open CPU threads do you have for a project like tose a min of 2 is needed per GPU possibly 3 i have 5 open on Bender as it was regularly hitting 100% usage on cpu and you could see ppd drop on the vcard due to that. i would rather loose some of a little vs some of alot...

on another note woke up to no more adv or normal gpu units...grrrrrrrrr 2 idled GPUS and 1 idled CPU did what i had to to get them crunching...

[marknd59]

2 minutes ago, cbigfoot said:

howmany open CPU threads do you have for a project like tose a min of 2 is needed per GPU possibly 3

 I am running a R7 1700. I am using 10 for F@H so that leaves 6 for everything else. CPU load is at  70 - 75%

[cbigfoot]

Just now, marknd59 said:

 I am running a R7 1700. I am using 10 for F@H so that leaves 6 for everything else. CPU load is at  70 - 75%

ok well could be the wus need to be reworked on their end too

that is 4 days of production wus lol look at that chonky # of wus on the 5th line lol that my 3900x ad need to look at the logs of my 1700 rx580 rig...

[Stormseeker9]

10:26:47:WARNING:WU03:FS02:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
10:26:47:WU03:FS02:Connecting to 18.218.241.186:80
10:26:47:WARNING:WU03:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
10:26:47:ERROR:WU03:FS02:Exception: Could not get an assignment

 

and it started again  Sigh!

 

Advanced flag did really help the last 24 hours. CPU folding away, just need to get GPU back up and runningn

[marknd59]

Well I got an answer to where my new hardware is.

"We are processing all orders but are continuing to experience unprecedented volumes and we want to assure you that our warehouse are working hard to meet the delivery estimated upon checkout, whilst this is our top priority delivery timeframes cannot currently be provided."

 

I think that means I'll get it when I get it stop bugging us.

[cbigfoot]

2 minutes ago, Stormseeker9 said:

10:26:47:WARNING:WU03:FS02:Failed to get assignment from '65.254.110.245:8080': No WUs available for this configuration
10:26:47:WU03:FS02:Connecting to 18.218.241.186:80
10:26:47:WARNING:WU03:FS02:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
10:26:47:ERROR:WU03:FS02:Exception: Could not get an assignment

 

and it started again  Sigh!

 

Advanced flag did really help the last 24 hours. CPU folding away, just need to get GPU back up and runningn

well i tried normal but got nothing after 3 tries so enabled beta as this is a competition lol but would switch back to advanced once you get a wu...which is what i did.

[chaozbandit]

47 minutes ago, Psedog said:

Welp, shutting down the CPU folding didn't change the fact that 14541 doesn't like AMD cards. Turning CPU folding back on, lol.

 

So disheartening to see 93% then fail

That's odd, I had two pass on separate 290s last night.

 

 

One of them did get 3 bad WUs from a separate project though.

[cbigfoot]

1 minute ago, chaozbandit said:

That's odd, I had two pass on separate 290s last night.

 

 

One of them did get 3 bad WUs from a separate project though.

yep all the fails on mty aio are 11xxx projects...so im rebooting it.

[efka112]

21 minutes ago, cbigfoot said:

howmany open CPU threads do you have for a project like tose a min of 2 is needed per GPU possibly 3 i have 5 open on Bender as it was regularly hitting 100% usage on cpu and you could see ppd drop on the vcard due to that. i would rather loose some of a little vs some of alot...

on another note woke up to no more adv or normal gpu units...grrrrrrrrr 2 idled GPUS and 1 idled CPU did what i had to to get them crunching...

i'm only using 1 for GPU and have no issue with any WU

[PineyCreek]

Lol...everyone kicking it into high gear at LTT based on team stats.

 

Monthly Production
Month	Points	WUs
04.20	27,318,409,576	1,161,456
03.20	37,275,460,526	1,570,501
02.20	4,752,852,713	126,196
01.20	5,080,434,187	137,732

[Fulminatrix]

1 hour ago, BlueSpartan said:

What IP's need to be added? I tried my local host IP and got "telnet closed/timed out" ETC.

Just delete the line '192.168.0.123',

If you run F@H on the same computer as the script it should work

[cbigfoot]

12 minutes ago, efka112 said:

i'm only using 1 for GPU and have no issue with any WU

well for that project i have seen it eat away at threads up to 2 sometimes 3 (now it may be able to cope with 1 but it may get slowed down...and CPU wus are so much less energy efficient...but for those that care here are my failures in pure view...note my nvidia card had 5 failures but that was due to opencl getting turned on somehow on it and it didnt report as bad wus it returned due to rejected...