LTT Folding Team's Emergency Response to Covid-19

[grahamjrg]

Just passed over 1,000,000 points in the event!  I'm in the top 3% of all folders (worldwide, not just the event). Amazing things you can do with a RX 580 and a couple weeks running on idle

[Bitter]

7 minutes ago, ObeliskAG said:

What's your sustained core clock on the Vega56?

1560 for folding at 1150mv, for gaming I run around 1590-1600 at 1125 since stability isn't as important. Due to how Vega clocks weird, when folding it bounces between 1550-1580 so I run the voltage a little higher than needed to ensure stability.

[danielocdh]

babysitter python script

updated version by Koppa315:

Spoiler

#!/usr/bin/python3
# 1.0 - Original Code Belongs to @danielocdh 
# 1.1 - Added ability to check FAH Control APIs on other ports
# 1.2 - Added support for Linux Control API (Slight changes in response checks)
# 1.3 - Fix for latest version of FAH Control API and Client (7.6.9)
# 1.4 - Added @danielocdh's feedback and his local changes around !# and spacing in the substitution
# 1.5 - @danielocdh updated expected result for authentication issues
# 1.6 - Removed un-needed tEnd references to end of readResult

################################################################################
##                                  options                                   ##
################################################################################
hosts = [ #list of quoted strings, hosts or IPs, with optional colon separted port (e.g. localhost:36331), separated by comma
  'localhost',
]
hostsPassword = '' #quoted string, if the host(s) don't use a password just leave it as: ''

restartLimit = 10 * 60 #in seconds, pause+unpause if next attempt to get WU is this or more
checkEvery = 2 * 60 #in seconds, do a check for all hosts every this seconds

tConTimeout = 15 #in seconds, connection timeout
tReadTimeout = 10 #in seconds, read timeout
testMode = False # if set to True: checkEvery=6 and restartLimit=0 but won't actually pause+unpause slots

################################################################################
##                                    code                                    ##
################################################################################
import json
import re
import telnetlib
import time
import datetime

if testMode:
    restartLimit = 0
    checkEvery = 6
countEvery = 1 #seconds, have to be a factor of checkEvery, default: 1
countEveryDec = max(0, str(countEvery)[::-1].find('.'))
countEveryDecStr = f'{{:.{countEveryDec}f}}'
def remSeconds(seconds):
    if seconds > 0:
        if (seconds * 10000) % (countEvery * 10000) == 0:
            secondsP = countEveryDecStr.format(seconds)
            pr(f'Next check in {secondsP} seconds', same=True)
        time.sleep(countEvery)
        seconds = round((seconds - countEvery) * 10000) / 10000
        remSeconds(seconds)

prLastLen = 0
prLastSame = False
def pr(t, indent=0, same=False, overPrev=False):
    global prLastLen, prLastSame
    if not overPrev and not same and prLastSame:
        prLastLen = 0
        print('')
    t = str(t)
    toPrint = ('  ' * indent) + t
    tLen = len(toPrint)
    print(toPrint + (' ' * max(0, prLastLen - tLen)), end='\r')
    prLastSame = same
    prLastLen = tLen
    if not same:
        print('')
        prLastLen = 0

def checkKeep():
    while (True):
        checkAll()
        remSeconds(checkEvery)

def checkAll():
    for host in hosts: check(host)
    now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    pr(f'check complete at {now}', 0, False, True)


prompt = '\n*>\s*'.encode('utf-8') 
pyonEnd = '\n---\n'.encode('utf-8')
def readResult(expected, expectedResult=''):
    index = expected[0]
    readB = expected[2]
    read = readB.decode('utf-8')
    #nothing
    if index < 0 or read == '': return [False, 'nothing was read']
    #expected result
    if expectedResult:
        result = re.sub('\s+>$', '', read.strip())
        if (result != expectedResult):
            return [False, f'{readB}']
    #PyON->json
    match = re.search('\n*PyON\s+(\d+)\s+([-_a-zA-Z\d]+)\n(.*)\n---\n', read, re.DOTALL)
    ###
    #print('');print('');print('');print(index);print(match);print("read");print(read);print("readB");print(readB);print('');
    if match:
        version = match.group(1)
        if version != '1': raise Exception('Response data version does not match')
        data = match.group(3)
        #to json
        data = re.sub('(:\s*)False', r'\1false', data)
        data = re.sub('(:\s*)True', r'\1true', data)
        data = re.sub('(:\s*)None', r'\1null', data)
        data = json.loads(data)
        return [True, data]
    #auth error
    match = re.search('\nERROR: unknown command or variable', read, re.DOTALL)
    if match:
        raise Exception('error sending command, wrong password?')
    #return read
    return [True, read]
def tnCreate(host):
    match = re.search('(.*):(\d+)', host);
    port = 36330;
    tEnd = [prompt];
    
    if match:
        host = match.group(1);
        port = match.group(2);
        
    tn = telnetlib.Telnet(host, port, tConTimeout)
    readResult(tn.expect(tEnd, tReadTimeout))
    return tn

def sendCmd(tn, cmd, par=''):
    ###
    #print(cmd);
    if cmd == 'auth':
        tEnd = [prompt]
        if hostsPassword:
            cmdStr = f'auth {hostsPassword}';
            tn.write(f'{cmdStr}\n'.encode('utf-8'))
            res = readResult(tn.expect(tEnd, tReadTimeout), 'OK')
            if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
            return res[1]
        return True
    elif cmd == 'exit':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        tEnd = [prompt]
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'slot-info' or cmd == 'queue-info':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        tEnd = [pyonEnd]
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'get-info-and-restart':
        queueData = sendCmd(tn, 'queue-info')
        slotData = sendCmd(tn, 'slot-info')
        
        ###
        #if type(queueData) == str: print('');print('');print('');print(queueData);print(queueData.encode('utf-8'));print('');
        #if type(slotData) == str: print('');print('');print('');print(slotData);print(slotData.encode('utf-8'));print('');
        restarted = []
        for slot in slotData:
            isStillRunning = False
            queueDl = False
            for queue in queueData:
                if queue['slot'] == slot['id']:
                    if queue['state'] == 'RUNNING': isStillRunning = True
                    if queue['state'] == 'DOWNLOAD': queueDl = queue
            if not isStillRunning and queueDl and queueDl['waitingon'] == 'WS Assignment':
                match = re.match('\s?(\d+ days?)?\s?(\d+ hours?)?\s?(\d+ mins?)?\s?([\d.]+ secs?)?', queueDl['nextattempt'])
                if match:
                    seconds = 0
                    if match.group(1): seconds += int(re.sub('[^\d.]', '', match.group(1))) * 3600 * 24
                    if match.group(2): seconds += int(re.sub('[^\d.]', '', match.group(2))) * 3600
                    if match.group(3): seconds += int(re.sub('[^\d.]', '', match.group(3))) * 60
                    if match.group(4): seconds += round(float(re.sub('[^\d.]', '', match.group(4))) * 1)
                    if seconds >= restartLimit:
                        if not testMode:
                            sendCmd(tn, 'pause', queueDl['slot'])
                            time.sleep(1)
                            sendCmd(tn, 'unpause', queueDl['slot'])
                        restarted.append([queueDl['slot'], queueDl['nextattempt']])
                else: raise Exception(f'Error with {cmd}, parsing queue nextattempt:{queueDl["nextattempt"]}')
        return restarted
    elif par and (cmd == 'pause' or cmd == 'unpause'):
        tEnd = [prompt]
        cmdStr = f'{cmd} {par}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    else : return False

def check(host):
    st = time.time()
    pr(f'checking {host}', 1, True)
    try:
        tn = tnCreate(host)
        sendCmd(tn, 'auth')
        restarted = sendCmd(tn, 'get-info-and-restart')
        if len(restarted):
            pr(f'{host}: restarted {len(restarted)} slot{"s" if len(restarted) > 1 else ""}: ' + ', '.join(map(lambda item: '' + (' with '.join(item)), restarted)), 1, False, True)
        sendCmd(tn, 'exit')
        ed = time.time()
        time.sleep(max(0, 1 - (ed - st)))
    except Exception as err:
        pr(f'{host} error: {err}', 1, False, True)

checkKeep()

 

It will automatically scan all the slots and pause+unpause slots that are "Waiting On: WS Assignment" and have too high "Next Attempt"

Need to set host(s) and password(only if you use a password)

If you find an issue with it, let me know via message

 

Looks like this when running:

 

It access the clients API in a similar(much simpler) way that FAHControl or HFM.NET do

It won't restart slots that are still running and trying to download a new WU at the same time.

 

original version(might not work anymore):

Spoiler

 

################################################################################
##                                  options                                   ##
################################################################################
hosts = [ #quoted strings, hosts or IPs separated by comma
  'localhost',
  '192.168.0.123',
]
hostsPassword = '' #quoted string, if the host(s) don't use a password just leave it as: ''

restartLimit = 10 * 60 #in seconds, pause+unpause if next attempt to get WU is this or more
checkEvery = 2 * 60 #in seconds, do a check for all hosts every this seconds

tConTimeout = 15 #in seconds, connection timeout
tReadTimeout = 10 #in seconds, read timeout
testMode = False # if set to True: checkEvery=6 and restartLimit=0 but won't actually pause+unpause slots

################################################################################
##                                    code                                    ##
################################################################################
import json
import re
import telnetlib
import time
import datetime

if testMode:
    restartLimit = 0
    checkEvery = 6
countEvery = 1 #seconds, have to be a factor of checkEvery, default: 1
countEveryDec = max(0, str(countEvery)[::-1].find('.'))
countEveryDecStr = f'{{:.{countEveryDec}f}}'
def remSeconds(seconds):
    if seconds > 0:
        if (seconds * 10000) % (countEvery * 10000) == 0:
            secondsP = countEveryDecStr.format(seconds)
            pr(f'Next check in {secondsP} seconds', same=True)
        time.sleep(countEvery)
        seconds = round((seconds - countEvery) * 10000) / 10000
        remSeconds(seconds)

prLastLen = 0
prLastSame = False
def pr(t, indent=0, same=False, overPrev=False):
    global prLastLen, prLastSame
    if not overPrev and not same and prLastSame:
        prLastLen = 0
        print('')
    t = str(t)
    toPrint = ('  ' * indent) + t
    tLen = len(toPrint)
    print(toPrint + (' ' * max(0, prLastLen - tLen)), end='\r')
    prLastSame = same
    prLastLen = tLen
    if not same:
        print('')
        prLastLen = 0

def checkKeep():
    while (True):
        checkAll()
        remSeconds(checkEvery)

def checkAll():
    for host in hosts: check(host)
    now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    pr(f'check complete at {now}', 0, False, True)


tEnd = ['\n> '.encode('utf-8'), '\n---\n'.encode('utf-8')]
def readResult(expected, expectedResult=''):
    index = expected[0]
    readB = expected[2]
    read = readB.decode('utf-8')
    #noting
    if index < 0 or read == '': return [False, 'nothing was read']
    #expected result
    if expectedResult:
        endWith = tEnd[index].decode('utf-8')
        readStrip = read[0:-len(endWith)].strip()
        if (readStrip != expectedResult):
            return [False, f'{readB}']
    #PyON->json
    match = re.search('\nPyON (\d+) ([-_a-zA-Z\d]+)\n(.*)\n---\n', read, re.DOTALL)
    ###print('');print('');print('');print(index);print(match);print(read);print(readB);print('');
    if match:
        version = match.group(1)
        if version != '1': raise Exception('Response data version does not match')
        data = match.group(3)
        #to json
        data = re.sub('(:\s)?False', r'\1false', data)
        data = re.sub('(:\s)?True', r'\1true', data)
        data = re.sub('(:\s)?None', r'\1null', data)
        data = json.loads(data)
        return [True, data]
    #auth error
    match = re.search('\nERROR: unknown command or variable', read, re.DOTALL)
    if match:
        raise Exception('error sending command, wrong password?')
    #return read
    return [True, read]
def tnCreate(host):
    tn = telnetlib.Telnet(host, 36330, tConTimeout)
    readResult(tn.expect(tEnd, tReadTimeout))
    return tn

def sendCmd(tn, cmd, par=''):
    if cmd == 'auth':
        if hostsPassword:
            cmdStr = f'auth {hostsPassword}';
            tn.write(f'{cmdStr}\n'.encode('utf-8'))
            res = readResult(tn.expect(tEnd, tReadTimeout), 'OK')
            if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
            return res[1]
        return True
    elif cmd == 'exit':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'slot-info' or cmd == 'queue-info':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'get-info-and-restart':
        queueData = sendCmd(tn, 'queue-info')
        slotData = sendCmd(tn, 'slot-info')
        ###
        #if type(queueData) == str: print('');print('');print('');print(queueData);print(queueData.encode('utf-8'));print('');
        #if type(slotData) == str: print('');print('');print('');print(slotData);print(slotData.encode('utf-8'));print('');
        restarted = []
        for slot in slotData:
            isStillRunning = False
            queueDl = False
            for queue in queueData:
                if queue['slot'] == slot['id']:
                    if queue['state'] == 'RUNNING': isStillRunning = True
                    if queue['state'] == 'DOWNLOAD': queueDl = queue
            if not isStillRunning and queueDl and queueDl['waitingon'] == 'WS Assignment':
                match = re.match('\s?(\d+ days?)?\s?(\d+ hours?)?\s?(\d+ mins?)?\s?([\d.]+ secs?)?', queueDl['nextattempt'])
                if match:
                    seconds = 0
                    if match.group(1): seconds += int(re.sub('[^\d.]', '', match.group(1))) * 3600 * 24
                    if match.group(2): seconds += int(re.sub('[^\d.]', '', match.group(2))) * 3600
                    if match.group(3): seconds += int(re.sub('[^\d.]', '', match.group(3))) * 60
                    if match.group(4): seconds += round(float(re.sub('[^\d.]', '', match.group(4))) * 1)
                    if seconds >= restartLimit:
                        if not testMode:
                          sendCmd(tn, 'pause', queueDl['slot'])
                          time.sleep(1)
                          sendCmd(tn, 'unpause', queueDl['slot'])
                        restarted.append([queueDl['slot'], queueDl['nextattempt']])
                else: raise Exception(f'Error with {cmd}, parsing queue nextattempt:{queueDl["nextattempt"]}')
        return restarted
    elif par and (cmd == 'pause' or cmd == 'unpause'):
        cmdStr = f'{cmd} {par}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    else : return False

def check(host):
    st = time.time()
    pr(f'checking {host}', 1, True)
    try:
        tn = tnCreate(host)
        sendCmd(tn, 'auth')
        restarted = sendCmd(tn, 'get-info-and-restart')
        if len(restarted):
            pr(f'{host}: restarted {len(restarted)} slot{"s" if len(restarted) > 1 else ""}: ' + ', '.join(map(lambda item: '' + (' with '.join(item)), restarted)), 1, False, True)
        sendCmd(tn, 'exit')
        ed = time.time()
        time.sleep(max(0, 1 - (ed - st)))
    except Exception as err:
        pr(f'{host} error: {err}', 1, False, True)

checkKeep()

 

 

 

 

[grahamjrg]

3 minutes ago, LFreeman said:

Anyone seen this error before? It keeps happening.

 

FahCore returned: INTERRUPTED (102 = 0x66)

I haven't seen that error before. But a good place to ask questions is the Folding Community Board.
https://linustechtips.com/main/topic/902973-folding-community-board/

 

[GOTSpectrum]

3 minutes ago, LFreeman said:

Anyone seen this error before? It keeps happening.

 

FahCore returned: INTERRUPTED (102 = 0x66)

I can't be sure, but I think it got interrupted. 

 

8 minutes ago, LFreeman said:

@GOTSpectrum I should have thought to look at that

That you should have! Honestly it is really helping me deal with the lockdown 

[jctappel67]

4 minutes ago, LFreeman said:

Anyone seen this error before? It keeps happening.

 

FahCore returned: INTERRUPTED (102 = 0x66)

Is that from one of your VMs?

It happened to me pretty consistently before I deleted the previous VMs and made new ones with 7.5GB of RAM instead of 1.8

[LFreeman]

Just now, GOTSpectrum said:

I can't be sure, but I think it got interrupted. 

It has happened 9 times even when stopping and restarting the program

[LFreeman]

Just now, jctappel67 said:

Is that from one of your VMs?

It happened to me pretty consistently before I deleted the previous VMs and made new ones with 7.5GB of RAM instead of 1.8

It is from one of the VMs but it has 2core 7.5gb as I couldn't find the 1.8gb version

[ObeliskAG]

2 minutes ago, Bitter said:

1560 for folding at 1150mv, for gaming I run around 1590-1600 at 1125 since stability isn't as important. Due to how Vega clocks weird, when folding it bounces between 1550-1580 so I run the voltage a little higher than needed to ensure stability.

1,560 @ 180 = 62.12 GFLOPS/watt, 1,580 @ 180 = 62.92 GFLOPS/watt - not to shabby  I guess the Vega 56 has a similar number of shaders to the Vega 64, but uses a lot less power.

[shaz2sxy]

WOOP!! WOOP!! i'm back in the top 100 overall table for LTT

 

[jctappel67]

1 minute ago, LFreeman said:

It has happened 9 times even when stopping and restarting the program

Sometimes they're just unstable like that. Hasn't been too much of a problem for me in the past few days, but the first day and a half was a bit rough. Sucks that it's giving you crap though...

Just now, LFreeman said:

It is from one of the VMs but it has 2core 7.5gb as I couldn't find the 1.8gb version

Good, the 28800 point WUs would overload the RAM on that CPU choice

[Inkertus]

I miss all that much while I've been laid up in bed with a stomach bug...again?  Only thing I've gathered so far is points are missing.

Upside is my machines have been going 24/7 AFAIK since then.

[LFreeman]

5 minutes ago, jctappel67 said:

Sometimes they're just unstable like that. Hasn't been too much of a problem for me in the past few days, but the first day and a half was a bit rough. Sucks that it's giving you crap though...

Good, the 28800 point WUs would overload the RAM on that CPU choice

Looks like it was a bad WU. I deleted the slot, restarted, re-added it, the restarted again and it picked up a new one and is chugging along.

[GOTSpectrum]

13 minutes ago, LFreeman said:

It has happened 9 times even when stopping and restarting the program

Sorry should have put a '/s' 

[GOTSpectrum]

11 minutes ago, shaz2sxy said:

WOOP!! WOOP!! i'm back in the top 100 overall table for LTT

 

Welcome back onto the top! 

 

Now, next, top 50?

[Bitter]

15 minutes ago, ObeliskAG said:

1,560 @ 180 = 62.12 GFLOPS/watt, 1,580 @ 180 = 62.92 GFLOPS/watt - not to shabby  I guess the Vega 56 has a similar number of shaders to the Vega 64, but uses a lot less power.

Correct! It's the better deal of the Vega cards, especially since it undervolts+overclocks for much better efficiency than it had out of the box. V64/Radeon 7 are very power hungry by comparison but also can be undervolted+overclocked but to a lesser extent and remain power hungry cards and are still commanding high prices compared to 56's which are going for $200 and under. I paid a little more for a limited edition Nitro+ with the big cooler on it and the fancy RGB lighting, only live once.

[jctappel67]

Was setting up a new VM for when I have to shut the old ones down (because I'm a cheap bastard), and was wondering why my GPU wasn't getting any WUs.

 

I forgot to set GPU to true... 

 

 

EDIT: It's humming along now

Edited April 5, 2020 by jctappel67

[LFreeman]

1 minute ago, jctappel67 said:

Was setting up a new VM for when I have to shut the old ones down (because I'm a cheap bastard), and was wondering why my GPU wasn't getting any WUs.

 

I forgot to set GPU to true... 

 

Ooops

[ffbr]

5 minutes ago, Bitter said:

Correct! It's the better deal of the Vega cards, especially since it undervolts+overclocks for much better efficiency than it had out of the box. V64/Radeon 7 are very power hungry by comparison but also can be undervolted+overclocked but to a lesser extent and remain power hungry cards and are still commanding high prices compared to 56's which are going for $200 and under. I paid a little more for a limited edition Nitro+ with the big cooler on it and the fancy RGB lighting, only live once.

Absolutely love mine, even after realizing that for the cost of the card and the custom watercooling block and rgb I could have had something quite better from team green 

[ConceptOW]

I'm completing WUs but not getting any points today. Not sure what is going on.

[cbigfoot]

5 minutes ago, ConceptOW said:

I'm completing WUs but not getting any points today. Not sure what is going on.

did you check here: https://stats.foldingathome.org/donor/ ?

eoc has been hit and miss for some people.  i still think im missing some point from both servers...but hard to prove...

[GOTSpectrum]

Why is it right when you need to get something done, another things pops up!

 

I'm too tired for this BS lmfao. Update is coming guys but may be delayed!

[Favebook]

1 hour ago, LFreeman said:

Anyone seen this error before? It keeps happening.

 

FahCore returned: INTERRUPTED (102 = 0x66)

That is happening when you pause WU while it's folding. Nothing unusual:

Spoiler

23:41:32:WU00:FS01:0x22:Completed 680000 out of 1000000 steps (68%)
23:42:53:FS01:Paused
23:42:53:FS01:Shutting core down
23:42:53:WU00:FS01:0x22:WARNING:Console control signal 1 on PID 1716
23:42:53:WU00:FS01:0x22:Exiting, please wait. . .
23:42:54:WU00:FS01:0x22:Folding@home Core Shutdown: INTERRUPTED
23:42:54:WU00:FS01:FahCore returned: INTERRUPTED (102 = 0x66)
23:43:27:FS01:Unpaused
23:43:27:WU00:FS01:Starting

 

[LFreeman]

2 minutes ago, GOTSpectrum said:

Why is it right when you need to get something done, another things pops up!

 

I'm too tired for this BS lmfao. Update is coming guys but may be delayed!

Take your time brother

[Bitter]

https://stats.foldingathome.org/donor/83637

I'm a bit confused, I've got nothing folding for PCMR. I assume the older entries are someone else with the same username. Can there currently be someone else folding with the same username I'm using?