LTT Folding Team's Emergency Response to Covid-19

[Kilrah]

It gives errors but works anyway.

[raulinbonn]

On 4/17/2020 at 4:10 AM, Koppa315 said:

I made a minor *cough* hacky *cough* change to @danielocdh's code to suite a use case I had. I started playing with connecting to one of my systems over local port forwarding SSH tunnel, partially because I didn't want to open firewall ports and mostly because FAH's API client is not secure. Either way in case anyone else had a same use case made it so the hosts variable can take a standard host:port notation. 

  Reveal hidden contents

################################################################################
##                                  options                                   ##
################################################################################
hosts = [ #list of quoted strings, hosts or IPs, with optional colon separted port, separated by comma
  'localhost',
  'localhost:36331'
]
hostsPassword = '' #quoted string, if the host(s) don't use a password just leave it as: ''

restartLimit = 10 * 60 #in seconds, pause+unpause if next attempt to get WU is this or more
checkEvery = 2 * 60 #in seconds, do a check for all hosts every this seconds

tConTimeout = 15 #in seconds, connection timeout
tReadTimeout = 10 #in seconds, read timeout
testMode = False # if set to True: checkEvery=6 and restartLimit=0 but won't actually pause+unpause slots

################################################################################
##                                    code                                    ##
################################################################################
import json
import re
import telnetlib
import time
import datetime

if testMode:
    restartLimit = 0
    checkEvery = 6
countEvery = 1 #seconds, have to be a factor of checkEvery, default: 1
countEveryDec = max(0, str(countEvery)[::-1].find('.'))
countEveryDecStr = f'{{:.{countEveryDec}f}}'
def remSeconds(seconds):
    if seconds > 0:
        if (seconds * 10000) % (countEvery * 10000) == 0:
            secondsP = countEveryDecStr.format(seconds)
            pr(f'Next check in {secondsP} seconds', same=True)
        time.sleep(countEvery)
        seconds = round((seconds - countEvery) * 10000) / 10000
        remSeconds(seconds)

prLastLen = 0
prLastSame = False
def pr(t, indent=0, same=False, overPrev=False):
    global prLastLen, prLastSame
    if not overPrev and not same and prLastSame:
        prLastLen = 0
        print('')
    t = str(t)
    toPrint = ('  ' * indent) + t
    tLen = len(toPrint)
    print(toPrint + (' ' * max(0, prLastLen - tLen)), end='\r')
    prLastSame = same
    prLastLen = tLen
    if not same:
        print('')
        prLastLen = 0

def checkKeep():
    while (True):
        checkAll()
        remSeconds(checkEvery)

def checkAll():
    for host in hosts: check(host)
    now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    pr(f'check complete at {now}', 0, False, True)


tEnd = ['\n>\s+'.encode('utf-8'), '\n---\n'.encode('utf-8')]
def readResult(expected, expectedResult=''):
    index = expected[0]
    readB = expected[2]
    
    read = readB.decode('utf-8')
    #noting
    if index < 0 or read == '': return [False, 'nothing was read']
    #expected result
    if expectedResult:
        endWith = tEnd[index].decode('utf-8')
        readStrip = read[0:-len(endWith)].strip()
        if (readStrip != expectedResult):
            return [False, f'{readB}']
    #PyON->json
    match = re.search('\n*PyON\s+(\d+)\s+([-_a-zA-Z\d]+)\n(.*)\n---\n', read, re.DOTALL)
    ###print('');print('');print('');print(index);print(match);print(read);print(readB);print('');
    if match:
        version = match.group(1)
        if version != '1': raise Exception('Response data version does not match')
        data = match.group(3)
        #to json
        data = re.sub('(:\s)?False', r'\1false', data)
        data = re.sub('(:\s)?True', r'\1true', data)
        data = re.sub('(:\s)?None', r'\1null', data)
        data = json.loads(data)
        return [True, data]
    #auth error
    match = re.search('\nERROR: unknown command or variable', read, re.DOTALL)
    if match:
        raise Exception('error sending command, wrong password?')
    #return read
    return [True, read]
def tnCreate(host):
    match = re.search('(.*):(\d+)', host);
    port = 36330;
    
    if match:
        host = match.group(1);
        port = match.group(2);
        
    tn = telnetlib.Telnet(host, port, tConTimeout)
    readResult(tn.expect(tEnd, tReadTimeout))
    return tn

def sendCmd(tn, cmd, par=''):
    if cmd == 'auth':
        if hostsPassword:
            cmdStr = f'auth {hostsPassword}';
            tn.write(f'{cmdStr}\n'.encode('utf-8'))
            res = readResult(tn.expect(tEnd, tReadTimeout), 'OK')
            if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
            return res[1]
        return True
    elif cmd == 'exit':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        try:
            readResult(tn.expect(tEnd, tReadTimeout))
        except EOFError as err:
            pass #Linux FAH Client without confirmation
        return;
    elif cmd == 'slot-info' or cmd == 'queue-info':
        cmdStr = f'{cmd}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    elif cmd == 'get-info-and-restart':
        queueData = sendCmd(tn, 'queue-info')
        slotData = sendCmd(tn, 'slot-info')
        ###
        #if type(queueData) == str: print('');print('');print('');print(queueData);print(queueData.encode('utf-8'));print('');
        #if type(slotData) == str: print('');print('');print('');print(slotData);print(slotData.encode('utf-8'));print('');
        restarted = []
        for slot in slotData:
            isStillRunning = False
            queueDl = False
            for queue in queueData:
                if queue['slot'] == slot['id']:
                    if queue['state'] == 'RUNNING': isStillRunning = True
                    if queue['state'] == 'DOWNLOAD': queueDl = queue
            if not isStillRunning and queueDl and queueDl['waitingon'] == 'WS Assignment':
                match = re.match('\s?(\d+ days?)?\s?(\d+ hours?)?\s?(\d+ mins?)?\s?([\d.]+ secs?)?', queueDl['nextattempt'])
                if match:
                    seconds = 0
                    if match.group(1): seconds += int(re.sub('[^\d.]', '', match.group(1))) * 3600 * 24
                    if match.group(2): seconds += int(re.sub('[^\d.]', '', match.group(2))) * 3600
                    if match.group(3): seconds += int(re.sub('[^\d.]', '', match.group(3))) * 60
                    if match.group(4): seconds += round(float(re.sub('[^\d.]', '', match.group(4))) * 1)
                    if seconds >= restartLimit:
                        if not testMode:
                          sendCmd(tn, 'pause', queueDl['slot'])
                          time.sleep(1)
                          sendCmd(tn, 'unpause', queueDl['slot'])
                        restarted.append([queueDl['slot'], queueDl['nextattempt']])
                else: raise Exception(f'Error with {cmd}, parsing queue nextattempt:{queueDl["nextattempt"]}')
        return restarted
    elif par and (cmd == 'pause' or cmd == 'unpause'):
        cmdStr = f'{cmd} {par}';
        tn.write(f'{cmdStr}\n'.encode('utf-8'))
        res = readResult(tn.expect(tEnd, tReadTimeout))
        if not res[0]: raise Exception(f'Error with {cmd}, {res[1]}')
        return res[1]
    else : return False

def check(host):
    st = time.time()
    pr(f'checking {host}', 1, True)
    try:
        tn = tnCreate(host)
        sendCmd(tn, 'auth')
        restarted = sendCmd(tn, 'get-info-and-restart')
        if len(restarted):
            pr(f'{host}: restarted {len(restarted)} slot{"s" if len(restarted) > 1 else ""}: ' + ', '.join(map(lambda item: '' + (' with '.join(item)), restarted)), 1, False, True)
        sendCmd(tn, 'exit')
        ed = time.time()
        time.sleep(max(0, 1 - (ed - st)))
    except Exception as err:
        pr(f'{host} error: {err}', 1, False, True)

checkKeep()

 

 

Definitely the client needs some work. Almost always whenever I see the GPU has not been folding for some time, I stop the client, restart it, and almost always consistently get a WU immediately. This should not be happening. The overall scheme should not make the clients seemingly get stuck with some server(s) which run out of WU's, while other servers may have WU's pending and no client is picking them up.

Will check and play with these scripts to see how much my average PPD might improve.

[raulinbonn]

I would think the overall scheme is a two-way multi-producer / multi-consumer scenario. So there could be one global distributed queue for WU's, and one global distributed queue of clients picking them up, then submitting their results to one global distributed queue of results for servers to pick up.

Whether it is implemented like that I really don't know, but the behavior of the clients seems to clearly not match that.

[Bitter]

I've got 6 finished WU's waiting to upload to 155.247.166.219 with an other CPU WU in progress that's going to try to upload to that same server.

[Alex Atkin UK]

7 hours ago, raulinbonn said:

I would think the overall scheme is a two-way multi-producer / multi-consumer scenario. So there could be one global distributed queue for WU's, and one global distributed queue of clients picking them up, then submitting their results to one global distributed queue of results for servers to pick up.

Whether it is implemented like that I really don't know, but the behavior of the clients seems to clearly not match that.

I think ultimately the issue is that F@H has never really been this big of a project.  Its exploded in popularity and clearly the original design of how it works just doesn't scale well.

 

Considering I had a few GPU WUs that for a moment I thought weren't going to submit before the timeout, its clear that the problem is not just with dishing out WUs but having the capacity to collect the results too.

[efka112]

31 minutes ago, Alex Atkin UK said:

I think ultimately the issue is that F@H has never really been this big of a project.  Its exploded in popularity and clearly the original design of how it works just doesn't scale well.

 

Considering I had a few GPU WUs that for a moment I thought weren't going to submit before the timeout, its clear that the problem is not just with dishing out WUs but having the capacity to collect the results too.

F@H said that main issue is that at the moment they din't had enough scientist to produce starting WU projections

[Koppa315]

20 hours ago, marknd59 said:

Don't update to the new version of F@H it broke the script when I tried it.

 

 

Interesting, I'll need to play around a bit. I have the latest FAH client running on my Windows machine and I'm not getting the Error with exit. I think I might just have a slightly newer version of that code (sharing code via forum is a bit tricky, I should see if @danielocdh wants to setup a git repo or just a gist with me, don't want to take credit as this was their baby originally and I was just tweaking for my purposes)

 

Here's a private gist for now: Folding @ Home - Babysitter can you try this out @marknd59

You may need to add the ip address for Ryzen-7 unless it's defined in DNS for you. Funny enough based on what I've seen that isn't actually an error that would interrupt the script's ability to monitor Folding @ Home, it's an error created when the script sends the exit command to the Control API. So it doesn't hang up nicely anymore... I have noticed that they also modified the slot pause details, which may be causing a temporary issue that I'm looking into

[Bitter]

Did they nerf the old clients already? FAH isn't connecting to advance or web control but it is running in the background.

 

Installed the latest version, still isn't working. It was working fine earlier today but since I restarted the computer...nothing. Working fine on both of my other PC's with the 7.5 version.

 

There it goes...still have 7 units for 155.247.166.219 that aren't sending though.

[Koppa315]

3 hours ago, Koppa315 said:

Interesting, I'll need to play around a bit. I have the latest FAH client running on my Windows machine and I'm not getting the Error with exit. I think I might just have a slightly newer version of that code (sharing code via forum is a bit tricky, I should see if @danielocdh wants to setup a git repo or just a gist with me, don't want to take credit as this was their baby originally and I was just tweaking for my purposes)

 

Here's a private gist for now: Folding @ Home - Babysitter can you try this out @marknd59

You may need to add the ip address for Ryzen-7 unless it's defined in DNS for you. Funny enough based on what I've seen that isn't actually an error that would interrupt the script's ability to monitor Folding @ Home, it's an error created when the script sends the exit command to the Control API. So it doesn't hang up nicely anymore... I have noticed that they also modified the slot pause details, which may be causing a temporary issue that I'm looking into

Updated the BabySitter script linked here under the Private Github Gist, should fix the issues with 7.6.9 and with Linux Control API...

Unfortunately Folding @ Home team doesn't version their Control API (they do but it's still at version 1) while changing the interface slightly, but their docs have gotten much nicer!

[bafo_ah]

3 hours ago, Koppa315 said:

Interesting, I'll need to play around a bit. I have the latest FAH client running on my Windows machine and I'm not getting the Error with exit. I think I might just have a slightly newer version of that code (sharing code via forum is a bit tricky, I should see if @danielocdh wants to setup a git repo or just a gist with me, don't want to take credit as this was their baby originally and I was just tweaking for my purposes)

 

Here's a private gist for now: Folding @ Home - Babysitter can you try this out @marknd59

You may need to add the ip address for Ryzen-7 unless it's defined in DNS for you. Funny enough based on what I've seen that isn't actually an error that would interrupt the script's ability to monitor Folding @ Home, it's an error created when the script sends the exit command to the Control API. So it doesn't hang up nicely anymore... I have noticed that they also modified the slot pause details, which may be causing a temporary issue that I'm looking into

It would be better if this "babysitter" can send us notification if something wrong happen, for now I just can think of telegram-bot, their API is super simple

Also note sometime FAHClient can stuck/hang and need to be restarted manually (you need listen to log-updates and specify the time limit)

 

I dont know if someone also writing this monitoring code, so I already start my own: https://gist.github.com/bafoah/16c026777216e197e2eb3eb89a4d4ec9

[Koppa315]

1 minute ago, bafo_ah said:

It would be better if this "babysitter" can send us notification if something wrong happen, for now I just can think of telegram-bot, their API is super simple

Also note sometime FAHClient can stuck/hang and need to be restarted manually (you need listen to log-updates and specify the time limit)

 

I dont know if someone also writing this monitoring code, so I already start my own: https://gist.github.com/bafoah/16c026777216e197e2eb3eb89a4d4ec9

Awesome! Yeah I wanted to play with the listening to updates as that would be more efficient. Funny enough I was thinking of writing my own as well but @danielocdh posted this already started, so I just modified it to support a use-case I had and have updated it as people were using it and saying that it wasn't working for 7.6.9

 

I still plan on playing with writing a monitoring tool and data visualizer for it in nodejs (prefer that to python personally). Though it seems that someone has already started building the grammar and the client so I might see if I can start contributing there and then building that visualizer 

[bafo_ah]

21 minutes ago, Koppa315 said:

Unfortunately Folding @ Home team doesn't version their Control API (they do but it's still at version 1) while changing the interface slightly, but their docs have gotten much nicer!

If you mean FAHClient version you can get it with get-info Build Version

Spoiler

 

[Koppa315]

1 minute ago, bafo_ah said:

If you mean FAHClient version you can get it with get-info Build Version

  Reveal hidden contents

 

The PyON API Version not the FAHClient version, they played around with some spacing and such that broke the codebase for the babysitter script because it uses telnetlib with expect regex which left it susceptible to minor changes causing some headaches.

 

Some of it may be alleviated by performing evals like you have in your codebase, but evals always make me uneasy especially when coming from an API/remote party, maybe with disabled global scopes and disabled local scopes? (I don't think the message will require either?) I've been reading through your code it's cool, I haven't run it yet, just scanning through what it does. Seems like yours is pretty far ahead in what it handles

[bafo_ah]

26 minutes ago, Koppa315 said:

The PyON API Version not the FAHClient version, they played around with some spacing and such that broke the codebase for the babysitter script because it uses telnetlib with expect regex which left it susceptible to minor changes causing some headaches.

 

Some of it may be alleviated by performing evals like you have in your codebase, but evals always make me uneasy especially when coming from an API/remote party, maybe with disabled global scopes and disabled local scopes? (I don't think the message will require either?) I've been reading through your code it's cool, I haven't run it yet, just scanning through what it does. Seems like yours is pretty far ahead in what it handles

ah, I see, that's why the response is start with PyON 1 units/ppd/slots/etc with 1 indicate PyON version, 1 mystery solved!

Spoiler

I re-check https://github.com/FoldingAtHome/fah-control/wiki/3rd-party-FAHClient-API

and my answer is there.... Silly me... I need to learn to READ!

 

adding "safe-eval" function to my to-do list

[Bitter]

3 hours ago, Bitter said:

Did they nerf the old clients already? FAH isn't connecting to advance or web control but it is running in the background.

 

Installed the latest version, still isn't working. It was working fine earlier today but since I restarted the computer...nothing. Working fine on both of my other PC's with the 7.5 version.

 

There it goes...still have 7 units for 155.247.166.219 that aren't sending though.

8 work units that won't upload to 155.247.166.219 now. How do I get them to upload or just clear?

[bafo_ah]

3 minutes ago, Bitter said:

8 work units that won't upload to 155.247.166.219 now. How do I get them to upload or just clear?

How many folding slot do you have? I NEVER have so many wu that wont uploaded...

It is from the same slot? or from multi-slot?

 

is your slot still folding? or Trying to download new wu?

 

Watch the log, if log is appending, that's normal, if log stuck maybe there is something wrong with your client

 

You can try to restart FAHClient/reboot machine....

Or if you want to force it to clear, you can delete folding slot, and then re-add it back (but please don't do it, it will dump your already finished wu)

Reinstall client will do the same

 

FYI, FAHClient 7.6.9 wont detect my GPU, so I still stuck with 7.5.1

 

[marknd59]

6 hours ago, Koppa315 said:

Interesting, I'll need to play around a bit. I have the latest FAH client running on my Windows machine and I'm not getting the Error with exit. I think I might just have a slightly newer version of that code (sharing code via forum is a bit tricky, I should see if @danielocdh wants to setup a git repo or just a gist with me, don't want to take credit as this was their baby originally and I was just tweaking for my purposes)

 

Here's a private gist for now: Folding @ Home - Babysitter can you try this out @marknd59

You may need to add the ip address for Ryzen-7 unless it's defined in DNS for you. Funny enough based on what I've seen that isn't actually an error that would interrupt the script's ability to monitor Folding @ Home, it's an error created when the script sends the exit command to the Control API. So it doesn't hang up nicely anymore... I have noticed that they also modified the slot pause details, which may be causing a temporary issue that I'm looking into

 

OK I'll give it a try in a little bit. Thanks for the help. This is how I have the hosts setup.

 

Spoiler

hosts = [ #list of quoted strings, hosts or IPs, with optional colon separted port, separated by comma
  'localhost',
  '127.0.0.1:36330',
  'Ryzen-7',
  '192.168.1.50:36330'
]

 

 

[Bitter]

1 hour ago, bafo_ah said:

How many folding slot do you have? I NEVER have so many wu that wont uploaded...

It is from the same slot? or from multi-slot?

 

is your slot still folding? or Trying to download new wu?

 

Watch the log, if log is appending, that's normal, if log stuck maybe there is something wrong with your client

 

You can try to restart FAHClient/reboot machine....

Or if you want to force it to clear, you can delete folding slot, and then re-add it back (but please don't do it, it will dump your already finished wu)

Reinstall client will do the same

 

FYI, FAHClient 7.6.9 wont detect my GPU, so I still stuck with 7.5.1

 

I have 3 slots, 2 4 core CPU slots and 1 GPU slot. 4 core work comes more regularly than I got single 8 core work. Often I have one 4 core work unit running while the other is waiting for work. All units are CPU units and all are trying to upload to the one server, logs show it's not able to upload to the server but I'm not sure exactly why.

https://apps.foldingathome.org/serverstats

Looks like it might be having some issues, 6 hours uptime and very little disk space left.

[bafo_ah]

2 minutes ago, Bitter said:

I have 3 slots, 2 4 core CPU slots and 1 GPU slot. 4 core work comes more regularly than I got single 8 core work. Often I have one 4 core work unit running while the other is waiting for work. All units are CPU units and all are trying to upload to the one server, logs show it's not able to upload to the server but I'm not sure exactly why.

https://apps.foldingathome.org/serverstats

Looks like it might be having some issues, 6 hours uptime and very little disk space left.

Looks like it server fault, maybe its better to let it be...

Or.. just in case, I dont know if it help or something... Try using VPN...

[marknd59]

OK I just test the new code on F@H 7.6.8 and it worked like a charm with no errors. I also released I am a idiot and had the host setup was wrong, it's what happens when you mess around with things you don't really understand. My host setup should look like this -

 

Spoiler

hosts = [ #list of quoted strings, hosts or IPs, with optional colon separted port, separated by comma
  'localhost',
  '192.168.1.50:36330'
]

 

[danielocdh]

8 hours ago, Koppa315 said:

Interesting, I'll need to play around a bit. I have the latest FAH client running on my Windows machine and I'm not getting the Error with exit. I think I might just have a slightly newer version of that code (sharing code via forum is a bit tricky, I should see if @danielocdh wants to setup a git repo or just a gist with me, don't want to take credit as this was their baby originally and I was just tweaking for my purposes)

 

Here's a private gist for now: Folding @ Home - Babysitter can you try this out @marknd59

You may need to add the ip address for Ryzen-7 unless it's defined in DNS for you. Funny enough based on what I've seen that isn't actually an error that would interrupt the script's ability to monitor Folding @ Home, it's an error created when the script sends the exit command to the Control API. So it doesn't hang up nicely anymore... I have noticed that they also modified the slot pause details, which may be causing a temporary issue that I'm looking into

nice job, added some notes on github

[marknd59]

@danielocdh and @Koppa315 I would like to thank both of you for the work you have done on this script, it makes things so much easier.

[Alex Atkin UK]

Finally ran out of Google Cloud credit and I must say their billing is garbage.

 

When I stopped the VM before going to wash for bed, it said I had £1.03 of credit left.

When I finally went to bed it had gone up to £1.13.

When I got up this morning it was £0.00 with £0.44 charges.

The billing report says this includes a Discount of -£4.27 so I had gone over by £4.71?

 

This was all AFTER the VM had already been stopped.

 

I mean I'm extremely happy I managed to judge it to get such a tiny overage charge, but wtf is their billing system doing to behave like this?  Its utterly hopeless for judging your costs.

[marknd59]

8 hours ago, Bitter said:

There it goes...still have 7 units for 155.247.166.219 that aren't sending though.

My 3 that were stuck for over a day have all now uploaded, I didn't do anything they just went off on there own at some point.

[Koppa315]

9 hours ago, Bitter said:

8 work units that won't upload to 155.247.166.219 now. How do I get them to upload or just clear?

 

8 hours ago, bafo_ah said:

How many folding slot do you have? I NEVER have so many wu that wont uploaded...

It is from the same slot? or from multi-slot?

 

is your slot still folding? or Trying to download new wu?

 

Watch the log, if log is appending, that's normal, if log stuck maybe there is something wrong with your client

 

You can try to restart FAHClient/reboot machine....

Or if you want to force it to clear, you can delete folding slot, and then re-add it back (but please don't do it, it will dump your already finished wu)

Reinstall client will do the same

 

FYI, FAHClient 7.6.9 wont detect my GPU, so I still stuck with 7.5.1

 

I saw on the GitHub issue tracker for FAHClient a few people were having that issue, have you posted you card and the version to the tracker? I think they've cleared up a few cards so far.

7 hours ago, marknd59 said:

OK I just test the new code on F@H 7.6.8 and it worked like a charm with no errors. I also released I am a idiot and had the host setup was wrong, it's what happens when you mess around with things you don't really understand. My host setup should look like this -

 

  Reveal hidden contents

hosts = [ #list of quoted strings, hosts or IPs, with optional colon separted port, separated by comma
  'localhost',
  '192.168.1.50:36330'
]

 

Awesome to hear that it's working @marknd59! Meh, experimenting is how you learn right? Trust me I had to do a lot of that for updating @danielocdh's code, I haven't touched python in 9+ years 

6 hours ago, danielocdh said:

nice job, added some notes on github

Awesome thank you @danielocdh, I'll go review!

4 hours ago, marknd59 said:

@danielocdh and @Koppa315 I would like to thank both of you for the work you have done on this script, it makes things so much easier.

No worries, glad the updates were useful  

4 hours ago, Alex Atkin UK said:

Finally ran out of Google Cloud credit and I must say their billing is garbage.

 

When I stopped the VM before going to wash for bed, it said I had £1.03 of credit left.

When I finally went to bed it had gone up to £1.13.

When I got up this morning it was £0.00 with £0.44 charges.

The billing report says this includes a Discount of -£4.27 so I had gone over by £4.71?

 

This was all AFTER the VM had already been stopped.

 

I mean I'm extremely happy I managed to judge it to get such a tiny overage charge, but wtf is their billing system doing to behave like this?  Its utterly hopeless for judging your costs.

I've heard that a bunch with Google, you almost need to preemptively stop much earlier since it's constantly caching up on the billing page. Granted you did an amazing job only getting £0.44 over. I haven't played with my Google credits yet, was contemplating setting up a Terraform instance just to learn the tool and play with some credits across my Google accounts and MSDN Azure Account.

 

For Azure MSDN Credit you get to use all of your credit without adding billing information, including GPU systems, but if you use all your credit before the month is up the instances disappear. So if you kill it with enough credit for the storage costs to continue for the remainder of the month you're good and can start all over again the next month, just in case having a Terraform job ready to spin back up a node the next month might be nice so I don't have to babysit the cost analyser (or script it... hahaha).