Cloud Computing During COVID

[Jemlin]

Hey all!

So, during COVID I've been helping my community by editing and distributing videos with people from our local community being in them.

I'm at my parents for lockdown and where they live the community tends to be slightly older (lots of at risk peoples as well as widowers), so its been nice to get them onto WhatsApp as well as using the local church to try and keep life as normal as possible for them.

Editing the videos has been great for me as it gives me something to do during lockdown (apparently aviation and travel restrictions don't go together too well... Who knew?!). However, with the community being as it is it means I've been having to do alot of resizing from SD (and every aspect ratio under the sun) to HD. AE has a great plugin for this which, while not being magically, really does make the footage look alot better. However, my render times are getting longer and longer. I had the idea of using cloud computing to set up a remote instance of Media Encoder which I could use to render out the videos. However, it seems that every single cloud computing company (Azure, AWS, Google) have no available space in the UK or Europe. I've tried using East Coast as well (ping times of 100... not bad at all) but have only been able to find space using super small instances (8 cores) that do a worse job than my current 4790K. Anyone got any recommendations? I've tried contacting support but I imagine they've got quite a large backlog and have been understandebly slow to respond.

Any advice you can give me would be much appreciated. I'm pretty sure I've figured out how to do it in terms of using VPNs to transfer files etc but I'm having trouble with actually carrying out my plan.

Thanks in advance. Hope you guys are keeping healthy (and sane) in these strange times.

[emosun]

well if you're in AE and using detail preserving upscaling or unsharp masking your way through sd footage then a cloud machine won't be any faster than your desktop at that. Only way you'll get any speed out of it is to gpu render it.

[igormp]

9 minutes ago, Jemlin said:

However, it seems that every single cloud computing company (Azure, AWS, Google) have no available space in the UK or Europe. I've tried using East Coast as well (ping times of 100... not bad at all) but have only been able to find space using super small instances (8 cores) that do a worse job than my current 4790K.

Hmmm, I'm not sure what happened to you, but I managed to deploy a N2 GCE instance with 16 cores and 16gb of RAM in London without problems.

[Jemlin]

1 hour ago, emosun said:

well if you're in AE and using detail preserving upscaling or unsharp masking your way through sd footage then a cloud machine won't be any faster than your desktop at that. Only way you'll get any speed out of it is to gpu render it.

Surely it would as the speed of it would scale with CPU horsepower, no?

[Jemlin]

1 hour ago, igormp said:

Hmmm, I'm not sure what happened to you, but I managed to deploy a N2 GCE instance with 16 cores and 16gb of RAM in London without problems.

Is that with AWS or Azure?

I've got my access to GPU pending while AWS do whatever they've got to do on their end to approve it

[igormp]

5 hours ago, Jemlin said:

Is that with AWS or Azure?

I've got my access to GPU pending while AWS do whatever they've got to do on their end to approve it

GCP

[emosun]

On 4/10/2020 at 3:07 PM, Jemlin said:

Surely it would as the speed of it would scale with CPU horsepower, no?

no

after effect and premeire are infamous in their ability to utilize a cpu incredibly poorly. a gpu renders far faster in both

[igormp]

51 minutes ago, emosun said:

no

after effect and premeire are infamous in their ability to utilize a cpu incredibly poorly. a gpu renders far faster in both

well, he can still attach a beefy gpu on the cloud.

[emosun]

1 minute ago, igormp said:

well, he can still attach a beefy gpu on the cloud.

I suppose. you'll have to calculate the upload times vs the time saved rendering to figure out if it's even worth using a cloud machine or not.

Just getting like 100gb raw file download might actually take longer than just rendering the project. trial and error i guess

[Jemlin]

So, just to feed back a bit. I ended up setting up an instance with AWS that gives me a T104 I think it is as well as 24 ish cores I think.

The render time I found to be around an hour and a half/2 hours faster apples to apples with my own rig.

However, it took an hour to upload all the files. So, time saved of around half an hour or so. But it meant my rig was able to be used during that time and also when I uploaded the project to youtube after it took a hell of alot less time to upload than compared with uploading from my rig at home.

 

Also, the part of the rendering that was slowing me down so much was that I was using after effect's detail preserving upscale to upscale some footage. I found that render doesn't use the GPU at all and just relies on the CPU. Hence me looking to cloud computing to try and throw some more cores at it (it really is time I upgrade my 4790k).

 

The one really interesting thing I've found is that on GCP, AWS, and Azure I had to get my account manually approved to get any kind of instance on a UK South server location. Goes to show how much people are using cloud computing during COVID. Azure I understand because Teams is taking a massive performance hit at the moment but I'm still waiting on a reply from GCP on trying to get server space in London and Azure have gone to their "resource allocation team". Want to try and do an apple for apples comparison between the three when (or if) I get approval, then will go from there to decide who I'll keep using.

 

I've also been having trouble getting my AWS server to see my home PC when using my home VPN. Anyone got any bright ideas for that? I've tried just about every fix I can find through google and am thinking of just trying to do it the other way round and connecting my home PC into an AWS VPC - or will I just run into the same problem on the other side?

[igormp]

2 hours ago, Jemlin said:

The one really interesting thing I've found is that on GCP, AWS, and Azure I had to get my account manually approved to get any kind of instance on a UK South server location. Goes to show how much people are using cloud computing during COVID. Azure I understand because Teams is taking a massive performance hit at the moment but I'm still waiting on a reply from GCP on trying to get server space in London and Azure have gone to their "resource allocation team". Want to try and do an apple for apples comparison between the three when (or if) I get approval, then will go from there to decide who I'll keep using.

Oh, I didn't know they were doing such things for new accounts. Sounds kinda weird since I have no problems at all creating 32+ cores machines on GCP in both my personal, university and work accounts.

 

Also, fwiw, GCP has some regions with Epyc processors available (not sure if London has them yet), maybe you could take a look at those.