Backing Up the Petabyte Server

[Daniel644]

53 minutes ago, poochyena said:

Wouldn't this make their 3rd storage location since youtube counts as one, right?

they backup the RAW uncompressed 8k video before it's edited in anyway, so NO Youtube and it's shitty bitrates doesn't count.

[rdos14]

43 minutes ago, adelintru said:

You could technically also abuse Carbonite

https://www.carbonite.com/backup-software/carbonite-safe/

 

or Backblaze

https://www.backblaze.com/business-backup.html

Specially blackblaze... they dont care about limiting data. Its unlimited period.

[raf42]

No idea if it can scale up, but I've found Wiztree (https://antibody-software.com/web/software/software/wiztree-finds-the-files-and-folders-using-the-most-disk-space-on-your-hard-drive/) works just like WinDirStat, but WAYY faster, at least at a small, few TB scale.

[JCHelios]

Quote

 Since we're already operating in Hyper-Pedantic Mode: 

• "i.e." means id est, "that is."  
• "e.g." means  exemplī grātiā, "for the sake of example."

In this context, you should've corrected it to "e.g."  

Hard to get away with something like that on a channel filled with nerds.  ?

[Kuhlimuh]

Hey,

 

There is a possibility for Google Users to upload unlimited data daily with rclone (using Drive as a Teamdrive and multiple APIs). Here is the link with an instruction: https://plexguide.com/news/wiki/pg-blitz-instructions/

 

Is there a better option to contact LTT and tell them about it?

 

greets

[render]

congratulations, you're the hacker heroes of the decade!

[whistlerracer]

6 hours ago, nicklmg said:

 

Have no idea how big your room is, but fireproofing, concrete walls, etc... Unless the fire starts in that room ?  It is really rare nowadays that fire causes tones of damage on a concrete structures (from the picture I have seen of your office) in places like Burnaby response is fast, I would built a concrete structures, with many sprinkler, etc around it... Could be close to your $20.000 or less... Obviously in Van the first contractor will want $100.000 but if you keep it simple. Electricity and cooling during the fire may be the expensive part, unless you could put it to sleep remotely... just thinking outloud !

 

[Egg-Roll]

Umm... No one has mentioned the audio being out at 1:14? It actually looks like it started at 57 seconds in...

[Jadedwolf]

6 hours ago, adelintru said:

You could technically also abuse Carbonite

https://www.carbonite.com/backup-software/carbonite-safe/

 

or Backblaze

https://www.backblaze.com/business-backup.html

I feel like they would flag that hard and it would fail.

[Quad Damage]

Thanks for killing this for the rest of us.

 

6 hours ago, raf42 said:

No idea if it can scale up, but I've found Wiztree (https://antibody-software.com/web/software/software/wiztree-finds-the-files-and-folders-using-the-most-disk-space-on-your-hard-drive/) works just like WinDirStat, but WAYY faster, at least at a small, few TB scale.

I tested most of the popular applications recently, and TreeSize Free was the fastest by a significant margin when working on 50TB of data (500K files).

The only one faster was TreeSize Professional, which is about 50% faster than the free version - and that's about 10x faster than WinDirStat.

WizTree was disqualified from the test by requiring Admin permissions to run. That's never going to happen.

[NotKalo]

Well, hope Google don't ban you guys!

[dalekphalm]

8 hours ago, VegetableStu said:

my entire life I though e.g. was just a shortening of example o_o

It makes sense that it's Latin - a lot of expressions in English come from Latin.

 

Plus, there's no 'g' in "example", so e.g. being a simple shortening of example doesn't realllly work.

[Helpsave_ferris]

19 hours ago, Verond said:

You should use Wasabi. They don't charge for pull and are like 80% cheaper than AWS and Google Cloud.

 

We use them for video storage and streaming. 

I was going to suggest that too! But if you do the math, at $5 per TB, that would end up costing $5,000 for a petabyte of storage. So the jank Google drive strategy starts sounding more worth it.

[GodAtum]

I do the same, you should have asked me for advice.

 

 

[Razor512]

One issue with the google drive system is that google has a history of removing accounts that they do not like, even if they do not violate and specifically stated rules, thus there is a good chance that google will notice the data use and ban the account.

[AnonymousGuy]

I put 18TB on Google Drive Enterprise ($10/mo for unlimited....I'm grandfathered when it allowed unlimited for 1 user) 2 years ago, as an offsite backup of my 26 drive raid 60 array.

 

Proof:

 

 

It's not really a great solution.  Their whole infrastructure is not meant to be a backup to handle gigantic 300GB files.  Both downloading and interrupting uploads.  Their app too also doesn't like working with 300GB files.  Maybe it would be better if I had > 35Mbps internet upload speed, but it really felt janky for me and not reliable.

 

TLDR: Make sure you can actually  DOWNLOAD the data after you upload it.

[FartingBob]

On 23/08/2018 at 9:38 PM, poochyena said:

Why would they need all of the raw footage though? Tbh, I don't see why they need to backup reviews of 3+ year old hardware anyways.

Linus is a data hoarder. Sure they dont need most of the raw footage from their old videos, but when you can make money on doing videos about your backup plan, you might as well, we get to geek out over petabyte project and Linus gets to geek out over it as well. Although at this point it's probably more sensible to only backup the finished videos in their highest quality on a cloud storage (and raw footage from more recent videos), while saving the raw footage locally while space is available.

[BachChain]

So they shelled out however many hundreds of thousands of dollars on this project, but suddenly a proper off-site backup is just too expensive.

[dalekphalm]

1 hour ago, BachChain said:

So they shelled out however many hundreds of thousands of dollars on this project, but suddenly a proper off-site backup is just too expensive.

I believe that 45 Drives and Seagate sponsored the original servers. 

 

Certainly they could afford $100K, but if they can find a cheaper way *and* get some video content out of it too? Why not. 

 

They might still end up with another set of servers co-located at their ISP datacentre. 

[GDRRiley]

32 minutes ago, dalekphalm said:

I believe that 45 Drives and Seagate sponsored the original servers. 

 

Certainly they could afford $100K, but if they can find a cheaper way *and* get some video content out of it too? Why not. 

 

They might still end up with another set of servers co-located at their ISP datacentre. 

yeah and funny thing they have 2-3 empty S45s so all they would need to do is buy HDD.

I get that linus probably didn't want to shell out the extra cost to go from 10 to 12TB drives on the backup in addition to making the raid groups larger so that they could fit it all in 2 of the 45 bay units.

[dalekphalm]

1 minute ago, GDRRiley said:

yeah and funny thing they have 2-3 empty S45s so all they would need to do is buy HDD.

I get that linus probably didn't want to shell out the extra cost to go from 10 to 12TB drives on the backup in addition to making the raid groups larger so that they could fit it all in 2 of the 45 bay units.

Personally I’d go tape drive. Get a tape library, pop it into a data centre, and replicate to tapes on a rotating basis. 

 

Maybe also have a 45 drive server as a temp dumping ground, or just as an additional level of redundancy. 

[Pedro Lamanche]

Anyone familiar with the website featured in the video to calculate storage costs across all the vendors?

[AlexDeHacker]

use https://www.backblaze.com/backup-pricing.html unlimited storage for 5$ a month

[dalekphalm]

14 minutes ago, AlexDeHacker said:

use https://www.backblaze.com/backup-pricing.html unlimited storage for 5$ a month

You've selected their Personal plan. While it is good value, they almost certainly have commercial uses banned via the EULA.

 

There's also business - $50/mo per computer - but I assume this is for non-server usage only.

 

They do have a commercial Server/NAS pricing option though - it's still pretty expensive, but much cheaper over a year than I think Linus might have realized.

 

I threw in some random figures:

100TB initial upload

50TB monthly upload

50GB monthly download

 

And it came to just under $8000 USD per year. That's a lot, but as a yearly business expense, it's not that much.

 

Granted, the Vault has almost 400TB - so you'd need to multiply costing by 4, pretty much.

[risk]

My guess is the 750G/day limit is there to prevent ddos type attacks, where you'd create a bunch of accounts over time and sync your botnet to try to upload petabytes of random per second burning through the accounts as it goes through.

 

Backups of stuff sound like a legit use case, and you're still paying a hefty chunk, I don't know why anyone would be mad. On the contrary, it's actually a creative way to use the service, I wish you did a segment where you explain how folks should setup rclone, but maybe that's more for wendell