Our data is GONE... Again

[jakkuh_t]

 

[Kn0wn]

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

[Lightwreather]

Who let Linus near the data again?

/s

[williamcll]

17 minutes ago, Kn0wn said:

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

Pretty sure they do. Wonder if it's around. 

[Nystemy]

To be fair, just rent a "cheap" office somewhere off in the distance, and put in a tape library.

Tape is really cheap per GB and when access speed isn't critical, it sure is useful.

 

But yes, doing regular checks for bit rotting is a rather important thing. Even a once every couple of months would have likely been a saving grace to be fair.

Though, the hard drive failures should have been noticed the very hour it happened (unless everyone responsible were asleep), such alarms shouldn't be easy to miss. This is something I would look into more deeply if I were in that position.

 

Hopefully LMG don't store any mission critical data in this frankly poor fashion.

In the end though, it is good to showcase that mistakes does happen, and sometimes it is due to simple things.

And yes, completely missing critical settings is a major downside of rolling an internal solution, for anything.

[manikyath]

not sure who at LTT to poke for this.. but have you guys considered a rackmount tape library and just writing your raw data to tape, then storing that in a vault? (vault mostly because EMI and temperature insulation being helpful for tape health)

 

i cant quote exact figures because it's too long ago and actually sort of sensitive information, but a 24-bay robotic library (because you dont want to swap tapes on a daily basis...) with some matching backup software on the storage server side isnt all THAT expensive in the grand scheme of things, and once it's set up you basicly have a (fairly) flat cost per TB stored to grow the archive.

 

it's horrid to restore from, yes, but it's FAR cheaper than cloud storage, more energy efficient than disks, and past just the archival job of swapping out the tapes in the library it's basicly set and forget.

 

in case anyone misinterprets and starts to ramble about the annoyance of single tape LTO drives (like in an LTT video a while back), i'm talking about a robotic library, those can hold a number of tapes (they have little 8-tape ones, and go up into systems like HP's LTO-8 system that scales to some hundreds of tapes in a rack) which can be accessed all independantly by a number of readers (usually one or two, i think the biggest ones from HP go up to 16 or something in that range.)

 

i've dealt with a customer that probably has more data in archive than LTT has, it's all on LTO, and it's surprisingly manageable.

[sub68]

7 minutes ago, Nystemy said:

Hopefully LMG don't store any mission critical data in this frankly poor fashion.

So from what I remember is Vaults are all the old back ups 

Whonnock is the active editing/templates server.

[The Oddball]

@jakkuh_t
Could you send me a DM? I'd like to start an eMail chain about this, I have a possible solution to your lack of backup, much cheaper than cloud storage, and could get quite a good video out of it.

[Roadsguy]

What happened to that tape drive you guys got?

Do you not still have that, or do you only use it for the other servers and not the Vaults? Sure, 2PB at 12TB per tape is a lot of tapes, but it wouldn't have been nearly as expensive as a backup server or cloud storage as discussed in the video.

[Nystemy]

14 minutes ago, sub68 said:

So from what I remember is Vaults are all the old back ups 

Whonnock is the active editing/templates server.

That should be the case to what I remembers as well.
Though, hope that they don't store everything on site. A fire or flood cloud set them back a lot. (If loosing the office and most stuff inside it weren't a huge setback already. But at least they are spreading out into neighboring buildings a bit now with the lab.)

[sub68]

5 minutes ago, Nystemy said:

Though, hope that they don't store everything on site.

I think there templates and stuff like that are backacked up off site.

but no footage

[Nystemy]

1 minute ago, Roadsguy said:

What happened to that tape drive you guys got?

Do you not still have that, or do you only use it for the other servers and not the Vaults? Sure, 2PB at 12TB per tape is a lot of tapes, but it wouldn't have been nearly as expensive as a backup server or cloud storage as discussed in the video.

Yes, tape backups is a perfect way to achieve data for "hope we never need to read these." applications, mainly due to its exceptionally low price per TB.

For online storage tapes is however awful, but the storage servers should still be there for that job since the tape would just be backup.

 

Preferably a tape backup would also be stored off site (the drive can however be at the office). For an example taking the tape home at the end of the week. On a weekly backup schedule, even for a large operation a 12TB tape is a decent chunk of data and might not even get full from just one week of content. _{(leaving room for a second "trailing" backup of what is already backed up in case a tape fails.)} One can always just write more than one tape per week as well.

[Zodiark1593]

"Oh look, everything is on fire. I wonder how that happened?"

 

LTT can certainly use a dedicated IT person. Having someone able to fix and maintain things during the off-hours would be quite valuable itself. IT can also better set up processes to keep track of the systems in-use, as well as what changes are made over their lifetime, working with procurement, and software/subscription license allocations. I'm not sure how Adobe enforces licensing, but Autodesk appears to be fairly active in pursuing audits. However, given the work environment, you'd probably want someone familiar with production or stage work as well, so as to acclimate easier and better assess the needs of the staff.

 

Regarding storage itself, I feel the reduced running and maintenance costs/requirements of tape would probably better suit the use case here. While not a "glamorous" solution in any sense of the word, risks of hardware failure killing your data is greatly reduced (especially given frequent power outages), and off-site backups are fairly straightforward. The downside of access times can be mitigated by planning in-advance what video clips are needed, and doing other work while waiting for the clips to be fetched. (certainly more doable with a robotic library, and can be milked for videos).

 

While the Vault can be run alongside tape storage, given LTT's ever expanding data requirements, I'm unsure if continually scaling up the Vault is an effective long-term solution.

 

[Bigun]

This just in, 2nd law is 2nd law

Also, the sun came up today.

 

Please guys hire someone to fix this

[falki]

Linus, as "CTIO" this is clearly on you.....

In an Enterprise, 10k$ a Month for 1Pbyte is Peanuts....
Perhaps you should put the old Storage in your new Pink rack at home for an offsite backup

[Muhlis]

Linus said about redownloading the videos from the channel, you can use Google Takeout which gives you the chance to download all your videos at their original state. No degradation in the quality. Tried this a few times on my other channel and it succeed. Mine was 250+ GB in total for 160 videos.

 

[Nystemy]

8 minutes ago, Zodiark1593 said:

"Oh look, everything is on fire. I wonder how that happened?"

 

LTT can certainly use a dedicated IT person. Having someone able to fix and maintain things during the off-hours would be quite valuable itself. IT can also better set up processes to keep track of the systems in-use, as well as what changes are made over their lifetime, working with procurement, and software/subscription license allocations. I'm not sure how Adobe enforces licensing, but Autodesk appears to be fairly active in pursuing audits. However, given the work environment, you'd probably want someone familiar with production or stage work as well, so as to acclimate easier and better assess the needs of the staff.

 

Regarding storage itself, I feel the reduced running and maintenance costs/requirements of tape would probably better suit the use case here. While not a "glamorous" solution in any sense of the word, risks of hardware failure killing your data is greatly reduced (especially given frequent power outages), and off-site backups are fairly straightforward. The downside of access times can be mitigated by planning in-advance what video clips are needed, and doing other work while waiting for the clips to be fetched. (certainly more doable with a robotic library, and can be milked for videos).

 

While the Vault can be run alongside tape storage, given LTT's ever expanding data requirements, I'm unsure if continually scaling up the Vault is an effective long-term solution.

 

Personally I don't see much issue with having their current HDD based storage, it brings in a lot more advantages than just fetching old videos, but also allows them to overflow from their main production server in case they push the occasional project onto the backburner for a couple of months and that has seemingly happened more than once before.

 

And waiting for a tape library is painfully slow, and muxing different projects for one editor just because Linus or someone else makes a callback to something they did 2 years ago is going to have its own set of organizational downsides, ie longer production/editing times, more room for errors/mixups, and a need for a couple of more projects in the pipe at a given moment. So there is value to having an active storage solution like they have.

But having a tape backup off site would be a wise move. Since it is easy to transport, exceptionally cheap, and the drive itself can stay at the office saving the need for pushing tons of data over the internet to somewhere else.

[Bigun]

Also, on the math, a 1-time investment into LTO-8 or 9 would have given you guys archival capabilities "fairly cheap" (rough figures in my head are anywhere between $60K to $70K USD initial capital and about $8K USD annual operating), with the option of removing some of the older videos from immediate access, reducing the need for so much TIER-1 & TIER-2 storage.  Of course I'd need more info on growth and such to get more accurate numbers.  Please let me work up a proposal - you guys could use it.  My treat.

[Ultraforce]

1 hour ago, Kn0wn said:

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

They have an offsite backup though offsite backup is solely for wonnick which is the mission critical server of ongoing projects.

9 minutes ago, Muhlis said:

Linus said about redownloading the videos from the channel, you can use Google Takeout which gives you the chance to download all your videos at their original state. No degradation in the quality. Tried this a few times on my other channel and it succeed. Mine was 250+ GB in total for 160 videos.

 

They don't upload their videos to youtube at the original film quality which is part of the whole reason they keep a back up of everything as youtube doesn't support 8k

[Zodiark1593]

6 minutes ago, Nystemy said:

Personally I don't see much issue with having their current HDD based storage, it brings in a lot more advantages than just fetching old videos, but also allows them to overflow from their main production server in case they push the occasional project onto the backburner for a couple of months and that has seemingly happened more than once before.

 

And waiting for a tape library is painfully slow, and muxing different projects for one editor just because Linus or someone else makes a callback to something they did 2 years ago is going to have its own set of organizational downsides, ie longer production/editing times, more room for errors/mixups, and a need for a couple of more projects in the pipe at a given moment. So there is value to having an active storage solution like they have.

But having a tape backup off site would be a wise move. Since it is easy to transport, exceptionally cheap, and the drive itself can stay at the office saving the need for pushing tons of data over the internet to somewhere else.

I was thinking more along the lines of "Ok, we requested our archive clip for Project A, let's work on getting everything else together in Project A (with the working footage from Whonnuck) while waiting for the old clip to arrive." Though admittedly, I'm not familiar with the access times of Tape. I figured worst-case scenario, maybe an hour to retrieve a given clip.

 

Though I definitely see your point, especially with multiple editors working and potentially needing accesses themselves.

[Red :)]

3 minutes ago, Ultraforce said:

youtube doesn't support 8k

[Bigun]

19 minutes ago, Nystemy said:

And waiting for a tape library is painfully slow....

This can be partially mitigated depending on the tape library being used, number of tape decks, and how the backups were originally setup.

 

With 5 libraries @ 3 decks a piece, monthly full backups of 2PB would take 5 days - rough math.

[Nystemy]

19 minutes ago, Bigun said:

Also, on the math, a 1-time investment into LTO-8 or 9 would have given you guys archival capabilities "fairly cheap" (rough figures in my head are anywhere between $60K to $70K USD initial capital and about $8K USD annual operating), with the option of removing some of the older videos from immediate access, reducing the need for so much TIER-1 & TIER-2 storage.  Of course I'd need more info on growth and such to get more accurate numbers.  Please let me work up a proposal - you guys could use it.  My treat.

Googling a bit, an LTO 8 drive costs about 5 grand.
And the LTO 8 tape is about 25$/TB, and this is what I can just straight up buy now...

So where did you get those exorbitant prices from?
An IKEA bookshelf in a closet at home isn't particularly expensive to own per year, and could store many tens of PB with ease. (Yes it is a lot less fancy than a robotic tape library, but for a pure backup it isn't needed.)

For online storage of the video, 1-3 PB isn't that silly to run. And HDD's are getting cheaper per TB as well, so those old videos are more or less not costing much over time.
Secondly, it gives LMG a made up reason to go for more absurd data storage solutions, and even if it technically isn't needed, it is still interesting content.

 

6 minutes ago, Bigun said:

This can be partially mitigated depending on the tape library being used, number of tape decks, and how the backups were originally setup.

For a backup, fancy "low latency" tape solutions isn't really cost effective. This is for a medium sized media production company, not CERN....

[Bigun]

@Nystemy

 

The tape libraries to run them.  I'm basing this off of using Scalar i3 libraries.  They hold about 50 tapes per library and 3 tape decks a piece.

 

*edit*

 

Oooh, also forgot the repository needs to be worked in.  I really need more info to get something realistic worked up.

[Ultraforce]

1 minute ago, Red said:

Okay guess I was wrong and looking apparently older videos up to like some from August 2015 can be viewed in 4k.