Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Our data is GONE... Again

jakkuh_t
 Share

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

Link to comment
Share on other sites

Link to post
Share on other sites

Who let Linus near the data again?

/s

If you think I'm wrong, correct me. If I've offended you in some way tell me what it is and how I can correct it. I want to learn, and along the way one can make mistakes; Being wrong helps you learn what's right.

Link to comment
Share on other sites

Link to post
Share on other sites

17 minutes ago, Kn0wn said:

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

Pretty sure they do. Wonder if it's around. 

Specs: Motherboard: Asus X470-PLUS TUF gaming (Yes I know it's poor but I wasn't informed) RAM: Corsair VENGEANCE® LPX DDR4 3200Mhz CL16-18-18-36 2x8GB

            CPU: Ryzen 9 5900X          Case: Antec P8     PSU: Corsair RM850x                        Cooler: Antec K240 with two Noctura Industrial PPC 3000 PWM

            Drives: Samsung 970 EVO plus 250GB, Micron 1100 2TB, Seagate ST4000DM000/1F2168 GPU: EVGA RTX 2080 ti Black edition

Link to comment
Share on other sites

Link to post
Share on other sites

To be fair, just rent a "cheap" office somewhere off in the distance, and put in a tape library.

Tape is really cheap per GB and when access speed isn't critical, it sure is useful.

 

But yes, doing regular checks for bit rotting is a rather important thing. Even a once every couple of months would have likely been a saving grace to be fair.

Though, the hard drive failures should have been noticed the very hour it happened (unless everyone responsible were asleep), such alarms shouldn't be easy to miss. This is something I would look into more deeply if I were in that position.

 

Hopefully LMG don't store any mission critical data in this frankly poor fashion.

In the end though, it is good to showcase that mistakes does happen, and sometimes it is due to simple things.

And yes, completely missing critical settings is a major downside of rolling an internal solution, for anything.

Link to comment
Share on other sites

Link to post
Share on other sites

not sure who at LTT to poke for this.. but have you guys considered a rackmount tape library and just writing your raw data to tape, then storing that in a vault? (vault mostly because EMI and temperature insulation being helpful for tape health)

 

i cant quote exact figures because it's too long ago and actually sort of sensitive information, but a 24-bay robotic library (because you dont want to swap tapes on a daily basis...) with some matching backup software on the storage server side isnt all THAT expensive in the grand scheme of things, and once it's set up you basicly have a (fairly) flat cost per TB stored to grow the archive.

 

it's horrid to restore from, yes, but it's FAR cheaper than cloud storage, more energy efficient than disks, and past just the archival job of swapping out the tapes in the library it's basicly set and forget.

 

in case anyone misinterprets and starts to ramble about the annoyance of single tape LTO drives (like in an LTT video a while back), i'm talking about a robotic library, those can hold a number of tapes (they have little 8-tape ones, and go up into systems like HP's LTO-8 system that scales to some hundreds of tapes in a rack) which can be accessed all independantly by a number of readers (usually one or two, i think the biggest ones from HP go up to 16 or something in that range.)

 

i've dealt with a customer that probably has more data in archive than LTT has, it's all on LTO, and it's surprisingly manageable.

Link to comment
Share on other sites

Link to post
Share on other sites

7 minutes ago, Nystemy said:

Hopefully LMG don't store any mission critical data in this frankly poor fashion.

So from what I remember is Vaults are all the old back ups 

Whonnock is the active editing/templates server.

Everyone, Creator初音ミク Hatsune Miku Google commercial.

 

 

Cameras: Main: Canon 70D - Secondary: Panasonic GX85 - Spare: Samsung ST68. - Action cams: GoPro Hero+, Akaso EK7000pro

Dead cameras: Nikion s4000, Canon XTi

 

Pc's

Spoiler

Dell prebuilt (main) - i5-6500- 20GB ram - 500gb HDD- dvd 

 

HP compaq 8300 prebuilt - Intel i5-3470 - 8GB ram - 500GB HDD - bluray drive

 

old windows 7 gaming desktop - Intel i5 2400 - lenovo CIH61M V:1.0 - 4GB ram - 1TB HDD - dual DVD roms 

 

main laptop acer e5 15 - Intel i3 7th gen - 16GB ram - 1TB HDD - dvd drive                                                                     

 

school laptop lenovo 300e chromebook 2nd gen - Intel celeron - 4GB ram - 32GB SSD 

#Muricaparrotgang                                                                                   

 

Link to comment
Share on other sites

Link to post
Share on other sites

@jakkuh_t
Could you send me a DM? I'd like to start an eMail chain about this, I have a possible solution to your lack of backup, much cheaper than cloud storage, and could get quite a good video out of it.

Link to comment
Share on other sites

Link to post
Share on other sites

What happened to that tape drive you guys got?

Do you not still have that, or do you only use it for the other servers and not the Vaults? Sure, 2PB at 12TB per tape is a lot of tapes, but it wouldn't have been nearly as expensive as a backup server or cloud storage as discussed in the video.

Link to comment
Share on other sites

Link to post
Share on other sites

14 minutes ago, sub68 said:

So from what I remember is Vaults are all the old back ups 

Whonnock is the active editing/templates server.

That should be the case to what I remembers as well.
Though, hope that they don't store everything on site. A fire or flood cloud set them back a lot. (If loosing the office and most stuff inside it weren't a huge setback already. But at least they are spreading out into neighboring buildings a bit now with the lab.)

Link to comment
Share on other sites

Link to post
Share on other sites

5 minutes ago, Nystemy said:

Though, hope that they don't store everything on site.

I think there templates and stuff like that are backacked up off site.

but no footage

Everyone, Creator初音ミク Hatsune Miku Google commercial.

 

 

Cameras: Main: Canon 70D - Secondary: Panasonic GX85 - Spare: Samsung ST68. - Action cams: GoPro Hero+, Akaso EK7000pro

Dead cameras: Nikion s4000, Canon XTi

 

Pc's

Spoiler

Dell prebuilt (main) - i5-6500- 20GB ram - 500gb HDD- dvd 

 

HP compaq 8300 prebuilt - Intel i5-3470 - 8GB ram - 500GB HDD - bluray drive

 

old windows 7 gaming desktop - Intel i5 2400 - lenovo CIH61M V:1.0 - 4GB ram - 1TB HDD - dual DVD roms 

 

main laptop acer e5 15 - Intel i3 7th gen - 16GB ram - 1TB HDD - dvd drive                                                                     

 

school laptop lenovo 300e chromebook 2nd gen - Intel celeron - 4GB ram - 32GB SSD 

#Muricaparrotgang                                                                                   

 

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Roadsguy said:

What happened to that tape drive you guys got?

Do you not still have that, or do you only use it for the other servers and not the Vaults? Sure, 2PB at 12TB per tape is a lot of tapes, but it wouldn't have been nearly as expensive as a backup server or cloud storage as discussed in the video.

Yes, tape backups is a perfect way to achieve data for "hope we never need to read these." applications, mainly due to its exceptionally low price per TB.

For online storage tapes is however awful, but the storage servers should still be there for that job since the tape would just be backup.

 

Preferably a tape backup would also be stored off site (the drive can however be at the office). For an example taking the tape home at the end of the week. On a weekly backup schedule, even for a large operation a 12TB tape is a decent chunk of data and might not even get full from just one week of content. (leaving room for a second "trailing" backup of what is already backed up in case a tape fails.) One can always just write more than one tape per week as well.

Link to comment
Share on other sites

Link to post
Share on other sites

"Oh look, everything is on fire. I wonder how that happened?" 😄

 

LTT can certainly use a dedicated IT person. Having someone able to fix and maintain things during the off-hours would be quite valuable itself. IT can also better set up processes to keep track of the systems in-use, as well as what changes are made over their lifetime, working with procurement, and software/subscription license allocations. I'm not sure how Adobe enforces licensing, but Autodesk appears to be fairly active in pursuing audits. However, given the work environment, you'd probably want someone familiar with production or stage work as well, so as to acclimate easier and better assess the needs of the staff.

 

Regarding storage itself, I feel the reduced running and maintenance costs/requirements of tape would probably better suit the use case here. While not a "glamorous" solution in any sense of the word, risks of hardware failure killing your data is greatly reduced (especially given frequent power outages), and off-site backups are fairly straightforward. The downside of access times can be mitigated by planning in-advance what video clips are needed, and doing other work while waiting for the clips to be fetched. (certainly more doable with a robotic library, and can be milked for videos).

 

While the Vault can be run alongside tape storage, given LTT's ever expanding data requirements, I'm unsure if continually scaling up the Vault is an effective long-term solution.

 

My eyes see the past…

My camera lens sees the present…

Link to comment
Share on other sites

Link to post
Share on other sites

This just in, 2nd law is 2nd law

Also, the sun came up today.

 

Please guys hire someone to fix this

"There is probably a special circle of Hell reserved for people who force software into a role it was never designed for."
- Radium_Angel

Link to comment
Share on other sites

Link to post
Share on other sites

Linus, as "CTIO" this is clearly on you.....

In an Enterprise, 10k$ a Month for 1Pbyte is Peanuts....
Perhaps you should put the old Storage in your new Pink rack at home for an offsite backup

Link to comment
Share on other sites

Link to post
Share on other sites

Linus said about redownloading the videos from the channel, you can use Google Takeout which gives you the chance to download all your videos at their original state. No degradation in the quality. Tried this a few times on my other channel and it succeed. Mine was 250+ GB in total for 160 videos.

 

image.png.78ed1c854b44e82de500c0709b8f51b5.png

Link to comment
Share on other sites

Link to post
Share on other sites

8 minutes ago, Zodiark1593 said:

"Oh look, everything is on fire. I wonder how that happened?" 😄

 

LTT can certainly use a dedicated IT person. Having someone able to fix and maintain things during the off-hours would be quite valuable itself. IT can also better set up processes to keep track of the systems in-use, as well as what changes are made over their lifetime, working with procurement, and software/subscription license allocations. I'm not sure how Adobe enforces licensing, but Autodesk appears to be fairly active in pursuing audits. However, given the work environment, you'd probably want someone familiar with production or stage work as well, so as to acclimate easier and better assess the needs of the staff.

 

Regarding storage itself, I feel the reduced running and maintenance costs/requirements of tape would probably better suit the use case here. While not a "glamorous" solution in any sense of the word, risks of hardware failure killing your data is greatly reduced (especially given frequent power outages), and off-site backups are fairly straightforward. The downside of access times can be mitigated by planning in-advance what video clips are needed, and doing other work while waiting for the clips to be fetched. (certainly more doable with a robotic library, and can be milked for videos).

 

While the Vault can be run alongside tape storage, given LTT's ever expanding data requirements, I'm unsure if continually scaling up the Vault is an effective long-term solution.

 

Personally I don't see much issue with having their current HDD based storage, it brings in a lot more advantages than just fetching old videos, but also allows them to overflow from their main production server in case they push the occasional project onto the backburner for a couple of months and that has seemingly happened more than once before.

 

And waiting for a tape library is painfully slow, and muxing different projects for one editor just because Linus or someone else makes a callback to something they did 2 years ago is going to have its own set of organizational downsides, ie longer production/editing times, more room for errors/mixups, and a need for a couple of more projects in the pipe at a given moment. So there is value to having an active storage solution like they have.


But having a tape backup off site would be a wise move. Since it is easy to transport, exceptionally cheap, and the drive itself can stay at the office saving the need for pushing tons of data over the internet to somewhere else.

Link to comment
Share on other sites

Link to post
Share on other sites

Also, on the math, a 1-time investment into LTO-8 or 9 would have given you guys archival capabilities "fairly cheap" (rough figures in my head are anywhere between $60K to $70K USD initial capital and about $8K USD annual operating), with the option of removing some of the older videos from immediate access, reducing the need for so much TIER-1 & TIER-2 storage.  Of course I'd need more info on growth and such to get more accurate numbers.  Please let me work up a proposal - you guys could use it.  My treat.

"There is probably a special circle of Hell reserved for people who force software into a role it was never designed for."
- Radium_Angel

Link to comment
Share on other sites

Link to post
Share on other sites

1 hour ago, Kn0wn said:

Are there any plans to have an offsite backup solution? I fear that a fire or natural disaster could really eff everything up in a really bad way.

They have an offsite backup though offsite backup is solely for wonnick which is the mission critical server of ongoing projects.

9 minutes ago, Muhlis said:

Linus said about redownloading the videos from the channel, you can use Google Takeout which gives you the chance to download all your videos at their original state. No degradation in the quality. Tried this a few times on my other channel and it succeed. Mine was 250+ GB in total for 160 videos.

 

image.png.78ed1c854b44e82de500c0709b8f51b5.png

They don't upload their videos to youtube at the original film quality which is part of the whole reason they keep a back up of everything as youtube doesn't support 8k

Link to comment
Share on other sites

Link to post
Share on other sites

6 minutes ago, Nystemy said:

Personally I don't see much issue with having their current HDD based storage, it brings in a lot more advantages than just fetching old videos, but also allows them to overflow from their main production server in case they push the occasional project onto the backburner for a couple of months and that has seemingly happened more than once before.

 

And waiting for a tape library is painfully slow, and muxing different projects for one editor just because Linus or someone else makes a callback to something they did 2 years ago is going to have its own set of organizational downsides, ie longer production/editing times, more room for errors/mixups, and a need for a couple of more projects in the pipe at a given moment. So there is value to having an active storage solution like they have.


But having a tape backup off site would be a wise move. Since it is easy to transport, exceptionally cheap, and the drive itself can stay at the office saving the need for pushing tons of data over the internet to somewhere else.

I was thinking more along the lines of "Ok, we requested our archive clip for Project A, let's work on getting everything else together in Project A (with the working footage from Whonnuck) while waiting for the old clip to arrive." Though admittedly, I'm not familiar with the access times of Tape. I figured worst-case scenario, maybe an hour to retrieve a given clip.

 

Though I definitely see your point, especially with multiple editors working and potentially needing accesses themselves.

My eyes see the past…

My camera lens sees the present…

Link to comment
Share on other sites

Link to post
Share on other sites

3 minutes ago, Ultraforce said:

youtube doesn't support 8k

image.thumb.png.245fb7a7e2829ff53705518cfdc79699.png

If you can read this you're using the wrong theme.  You can change it at the bottom.

#MuricaParrotGang

The meme thread

 

 

 

All of my image memes are made with GIMP.

 

My specs are crap but if you are interested:

Spoiler

 

The meme-making machine - Optiplex 780:

CPU: Intel Core 2 Duo E8400 @ 3.0 GHz

GPU: NVidia Quadro FX 580

RAM: 2 GB

SSD: Non-existent

HDD: 1 TB

OS: Windows 7

 

Laptop: HP 255 G7

CPU: Ryzen 5 3500U

GPU: Radeon Vega 8

RAM: 8 GB

SSD: 500 GB NVMe

OS: Windows 10

 

Link to comment
Share on other sites

Link to post
Share on other sites

19 minutes ago, Nystemy said:

And waiting for a tape library is painfully slow....

This can be partially mitigated depending on the tape library being used, number of tape decks, and how the backups were originally setup.

 

With 5 libraries @ 3 decks a piece, monthly full backups of 2PB would take 5 days - rough math.

"There is probably a special circle of Hell reserved for people who force software into a role it was never designed for."
- Radium_Angel

Link to comment
Share on other sites

Link to post
Share on other sites

19 minutes ago, Bigun said:

Also, on the math, a 1-time investment into LTO-8 or 9 would have given you guys archival capabilities "fairly cheap" (rough figures in my head are anywhere between $60K to $70K USD initial capital and about $8K USD annual operating), with the option of removing some of the older videos from immediate access, reducing the need for so much TIER-1 & TIER-2 storage.  Of course I'd need more info on growth and such to get more accurate numbers.  Please let me work up a proposal - you guys could use it.  My treat.

Googling a bit, an LTO 8 drive costs about 5 grand.
And the LTO 8 tape is about 25$/TB, and this is what I can just straight up buy now...

So where did you get those exorbitant prices from?
An IKEA bookshelf in a closet at home isn't particularly expensive to own per year, and could store many tens of PB with ease. (Yes it is a lot less fancy than a robotic tape library, but for a pure backup it isn't needed.)

For online storage of the video, 1-3 PB isn't that silly to run. And HDD's are getting cheaper per TB as well, so those old videos are more or less not costing much over time.
Secondly, it gives LMG a made up reason to go for more absurd data storage solutions, and even if it technically isn't needed, it is still interesting content.

 

6 minutes ago, Bigun said:

This can be partially mitigated depending on the tape library being used, number of tape decks, and how the backups were originally setup.

For a backup, fancy "low latency" tape solutions isn't really cost effective. This is for a medium sized media production company, not CERN....

Link to comment
Share on other sites

Link to post
Share on other sites

@Nystemy

 

The tape libraries to run them.  I'm basing this off of using Scalar i3 libraries.  They hold about 50 tapes per library and 3 tape decks a piece.

 

*edit*

 

Oooh, also forgot the repository needs to be worked in.  I really need more info to get something realistic worked up.

"There is probably a special circle of Hell reserved for people who force software into a role it was never designed for."
- Radium_Angel

Link to comment
Share on other sites

Link to post
Share on other sites

1 minute ago, Red 🙂 said:

image.thumb.png.245fb7a7e2829ff53705518cfdc79699.png

Okay guess I was wrong and looking apparently older videos up to like some from August 2015 can be viewed in 4k.

 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share



×