Whonnock RAID Recovery Vlog

[nicklmg]

Amazon: http://geni.us/3ZYW

 

We experienced a MASSIVE RAID failure on Whonnock server and lost a crapload of data. Can we get it back?

 

[givingtnt]

lol that was fun.

 

 

 

 

for me at least

[kameshss]

I was really touched by this video.

 

So intense.

 

@nicklmg @LinusTech How much did you guys spent for this recovery?

[Jacktastic-Mofo]

I almost had a heart attack watching this video on Vessel.

[Syntaxvgm]

well, at least you go your data 100% back. More than Logan from tek syndicate can say. You should send Wendell and him a condolence card and a screenshot of your recovered data with "I survived my parity raid ordeal, suck it" written in the card.

Also, attaching the viewership statistics from the 7 gamer PC video wouldn't hurt. 

[Kobathor]

Trust me! I've already watched all 22 minutes.

 

I can tell from the title that I'm gonna laugh. Sorry LMG.

EDIT- Yeah, I didn't laugh all that much.

[Agent181]

FUCK THIS VIDEO IT PISSED ME OFF 

 

 

still wub u guys <3

 

Edit: so when is the vessel vid up?

Edit 2: its up

[givingtnt]

Amazon: http://geni.us/3ZYW

 

We experienced a MASSIVE RAID failure on Whonnock server and lost a crapload of data. Can we get it back?

 

 

this raises the question.

soudn't you guys be doing a save on the server and simultaneously saving to the computer its being worked on ? that way you have a copy on the server and on your pc. so if one of them fails, then not only you can keep working. but you don't get to loose all of your current work.

@LinusTech

[Agent181]

this raises the question.

soudn't you guys be doing a save on the server and simultaneously saving to the computer its being worked on ? that way you have a copy on the server and on your pc. so if one of them fails, then not only you can keep working. but you don't get to loose all of your current work.

@LinusTech

this was a while ago so they have figured it out with a backup server off site 

[Tmt97]

I love how the titles for these vids are always the opposite of the truth. "7 gamers one cpu": has two cpu's... "All of our data is GONE": "It's back!" I'm gonna call it now, when linuses personal rig update comes out, it will be impersonal, and about someone else's rig.

[fishy]

i wonder how much something like this costs

and if Linus thinks it's not even nearly enough for what they did

they are our heroes as well!

[blu4]

Unless you have experienced a failure like this you have no idea how stressful this is. Man, I remember having a network issue on my work server.. Most stressful day in my career.

 

Big virtual hug to @LinusTech

[cesrai]

this raises the question.

soudn't you guys be doing a save on the server and simultaneously saving to the computer its being worked on ? that way you have a copy on the server and on your pc. so if one of them fails, then not only you can keep working. but you don't get to loose all of your current work.

@LinusTech

Storage space ?

[AxelRantila]

This was intense. Good that you got your data back.

 

Really shows how important offsite backups are

[PolosElite23]

Welp...that was intense. Glad to see everything is back and that everything didnt explode.

[JacobLambda]

Well, something that could be interesting to do is to cache all the un-backed up data onto workstations until the offsite vault can get to it. 

 

If you were to do it you could keep track of everything between the server and offsite with some fairly simple diff script and then take all the files that aren't synced and copy them to hidden folders on the workstations. You could assign quotas to the folders as to not use up too much space. When syncs are completed, scripts could then verify success and then remove the data from the workstation folders as new un-synchronized data arrives. Additionally, a script could be used to link all the files/directories cached on the workstations as a network device hosted off another machine such as the router or even as an entirely separate virtual machine on any one of the racks.

 

This would allow most if not all data from being lost, remove the stress of dealing with recovery agents, and almost entirely eliminate downtime. While I know you guys got all the data back in the end, I believe a system such as this would greatly reduce the hassle and stress of any possible future worst case events similar to this.

[linushellskitchentips]

thank god that you saved the data because if you would lost any  data than ye the whole chanel woud ve been down for monthses

[beigehornet]

Curious to know the approx cost for this service. I will be recommending this service to others just from this video.

 

They need to sponsor LTT

[Mandrewoid]

Would this whole debacle be avoided/mitigated by using UnRAID?

[Denned]

And that's why you get proper storage.

[kanliot]

Hi, this is my first post to linustechtips.  Video was great, hope you guys get some good luck soon!

 

I would like to know how the motherboard failed, and would like to know exactly how the RAID card failed.  It seems like you are really lucky not to have more corrupt data, IMHO, must be a good RAID card.   Also what happened to your nightly backups.

 

TIA,

karl

[MichaelScrip]

How many gigabytes, or even terabytes, does LMG generate for each video?  It sounds like a lot!!!

 

I'm so glad I don't have to deal with that much data.  I do video production... but I usually only come home with about 50-60GB of video files.  My camcorder shoots to two SD cards so I have redundancy in the camera. Then I don't delete the cards until I have copied, and verified the data, onto at least two separate hard drives.  (And sometimes there will also be copies on a laptop and external hard drive in the field before I get home)

 

Before I start working... I'll make sure I've got two copies of the video files on a pair of vault drives.  Then once I start working... I'll periodically mirror the project folder on my work drive to one of my vault drives. And then a second mirrored copy ends up on my 2nd vault drive.

 

I don't have any offsite storage yet... but it's on my list.   Short of a house fire... I feel pretty safe.

 

I usually have the original two SD cards... possibly a copy on the laptop and an external portable drive... then the work drive... then two vault drives.

 

So that's up to 7 copies of the video files.  And once the project is finished and delivered... it will remain in perpetuity on the pair of vault drives.

 

My plan is to get a 3rd vault drive... and rotate that at a friend's house or something.

 

Luckily I don't have so much data that I need giant RAIDs... nor do I need networked high-speed storage.  I can get by with standard hard drives on a single desktop system.

 

When I saw the title of this video... my heart sank.  I'm glad they recovered!

[scottyseng]

Great video, very intense. I'm happy the data was able to be recovered.

 

It's a painful reminder to everyone to make sure to have backups of your data. I personally wouldn't do RAID0 on arrays across controllers...too risky.

 

The motherboard looks to be completely dead (or soon to be dead). I kind of wonder what abuse that LSI controller took from the motherboard to fail like that.

 

Also would be neat if LSI or the motherboard company responds to the issue or if it can be found what exactly caused the motherboard to start failing like that.

[Zaski]

Amazon: http://geni.us/3ZYW

 

We experienced a MASSIVE RAID failure on Whonnock server and lost a crapload of data. Can we get it back?

 

 

Just for the sake of learning from your experience: would you guys consider publishing some kind of detailed guide on what you had, how it was configured, what exactly happened, what you did to solve the problem and how you intend to prevent the same problem from happening again? With schemas and things like that. I'm currently studying Computer Engineering and I am quite intrigued.

 

I think we can all learn from what happened here.

[me99771]

For anyone wondering how much that recovery service costs, I just checked their site and it says that the remote recovery service starts at $3000 USD.