Storage server issues

evan_pyle · December 28, 2015

Hello,

After watching your latest video on vessel about the server RAID failure I'm just left a little speechless, not quite as speechless as hard drives on a shelf in a bathroom but still. I really like your "get it done" solutions to a lot of problems and I have been known to partake:

1. http://i.imgur.com/n2jjoBy.jpg overkill PFSense servers

2. http://i.imgur.com/DB6odL2.jpg re purposing a VDI cluster because a video render is taking to long

But homebrew solutions to storage isn't something I recommend. Here's what happens if there is a failure one of the SANs, I get an email notification saying that the SAN has had an issue and switched over to a backup controller/drive/PSU. I call the vendor and a part and or technician is at the door in 4-24 hours all with zero downtime. Sure the service and device costs more for less performance than yours but I have never once had a 14 hour day stressing my guts out with the orginisation unable to function.

Have a look at the real costs associated with a server failure. How long were your workers (and you) doing nothing valuable on company time, how much revenue did/would have you missed by not posting content, how many extra hours would have everyone had to work to get the lost content back, how many extra hours would have everyone had to work to get back on schedule, and how often does something like this happen?

Now the RAID5 thing, along with a controller that can't import configuration (and work), has to be a joke right? Two drives and all the data is gone, it's just crazy. RAID 6 minimum for anything over a few TB and either qualify your systems (test what does and doesn't cause a failure and how/if you can recover) or buy systems that have been (HP/Dell/Cisco/Whoever).

As far as the stripe section goes your primary risk is the three controllers, a single failure will kill everything. Backup can not be seen as something that should be done with a stripe, it must be done. Have that thing backed up nightly and have the footage ingested to a second destination as part of your workflow, or hold onto the cards until after a backup has been taken.

Have a look at your actual requirements when sizing a replacement, while all SSD is nice the price is just crazy. A new SAN this year will have 8TB of SSD and 60ish TB of HDD, with the SSD's handling all most all writes and caching reads from frequently accessed blocks automatically and the system is really fast 99% of the time for our usage pattern.

UPS wise get an electrician to log your power over a day/week and see if you need to switch to a double conversion UPS. Doing so has solved a whole bunch of minor issues with equipment here because we have some very large and intermittent loads.

Good luck!

GoldenBE · December 28, 2015

I'm pretty sure they can pay a proffesional to do it 10 times better but they don't want to since it's no fun.

They enjoy building these things and so does their audience, the amount of videos they were able to make because of these servers is pretty amazing. And it's worth risking the data in my opinion.