What should I do?

[Gudawhan]

I have 15x 3TB external HDD's full.  

I have 4  enclosures, 2x 4x4TB 1x 8x2TB & 1x 5TB. To use as backup

 

The drives are full of a mixture of RAW, master files & rendered versions. I want to organize it all, its all old stuff I'm not trying to loose. But if I organize the backup, & loose a main drive, It will be impossible know where the backup is. 

 

I use SoftRAID on Mac. The newest drive in the bunch is one that just came in from the recovery company (ouch). The rest are all pretty much from 2010-2013. Ideally it would all be in a nice RAID10, or 2 nice RAID10's. But$.

 

What would you do? Thanks!

 

 

 

 

The photo is the cart & Mac I setup to maintain the 15x external HDD enclosures. I had to cut out a piece off the back to get that many USB into it. 

[Streetguru]

2 minutes ago, Gudawhan said:

 

What exactly is the problem?

Why not make it a freeNAS box or build a freeNAS box?

[Gudawhan]

That would cost $. I am trying to use what I own. 

 

There is no "problem". There is more than one solution. Trying to find out what people smarter than me would do. Yes, smarter people would probably have more $, & just buy whatever. But, for the sake of (something) Lets have the smart people pretend that they are stuck on a island or some place they have to use the same equipment I have. 

[Streetguru]

3 minutes ago, Gudawhan said:

 

Do you need like access to this data at all times? If not I would suggest just paying for a yearly back up service or something

[Streetguru]

3 minutes ago, Gudawhan said:

 

You could also consider selling/trading the mac for something with a lot of sata ports to start with, like an older server type thing that comes with plenty of RAM as FreeNAS/ZFS loves RAM

[Gudawhan]

No I don't. I don't use any service like that. I won't. I handle all my own data. 

[leadeater]

@Gudawhan

Don't use RAID 10 for backups, RAID 6 is safer in larger disk arrays with many disks and performs batter (maybe not in Mac softRAID though?) and is far more capacity efficient. In RAID 10 if a single mirror dies everything is lost, the most common use case for RAID 10 is for highly random write I/O like database servers. It's not the best candidate for backups.

 

I'm not sure how easy it is to make any configuration changes though, to reuse the disks in a different kind of setup the data on them will get wiped. You'll need to be able to free up at least 3-5 disk to do an initial configuration.

 

Have you considered something like a used LTO-5 tape drive off ebay? They are around $300 USD and new tapes when I was buying them were $29.90 NZD.

[kb5zue]

2 hours ago, Gudawhan said:

I have 15x 3TB external HDD's full.  

I have 4  enclosures, 2x 4x4TB 1x 8x2TB & 1x 5TB. To use as backup

 

The drives are full of a mixture of RAW, master files & rendered versions. I want to organize it all, its all old stuff I'm not trying to loose. But if I organize the backup, & loose a main drive, It will be impossible know where the backup is. 

 

I use SoftRAID on Mac. The newest drive in the bunch is one that just came in from the recovery company (ouch). The rest are all pretty much from 2010-2013. Ideally it would all be in a nice RAID10, or 2 nice RAID10's. But$.

 

What would you do? Thanks!

 

The photo is the cart & Mac I setup to maintain the 15x external HDD enclosures. I had to cut out a piece off the back to get that many USB into it.

First thing to do.... Get rid of all the crap over six months old.  That would free up a LOT of space making things more manageable. 

 

[paddy-stone]

OK, can't get my head around all that. But for starters I would have a look and see what I could live without. Then convert all the raw footage to some resolution and format/compression level you can live with, and then delete the raw footage, organise the stuff as you go, so the output files end up where you want them.. you may need additional drives for this of course, depending on how full all your storage is ATM.

[Gudawhan]

I think I'm going to move a bunch of data around. Going to move all my master files to my RAID10. Then I am going move the oldest RAW files to the 2 16TB externals, keep 1 at home & one at the office. Not sure how the math is going to work out. 

 

Quote

Have you considered something like a used LTO-5 tape drive off ebay? They are around $300 USD and new tapes when I was buying them were $29.90 NZD.

I had not seen those. Looks like a very affordable option I may adopt. 

 

Quote

Don't use RAID 10 for backups, RAID 6 is safer

Good to know. My current RAID10 is at the office. They read & write to it constantly. I thought RAID10 was the bee's knees. Up to 2 drives can die. But when I was setting it up a drive died & the whole RAID died. Might have been a eSATA connection, but one whole external ejected. I am hoping now that the RAID10 has been running & the major transfers are done, it will be more stable. 

 

Quote

First thing to do.... Get rid of all the crap over six months old.  That would free up a LOT of space making things more manageable. 

No can do. I've already had an occasion when I needed a bunch of my old master files. Upscaling can make old content new. 

 

 

 

[leadeater]

3 hours ago, Gudawhan said:

Good to know. My current RAID10 is at the office. They read & write to it constantly. I thought RAID10 was the bee's knees. Up to 2 drives can die. But when I was setting it up a drive died & the whole RAID died. Might have been a eSATA connection, but one whole external ejected. I am hoping now that the RAID10 has been running & the major transfers are done, it will be more stable. 

RAID 10 is still good, if you're using it to edit off of it may be the better RAID level to use with all the footage scrubbing and multiple streams etc. Also since you're using Mac softRAID the performance will be good which I can't say it would be for RAID 5/6.

 

Parity RAID levels like 5 & 6 require write-back cache to have even usable write performance, these are optional extras on hardware RAID cards and people call them BBU or Flash Cache. Software RAID typically doesn't have write-back cache due to data corruption risk.

 

Mainly I'd just recommend your online disk backup be RAID 6 to keep the cost down, allow for expansion because typically you cannot expand a RAID 10 array, and for resiliency if you end up with 16 disks in a single array.

 

I'd also explore the LTO option further that I mentioned, the drives can be expensive if you want a newer generation like LTO-6 but long term can be a heck of a lot cheaper than buying more HDDs and introducing more and more failure points, every spinning disk is a failure point.

 

You can get a new LTO-6 drive for around $1,600 which yes is expensive but compare that to a few 10TB HDDs versus $25 per LTO-6 tape which will store 2.5TB uncompressed or 6.25TB compressed.

https://www.amazon.com/Quantum-Ultrium-6-Drive-Height-Internal/dp/B00F0RJ2XY/ref=sr_1_18?ie=UTF8&qid=1509925050&sr=8-18&keywords=LTO-6

https://www.amazon.com/HP-HEWC7976A-Ultrium-6-25TB-Cartridge/dp/B00AHQUV3S/ref=sr_1_4?s=electronics&ie=UTF8&qid=1509925596&sr=1-4&keywords=LTO-6

 

Comes down to analysis on your end to how much data you think you'll have over time, how much you value keeping raw footage, how much you want to invest in to and manage a large storage array etc.