Videographer Storage Management Solution

[bern_eK]

I just want to preface this thread with the fact I've looked through multiple YouTube videos, online threads etc about the matter - but they've mostly served to confuse me more. Let's cut to the chase. 

 

I'm a fulltime videographer & photographer, I shoot multiple projects weekly as well as personal projects. I work off a single laptop with an internal SSD, and have 5 slow external HDDs and a single 10TB backup drive that sits on my desk at home. Right now, it's an absolute mess, I'm editing projects off my slow external drives because my SSD is full, I have footage all over the place with no clear backups.... It is just a mess and I have been trying to figure out the best way to clean everything up. 

 

I was planning on buying a new 2tb inernal SSD and installing it, editing my projects off there before transfering to a backup drive (potentially an external SSD incase my laptop fills up again) and then another big backup drive. 

 

But I've seen so much online about storage methods, having RAID drives - networks... there is so much info i don't know where to start. 

 

The caveat here is that I spend quite a lot of time on the road, and am about to embark on a 5 week project - so being able to backup and store footage during that time will need to be on something portable... 

 

Any help would be much apprecaited! 

[WWicket]

How much space do you need over the 5 weeks? (2tb? 8tb?)
How portable do you need it? (Carry with you in a backpack while hiking to a remote shoot or just take in luggage?)

What sort of budget?

[GoldenLag]

28 minutes ago, bern_eK said:

I was planning on buying a new 2tb inernal SSD and installing it, editing my projects off there before transfering to a backup drive (potentially an external SSD incase my laptop fills up again) and then another big backup drive. 

just use google drive or any similar infinite cloud storage solutions for backup. 

[Streetguru]

44 minutes ago, bern_eK said:

 

Are you going to have access to high speed internet while on the road, or do you have unlimited upload from a 4G provider? You can just use a service like backblaze in that case.

 

Otherwise you'd have to set up a NAS at home and dump things onto that. Still need unlimited data and good 4G though.

[Tamesh16]

3 hours ago, Streetguru said:

NAS at home and dump things onto that.

i do videography,but not for a very long time, ( shoot mostly in hight bitrate 1080p) so my 2tb hard drive and 1 tb cloud storage is enough, but more than likely a nas would be the best way about this

[Thanatopsis]

From working in a video production house for 10 years all i can say is, what level of effort/work on a daily weekly monthly bases do you feel comfortable putting in?

 

 

Its easy to say do a 3 teir setup but if you dont have the time or willingness then its a waste of money.

 

With this being said you can build your own external ssd drives and depending on the laptops IO you can run between sata max bandwith and 1000mbs if you have thunderbolt on your LT.

 

 

[Streetguru]

12 hours ago, Tamesh16 said:

i do videography,but not for a very long time, ( shoot mostly in hight bitrate 1080p) so my 2tb hard drive and 1 tb cloud storage is enough, but more than likely a nas would be the best way about this

It's either between buying a synology NAS and just tossing some 4-6TB drives in it.

Or building your own FreeNAS box of some kind. Lot more work, potentially more reliable, way more performance if you want to do some automated stuff like file compression, or also make it a media server.

https://www.synology.com/en-us/knowledgebase/DSM/tutorial/Network/How_to_make_Synology_NAS_accessible_over_the_Internet

https://www.amazon.com/Synology-bay-DiskStation-DS418-Diskless/dp/B075N17DM6/ref=sr_1_4?keywords=4+bay+NAS&qid=1568844363&s=electronics&sr=1-4

[bern_eK]

18 hours ago, WWicket said:

How much space do you need over the 5 weeks? (2tb? 8tb?)
How portable do you need it? (Carry with you in a backpack while hiking to a remote shoot or just take in luggage?)

What sort of budget?

- Probably around 2-3tb

- Camera bag packable 

- For the whole storage solution, around $2kAUD roughly 

 

I want to find a solution that will work long term, not just for the trip... but I may not be able to cover both in one go. 

[WWicket]

53 minutes ago, bern_eK said:

- Probably around 2-3tb

- Camera bag packable 

- For the whole storage solution, around $2kAUD roughly 

 

I want to find a solution that will work long term, not just for the trip... but I may not be able to cover both in one go. 

For your trip, I'd probably get a 4tb internal ssd and a 2tb external ssd (for backup).

You should be able to get a 4tb ssd for just over $800AUD ($500USD) and an external 2tb ssd for around $370AUD ($250USD). 

 

Then for long term backup get a 3-bay+ nas enclosure with raid 5 support and couple more 10tb drives to populate it with (probably around $1200AUD). (Or better yet, if you are up for a small project, build a nas).

[bern_eK]

16 hours ago, Tamesh16 said:

i do videography,but not for a very long time, ( shoot mostly in hight bitrate 1080p) so my 2tb hard drive and 1 tb cloud storage is enough, but more than likely a nas would be the best way about this

You edit off that 2tb HDD - how is that for you? I've found editing off external HDDs slow, but mine aren't the fastest. What cloud storage are you using?

[bern_eK]

13 hours ago, Thanatopsis said:

From working in a video production house for 10 years all i can say is, what level of effort/work on a daily weekly monthly bases do you feel comfortable putting in?

 

 

Its easy to say do a 3 teir setup but if you dont have the time or willingness then its a waste of money.

 

With this being said you can build your own external ssd drives and depending on the laptops IO you can run between sata max bandwith and 1000mbs if you have thunderbolt on your LT.

 

 

Could you explain the 3 tier setup and what effort & cost it requires? 

 

 

[bern_eK]

19 hours ago, GoldenLag said:

just use google drive or any similar infinite cloud storage solutions for backup. 

The problem is my home internet is fairly miserable, so getting 6TB onto the cloud would take an age... also paying for 6tb of google drive storage is pretty expensive, right?

[bern_eK]

53 minutes ago, WWicket said:

For your trip, I'd probably get a 4tb internal ssd and a 2tb external ssd (for backup).

You should be able to get a 4tb ssd for just over $800AUD ($500USD) and an external 2tb ssd for around $370AUD ($250USD). 

 

Then for long term backup get a 3-bay+ nas enclosure with raid 5 support and couple more 10tb drives to populate it with (probably around $1200AUD). (Or better yet, if you are up for a small project, build a nas).

This is sounding promising - so a 4tb to edit off, external to back up while on the go - and the nas for futurue backups. 

 

Any brands/products you would recommend of the internal, external & NAS?

[Tamesh16]

36 minutes ago, bern_eK said:

You edit off that 2tb HDD - how is that for you? I've found editing off external HDDs slow, but mine aren't the fastest. What cloud storage are you using?

no i have about 750 gbs of NVME storage ( in signature) on my laptop that i use to edit, the 2tb is bulk storage and the 1tb is from one drive that i use to share files with clients 

if it helps i pay for 1 tb once drive, along with office 365 for $7 a month

[WWicket]

28 minutes ago, bern_eK said:

This is sounding promising - so a 4tb to edit off, external to back up while on the go - and the nas for futurue backups. 

 

Any brands/products you would recommend of the internal, external & NAS?

Samsung, WD, or Intel for the internal. For the external, I'd just go with the best deal you can find when you are ready to buy. 

 

For the NAS, diy with freenas or unraid is going to be better than any ready-made budget NAS, but is obviously more of a project.

[MandicReally]

Another vote for an Unraid / Freenas server.  I had a Qnap NAS for a few months and I hated it.  It was slow, the interface was slow and clunky, it was just a pain to use.  I sold it and built an Unraid Server with used Xeon/Server hardware off of Ebay for not much more.  It is definitely more work to setup but once that is done it works so much smoother for me. 
 

For note my setup is:

-Edit on a 500GB NVMe drive (fast and makes editing a breeze on 4k footage)

-4TB External Drive (Backup finished project exports to this drive.)

-26TB Unraid NAS (Backup ALL project files there as well as complete Exports.)

 

I really need to figure out a Cloud storage / off-site storage setup at least for my complete project files. 

[harryk]

On 9/18/2019 at 5:28 PM, bern_eK said:

- Probably around 2-3tb

- Camera bag packable 

- For the whole storage solution, around $2kAUD roughly 

 

I want to find a solution that will work long term, not just for the trip... but I may not be able to cover both in one go. 

I'd recommend getting a couple of Samsung T5 portable SSDs for working on the go. Ideally two or more so you can duplicate stuff, (and don't store them in the same bag). They're fast enough to edit with, very small and robust, and not extremely expensive.

 

For cold storage get whatever cheap external drives you can find with reputable name. I've purchased a few of these Seagate 8TB drives over the last year and are pleased with them. Again double the number of drives for redundancy and don't store the drives together (ideally store one copy in a completely different place such as a friends house).

 

For hot storage either buy or build a NAS. Arguably not worth it unless you need to regularly access archive footage or have an interest in building a server.

[Thanatopsis]

On 9/18/2019 at 10:05 PM, bern_eK said:

Could you explain the 3 tier setup and what effort & cost it requires? 

The 3 tier setup is Immediate edit storage, mid range storage, and long term storage.

 

the immediate storage is the fastest external drives you have either ssd over usb3.0 or thunderbolt raid enclosures depending on what you can afford and what your computer supports.  this is where all your project files will live  as you work on them, this is any jpeg, sfx, animation, audio and master footage along with all the intermediate revisions of such.  generally there will be no off site backup and maybe only backup of master footage.

 

the Mid Range Storage, is the storage normally a NAS for the projects over the last 3-8 months and anything that you know you activity will be pulling back up weekly or monthly.  this is the storage to bridge availability and speed.  you generally will put the finished export, master files, project files and final revisions of any graphics extra audio, sfx, images and the like.  You will offsite backup this either to the cloud or some other method.

 

Long term storage is normally going to be the storage for just the master footage and final export for a project, this is the storage that you use when you don't think you will be going back to the footage except to pull a random clip from something once every month or every few months.  You will want to to make an off site backup for this also but you may decide that only the final export is what get the off site back up or not depending on what you need.

 

This can become a lot of work to manage the transfers of info depending on how many edits or projects you have in motion in a given week. You will have to keep some type of record keeping and policy in place for when x project moves from one place to the next and the price of keeping 2 seperate off site backups can get expensive especially if you capture footage in ProRes or a Raw format.  Another draw back to this method is cost,  NAS's aren't Cheap and you will probably end up storing duplicates of data between the Mid Range and Long Term Storage at one point or another.

 

 

Data Management for video can be its own job for larger productions or even just productions that use lots of RAW media.

 

 

After reading your Explanation of what you need right away, I would look at getting mid Tier NVME drives and put them in a USB enclosure as they will saturate any usb 3.0 connection you will have and from my personal use have been great editing tools for Day to day editing on the go.  depending on where your going you may not be able to cloud backup but Backblaze is generally a good option for cost effective backups.

 

Edited September 20, 2019 by Thanatopsis
I saw follow on posts.