Solution for Black Magic 4K production storage?

[Tyler 19xx]

Hi all. I work with a small company doing Video production. We are running into a Issue with long term storage of all our 4K footage. I believe only 2- 3 months ago we acquired a 36TB Raid, with is now full. We are using 4 Blackmagic Pocket cinema camera's in 4K (Blackmagic Raw/12:1) on a shoot, ranging from 1.5 Hour, to 6 hours. Just this week we have done 3 days in the studio, recording 4 hours each time with 3-4 camera angles. 

 Some key points-

- Currently there is 1 part time experienced editor. So deleting unusable clips will be slow to happen.

- End products are hours long, compared to more 4k YouTube videos being about 20 minutes.

- Being a small company, money is somewhat of a issue.

 

So I hope you can see our dilemma here. At the time being we are just accumulating raid boxes, and that's not attractive. 

 

 

Thanks in advance for your knowledge and time! 

 Tyler~

[leadeater]

Have a look at the company ProMAX, https://www.promax.com/. They specialize in storage for video production and have solutions for NAS live editing as well as archive storage for long term retention of projects. This might be slightly higher $/TB than what you are doing now but if you get a good storage array/NAS controller you'll be able to scale up the amount of attached disk shelves which long term is cheaper than buying multiple NAS's and multiple TB/USB JBODs and RAID enclosures.

[unijab]

Eposvox on YT.. talks about compressing or encoding videos to save NAS space.

[Chadahc]

Invest in an LTO drive?  You can get 9TB per tape.  The drive will cost you about $3500, but the tapes can be had for $60, which puts you at $7/TB, which is less than 1/4 of what a bare HDD costs. 

[dalekphalm]

How long do you need to keep footage for? Forever? Or just a set time period?

 

If forever, I'd probably institute a policy where after a certain period past project completion (say, 6 months), you transcode the footage into a much lower bitrate - something usable if you need to pull really old footage, but something far lower quality than the raw footage.

 

In terms of hardware, you're going to want to invest in a scalable hardware solution, where you can just buy an expansion shelf that will hold more disks.

[dtmcnamara]

Everything everyone has suggested will work, but honestly isn't needed. I just recently setup a server for someone in a similar situation. They have 5-6 people that shoot but only have one editor, which makes things super simple. Also they are a not for profit company so funding is always tight. FreeNAS will work just fine with a single editor, and can even scale out to multiple editors with the correct hardware. I would suggest a 10GB switch in between the server and the workstation in order to speed up everything, but it can be done on a multi 1GB NIC setup.

 

Just remember if you go this way, in order to get the best performance with FreeNAS 1GB of RAM per 1TB of storage.

[dalekphalm]

3 minutes ago, dtmcnamara said:

Just remember if you go this way, in order to get the best performance with FreeNAS 1GB of RAM per 1TB fo storage.

This is entirely false information, based off of people misreading the ZFS technical documentation.

 

You only require 1GB of RAM per 1TB of Storage if you are using deduplication, which for a video editing office is not recommended anyway.

 

For regular operations, adding more RAM has very limited performance benefits. It acts as a very stupid read cache. It won't help at all with write performance, and with read performance, it'll only help the most frequently accessed content. Over a Gigabit LAN, ARC essentially becomes useless when you've got a decent storage setup with good IO speeds (and that includes mechanical drives)

 

If you need to increase performance that much (Especially over a 10Gig network), you're better off setting up an L2ARC with SSD's - or switching entirely to an SSD based pool (you can even setup an SSD based pool as the "dumping ground", and then setup a Cron job that will transfer the files over to the mechanical pool at a later time - though that's getting into a more advanced setup).

[dtmcnamara]

25 minutes ago, dalekphalm said:

This is entirely false information, based off of people misreading the ZFS technical documentation.

I must admit, I vaguely remember hearing this and upon looking into it you are 100% correct. Luckily RAM is cheap right now so its usually a small part of the cost of a system even if you build it at 1GB/TB. I might pull my system our of production and see what I can lower the RAM down too before I start to see any speed drops. Do you know of any good write ups on storage size VS RAM with ZFS? Id be interested in seeing what is really needed on larger storage setups, something like 100-200TB. I keep seeing all thats needed is 8GB, but I seriously doubt that with 100+TB. I know Synology recommends something like 32GB for 240+TB, same with a few other companies.

[dalekphalm]

5 minutes ago, dtmcnamara said:

I must admit, I vaguely remember hearing this and upon looking into it you are 100% correct. Luckily RAM is cheap right now so its usually a small part of the cost of a system even if you build it at 1GB/TB. I might pull my system our of production and see what I can lower the RAM down too before I start to see any speed drops. Do you know of any good write ups on storage size VS RAM with VFS? Id be interested in seeing what is really needed on larger storage setups, something like 100-200TB. I keep seeing all thats needed is 8GB, but I seriously doubt that with 100+TB. I know Synology recommends something like 32GB for 240+TB, same with a few other companies.

I assume you meant Storage Size vs RAM with ZFS, not VFS?

 

Anyway - I don't have any whitepapers or anything that compare and contrast various RAM capacities, but the short answer is 8GB is generally sufficient for most scenarios (outside of additional services running that will use more RAM, like VM's, plugins, etc) - you can even run FreeNAS on 4GB or less of RAM (not recommended though).

 

For super large datastores? More might be good, but it really depends on your data needs, and whether you'd benefit from more ARC capacity - people that read a lot of the same static documents over and over again benefit a lot from ARC. People who are constantly changing data does not benefit much from ARC.

 

RAM might be cheap, but considering you should be using ECC (which decreases the value proposition, as ECC is more expensive), to get better performance, SSD's might start to become more appealing for at least some of the workload.