For Discussion: Tackling the Pain of Video Production Storage

[45Drives]

Hey Guys,

 

We posted our blog on here yesterday for discussion of our latest article  "How to Tune a NAS for Direct-from-Server Editing of 5K Video". You can check out the blog here if you haven't already: http://45drives.blogspot.ca/2016/05/how-to-tune-nas-for-direct-from-server.html (thanks everyone for their feedback, too!)

 

We also wanted to post this video of our first of many user spotlights featuring video production studios as we help them tackle the pain of video production storage. Feel free to check out the video here: 

 

We have worked with @LinusTech over the last couple years assisting with his media storage pains and we wanted to feature some more.

 

With that being said, there are some video production professionals on here that we would love to hear from. 

 

A). What do you guys think is the biggest pain of media/post production storage today?

B). Where do you think the solution is? Or, how have you solved it in the past?

 

Would love to hear from you.

 

Thanks,

[Aytex]

Solution is Cloud drive

[leadeater]

@45Drives One of the biggest problems I see with video production is data protection and data backup due to very large data size requirements. For raw footage and final content/archive object storage across multiple servers or sites is a good way to protect data and reduce raw storage requirements (erasure coding).

[tt2468]

@45Drives One big issue I find with HDD solutions is speed. With SSD servers, I have been able to saturate multiple 10Gb links, but with HDDs, that hasn't been possible. A good caching solution would probably help the best. And if the cache was big enough, it could be used for entire projects at a time. I personally don't have these issues, because I work for an SSD company.

[45Drives]

@leadeater Agree with this statement wholeheartedly, but Object storage is not for everyone quite yet. The free and open source packages like Ceph are not the simplest to implement, and other streamlined offerings are expensive.

 

But I do agree object storage is a much better way to store data, especially in large quantities. Erasure coding significantly reduces the number of physical servers needed to protect the data which is very nice on the wallet and the environment.

 

A nice middle-ground is you could use a clustering file-system like Gluster, BeeGFS or others so you can scale out your servers beyond one pod. This way, you can grow (rather than replace) your storage solution or use replication for data protection.

[leadeater]

1 hour ago, 45Drives said:

@leadeater Agree with this statement wholeheartedly, but Object storage is not for everyone quite yet. The free and open source packages like Ceph are not the simplest to implement, and other streamlined offerings are expensive.

 

But I do agree object storage is a much better way to store data, especially in large quantities. Erasure coding significantly reduces the number of physical servers needed to protect the data which is very nice on the wallet and the environment.

 

A nice middle-ground is you could use a clustering file-system like Gluster, BeeGFS or others so you can scale out your servers beyond one pod. This way, you can grow (rather than replace) your storage solution or use replication for data protection.

Yes that is the critical issue currently with object storage, it is still very immature as an industry solution and also has a high upfront cost to implement since small deployments do not have any significant cost savings to over come the initial setup complexities and the professional development staff would require to manage the system.

 

We are currently about to start trialing Netapp Storage Grid, our current storage vendor, and also Storage Spaces Direct. Storage Spaces Direct is completely different but thought it would be worth mentioning anyway, shows some promise in other areas.

 

There are two key drivers for why we are looking at Object Storage, long term disk backup retention and storing mass amounts of research data securely and protected.

 

We currently using Commvault as our backup solution and if you have ever seen how much it costs per TB on capacity based licensing you'll quickly understand why we are looking at other areas. We have 6 months of on disk backup retention with an aim to have 1 year on disk before offloading to tape but this is already putting significant strain on the SSD's that host the deduplication databases. Re-hydrated the data storage required would be 1.55PB and then we have two copies so would be 3.1PB. Assuming linear growth if we were keeping 1 year that would be 6.2PB so we are also looking at ways to be putting data in to the Object Store deduplicated possibly using an appliance driven through Commvault using S3 connector which does not require any capacity license.

 

The other use case I mentioned was the research data storage. This is an area where traditional backup methods don't scale both in cost and time required to even do this task. Multi PB data storage is just difficult to deal with especially when we are still operating in a traditional data storage mindset with core business systems being around 200TB.

 

Edit: Removed some information details about our environment,.

[45Drives]

@tt2468 Definitely agree, SSDs are inherently faster than HDDs. However, large disk count solutions can indeed saturate multiple 10GbE NICs. We have tested our Storinator S45, fully loaded with 45 HDDs, saturating 3 of them. This might be good to show for our next blog, perhaps.