Storage for Documentary Feature

[Crimsontarget]

Hi, 

 

Just joined the forum but am long time watcher of Linus and the team.

 

I'm going to be editing a feature documentary to be shot at 4k or 6k depending on what the DoP opts for. As part of the budget I get to have built a PC rig for the job. Get to keep it after, perks of the job and will also use it for gaming.

 

My proposed rig is as follows: https://uk.pcpartpicker.com/list/7Jm4D2 

In addition I will be have the OS and Premiere, etc running off an Optane 900p I already own.

 

My question is regarding media storage for the film footage. Which would people recommend more?

 

1) A thunderbolt 3 external RAID5 such as LaCie 12 Big 72 TB Thunderbolt 3 

2) Utilizing the 7 bays in the case (Be Quiet Base 900 Pro - with 7 bays) to create a RAID5 off the motherboard

 

In terms of speed, stability, efficiency, noise and any factors you can think of, which would work better? I am working from home and would prefer not to have to put a server in place, I am short on living space as it is with three kids. 

 

I hoping a 70TB RAID5 will be enough for the job, but any advice to the contrary is most welcome. We will be shooting with two cameras for the majority and three for a couple of days for a round table. 

 

Lastly, if I do opt for the internal RAID off the motherboard, would an Optane Memory module help in any way?

[emosun]

This in no way relates to your question but your monitor is a lower resolution than your filming medium which is usually a no no in editing. You don't preview/edit your work on a display that is worse than the source. With such a large budget there's no reason to use the wrong monitor.

edit: you also have a 1000w power supply but nothing in the system that uses 1000 watts.

[thelordofwarr]

Depends on the amount of footage you want to store, but 70TB should be enought (but I don't know exactly the size of these files) the most stable approach would be RAID5 as if one disks dies you still don't loose all the data. That RAM tbh is more worth than my car

[Crimsontarget]

5 minutes ago, emosun said:

This in no way relates to your question but your monitor is a lower resolution than your filming medium which is usually a no no in editing. You don't preview/edit your work on a display that is worse than the source. With such a large budget there's no reason to use the wrong monitor.

edit: you also have a 1000w power supply but nothing in the system that uses 1000 watts.

I will be editing on lower res proxies. When I need to check the footage will plug in my 4k TV. 

 

As to the power supply I want to run as quiet as possible and understand this model won't spin up if if not taxed. Also gives me head room for more pcie devices and overclock. 

[thelordofwarr]

http://www.lg.com/us/monitors/lg-27MD5KA-B-5k-uhd-led-monitor this is the only 5k monitor I was able to find on the go, but you can't use it for gaming...for gaming with the 1080ti you would want to go with a 1440p monitor, as there you can easily maintain more than 60fps with it (In my opinion)

[Radium_Angel]

RAID 5 at a minimum. You don't want to lose months of work to a faulty disk. I'd look into a professional NAS rig and use the internal storage as a backup to the RAID (if this event you are doing is as important as you make it out to be)

[emosun]

Now that I look at the build again I'm seeing a lot of "expensive" choices over work oriented ones , which leads me to believe this is more of a "get the most hardware that i can from the clients budget" style build. Which is fine but not really something I'd like to help someone do to a client.

[Crimsontarget]

2 minutes ago, thelordofwarr said:

http://www.lg.com/us/monitors/lg-27MD5KA-B-5k-uhd-led-monitor this is the only 5k monitor I was able to find on the go, but you can't use it for gaming...for gaming with the 1080ti you would want to go with a 1440p monitor, as there you can easily maintain more than 60fps with it (In my opinion)

If I can get the budget down with a good storage solution I will get another monitor. That one does look good. Thanks. The LaCie is almost £6000 here in the UK. I really want to use that money elsewhere on the system. 

[thelordofwarr]

But it's still better to be able to view your footage in native resolution, otherwise it's just...meh. 

[thelordofwarr]

For work I would switch the 1080ti for a quaddro, just because that's what it's supposed to be, also you could choose cheaper ram

[Crimsontarget]

22 minutes ago, emosun said:

Now that I look at the build again I'm seeing a lot of "expensive" choices over work oriented ones , which leads me to believe this is more of a "get the most hardware that i can from the clients budget" style build. Which is fine but not really something I'd like to help someone do to a client.

I am the producer of the documentary and this is coming out of my payment as editor not out of the investor's pocket. This build is meant to allow me to work as efficently as possible rather than renting an editing space. I want to be able to have strong playback ability and fast render. The RAM is crazy expensive for 128GB but that is the price here. What can I do about it? If I was exploiting the budget, I would put in more SSD storage. I want the cost down, not up, believe me. Hence trying to find a way to reduce the horrendous cost of the thunderbolt DAS. 

 

If you can see ways to bring this down with me, I would appreciate it. I have not build a rig in years and my last edit was on 2k footage and my current laptop (MSI GS70QE) and a 3TB USB3 drive was enough. This is a bigger project and I want it done right.  

 

BTW the price listed on partpicker for the RAM is £600 higher than I have found elsewhere.

Edited March 7, 2018 by Crimsontarget

[palespartan]

I personally with a project with as sensitive information as a film would like a raid 6 rather than raid 5 the extra redundancy of on a second drive being able to fail would be worth the slightly less storage you would get in my opinion is worth it. Otherwise the system is great. But getting the same capacity with raid 6 would require one extra drive and I'm not sure how many sata ports are available on that motherboard. Worst case scenario it shouldn't be too much to be able to add in a raid card.

[palespartan]

7 minutes ago, thelordofwarr said:

For work I would switch the 1080ti for a quaddro, just because that's what it's supposed to be, also you could choose cheaper ram

I don't really work with video editing that much but wouldn't it be more of a benefit if he switched the platform to workstation xeon and used ecc ram for redundancy rather than getting a quadro? I remember seeing somewhere that unless you were going to step it up to a really expensive quadro the performance of the 1080ti was great. 

[thelordofwarr]

That's a valid question...tbh I don't have much experience either... Mx thought process just is that it's better in workstation workloads. But in a way you are right. Hmm. 

[Crimsontarget]

8 minutes ago, thelordofwarr said:

That's a valid question...tbh I don't have much experience either... Mx thought process just is that it's better in workstation workloads. But in a way you are right. Hmm. 

The 1080ti or even the 1080 are often found in editing rigs. They do just fine. Even with the price as is they are a considerably cheaper than the comparable Quadro cards which are something like $2500. 

 

Example PC here with 1070ti. This would be fine for 4k editing but in the likelihood I edit 6k the RAM and extra boost from the 11GB 1080ti will make a difference, https://www.scan.co.uk/3xs/configurator/3xs-i7-x299-7800x-evolve-nle-4k

Edited March 7, 2018 by Crimsontarget

[Crimsontarget]

Is anyone able to offer guidance on the pros and cons of a thunderbolt3 external RAID over an internal RAID. We are talking 7x10GB HDDs here. 

 

If I don't have to go the thunderbolt route I can opt for another MB and bring the costs down. But if the external RAID has more benefits then I would prefer to make an informed decision.

 

Thanks in advance and thanks for comments thus far.

[leadeater]

3 hours ago, Crimsontarget said:

2) Utilizing the 7 bays in the case (Be Quiet Base 900 Pro - with 7 bays) to create a RAID5 off the motherboard

Never this, motherboard RAID 5 is terrible avoid at all costs.

 

Also get Windows 10 Pro no home.

[Crimsontarget]

22 minutes ago, leadeater said:

Never this, motherboard RAID 5 is terrible avoid at all costs.

 

Also get Windows 10 Pro no home.

What exactly are the issues with motherboard raid? Would using a raid controller on the pcie remove them?

 

Will switch to pro version thx.

[leadeater]

@Crimsontarget

I would go with either of these for your storage and only put a OS SSD in the actual system, the extra SSD just adds cost for not much gain. That depends on how fast your remote storage is though, local edit cache is commonly used but only when the remote storage is slow enough to warrant it.

 

QNAP TS-1685-D1531-32G

https://www.qnap.com/en/product/ts-1685

https://www.amazon.com/Qnap-TS-1685-D1531-32G-US-High-Capacity-10GbE-iSCSI/dp/B06XT2CR5W/ref=sr_1_1?s=electronics&ie=UTF8&qid=1520465627&sr=1-1&keywords=TS-1685-D1531

 

12 3.5 HDD

4 2.5 SSD

6 M.2 SSD (SATA)

 

Supports SSD cache of the HDDs

Inbuilt 10Gb networking with 40Gb support.

Supports PCIe SSD AIC

Small physical size

 

Requirement for this setup is swap out your motherboard for either an Asus workstation board that has inbuilt 10Gb or a Supermicro workstation board or buy a 10Gb network card.

 

QNAP TVS-1282T3-i5-16G

https://www.qnap.com/en/product/tvs-1282t3

https://www.amazon.com/TVS-1282T3-i5-16G-US-Ultra-High-Thunderbolt-Thunderbolt3-10Gbase-T/dp/B071R3QT4N/ref=sr_1_1?ie=UTF8&qid=1520465301&sr=8-1&keywords=TVS-1282T3-i5-16G

 

8 3.5 HDD

4 2.5 SSD

2 M.2 SSD (SATA)

 

Supports SSD cache of the HDDs

Thunderbolt 3

Inbuilt 10Gb networking with 40Gb support.

Supports PCIe SSD AIC

Small physical size

 

Unfortunately yes both are reasonably expensive but they are much cheaper than the LaCie 12 Big while also being much more feature rich and I know the QNAP devices are excellent devices.

[leadeater]

18 minutes ago, Crimsontarget said:

What exactly are the issues with motherboard raid? Would using a raid controller on the pcie remove them?

 

Will switch to pro version thx.

A proper RAID controller is fine but you must also buy the optional Flash Cache/BBU module or write speeds with be around 60MB/s, this is why motherboard RAID is so bad because they (not all but mostly all) do not support Flash Cache/BBU at all and also they lack a dedicated parity CPU. Motherboard RAID tends to be rather unstable as well.

 

Edit:

Flash Cache/BBU required for parity RAID only, RAID 5/6

[leadeater]

1 hour ago, Crimsontarget said:

Is anyone able to offer guidance on the pros and cons of a thunderbolt3 external RAID over an internal RAID. We are talking 7x10GB HDDs here. 

 

If I don't have to go the thunderbolt route I can opt for another MB and bring the costs down. But if the external RAID has more benefits then I would prefer to make an informed decision.

 

Thanks in advance and thanks for comments thus far.

Internal works great if you get something like an LSI 9631-8i which you can use SAS to 4 SATA splitters to directly plug in 8 HDDs, you can connect more than 8 HDDs to the single RAID card but you have to use SAS splitters to do it which gets a tad messy not in a server chassis.

[Crimsontarget]

58 minutes ago, leadeater said:

Internal works great if you get something like an LSI 9631-8i which you can use SAS to 4 SATA splitters to directly plug in 8 HDDs, you can connect more than 8 HDDs to the single RAID card but you have to use SAS splitters to do it which gets a tad messy not in a server chassis.

Thanks for input. Will look that the QNAP devices.