Storage solution for 4K video project/source files

[ALwin]

Ok I'd like some advice about some storage solutions for my workstation.

The following is what I currently have in my case (Cooler Master Cosmos II) attached to my logic board (MSI z97 xPower AC)

• 1x 480GB SSD for OS
• 2x 500GB SSD in RAID 0 for games and software
• 2x 2TB WD Green in RAID 0 and 2x 2TB Seagate Barracuda in RAID 0, drives that were taken from older machines
• The Seagate drives are acting as a Mirror for the WD drives
• The WD Green drives in RAID 0 are about 50% full already, music, videos, etc.

All of the above storage drives, along with the optical drive and an unplugged SSD that stores a clone of the OS SSD with Windows 7 (before I upgraded to Windows 10) use up all the SATA ports on the logic board.

In the near future I will be starting to work with 4K resolution videos that will use eat up a lot of storage space when being worked on. I have external storage devices for archival and backup. However I am considering adding more storage drives for my workstation. These are my 3 options

• I am considering either a USB / USB+Thunderbolt ready multi-bay external RAID solution for the working files and projects
• I buy another NAS, such as a Synology 8-Bay system, however I am concerned that the speed of a network connection may become a bottleneck for working with large video files (especially 4K ones)
• I buy a PCI Express RAID installer card and add more drives inside my case*

*My logic board comes with 10 onboard SATA ports which are all used up (4x HDD, 4x SSD, 1x Optical, 1x eSata port on the Cosmos II case). But I have lots of unused HDD/SSD cages/bays in the Cosmos II, I just don't have spare ports on the logic board.

Hence, what are your thoughts about the 3 options to expand my storage for the files that I am working on?

The pros of looking at an external storage solution (No. 1 and No. 2) would be that I can also work on the projects with my portable workstation and not be tied down to my desktop. The con is space to put that external drive.

What is a very reliable PCI Express RAID controller card I can install internally? Something robust, reliable and can let me connect between 4-8 drives (either SATA or SAS).

I already have ample storage systems for archiving and backup, both onsite and offsite, so I do not need advice in this matter. For projects that I have completed, I will take them off the "working drives" and move them to my archive/backup storage areas, however I tend to work on several projects at the same time and projects can last from a week to a few months to complete, hence I wish to expand the storage space for my workstation. Project files will consist of video files a few minutes in length to a several hours long, most likely split into many short clips.

[scottyseng]

-snip- 

 

Since you already have a system for backup and archiving, I would say to get a NAS (or build your own with 10Gb/s Ethernet) or the PCI Express RAID card (leaning more towards the RAID card, but I have no idea how much space is left inside of your PC case...If you're running out of space in your case, I would push more towards the NAS / 10Gb/s Ethernet direct attached to your PC would give you 1.25GB/s at full speed..Probably 1GB/s realistically). I would stay away from thunderbolt enclosures simply because you're too forced into that ecosystem and a NAS is usually more capable/expandable.

 

A good PCI Express RAID card is the LSI MegaRAID 9361-8i (Make sure to buy the cachevault battery backup for it). It has two SAS ports and with SAS to four SATA breakout cables, it will handle 8 drives. You can optionally get a SAS expander and have a larger set of drives or plug the SAS into a server backplane and have it expand out that way. However, you need good cooling for the RAID card, they get really hot as they are meant to be in a server chassis with high airflow and not in a silent gaming PC. Any decent RAID card will cost around $500-700. Do not get the entry level RAID cards as they aren't true RAID cards. The true RAID cards have their own DDR2 / DDR3 cache.

 

As for drives, look at WD Reds, Red Pros, or the enterprise SE or RE lineup. HGST NAS drives are pretty good as well.

 

On my RAID card, the LSI 9260CV-8i, six 4TB WD Reds in RAID 10 get around 500MB/s read and write sequential. On my old array, four 4TB Reds got around 350MB/s read / write sequential. However, 1Gb/s Ethernet limits my data speed from the NAS to a measly 80MB/s. I'm planning on getting 10Gb/s NICs soon.

[ALwin]

snip

 

Inside my case I have room to add about 8 more HDD/SSDs, I simply am out of SATA ports on the logic board.

 

How does LSI compare to Intel RAID controllers?

[scottyseng]

Inside my case I have room to add about 8 more HDD/SSDs, I simply am out of SATA ports on the logic board.

 

How does LSI compare to Intel RAID controllers?

 

Intel RAID controllers are rebranded LSI controller (I mean the Intel PCIe RAID cards). Before, Intel used to make their own RAID ships (7+ years ago I think), but these days they just rebrand LSI chips. Both controllers are very good though.

 

If you have 8 hard drives spaces, then yeah, I would say get the RAID controller. Keep in mind your PC will get even heavier (eight hard drives do add quite a bit of weight to the system)

[JWMutant]

I would say get a SAS controller. I have one in my NAS and get full SATA 3 speed off it.

 

If your looking for a cheaper option PM me as I may have a 2 port IBM SAS controller with BB.

[ALwin]

snip

 

Initially I was thinking an external USB only or USB + Thunderbolt enclosure so that I can connect it to both my Windows workstation and Macbook Pro.  But then I figured I only use the Macbook for editing when I am mobile, in which case I can just get a portable 2TB or larger capacity drive with copies of my current files.

 

As for the NAS solution, I am hesitating because I don't have much room left in my apartment* and I don't want to be running cables.

 

So I guess I will look into a PCI express RAID controller, as I do have room to add more drives in my case.  I am thinking of adding either 2x 6TB drives in RAID 1 or 6x 3TB drives in RAID 6.

 

*Small 1 bedroom apartment, living room is already low on space because of camera and lighting equipment, tons of books, computer/editing desk and scuba diving gear.

[ALwin]

Which of these are good controllers?  Or more like which one should I get?

http://www.toppreise.ch/index.php?k3=1408&offset=0&manu=1877&filter=&o=pd

[scottyseng]

Which of these are good controllers?  Or more like which one should I get?

http://www.toppreise.ch/index.php?k3=1408&offset=0&manu=1877&filter=&o=pd

 

The LSI 9361-8i is a good card. However, I don't know the language...you need to find the battery backup for the card, which is the CacheVault Flash Module (LSICVM02) Includes CVFM04, CVPM02, mounting clip, and cable

 

http://www.toppreise.ch/prod_393250.html

 

Please keep in mind with the LSI cards, you cannot expand the RAID array when it's built, so buy all of your hard drives upfront and decide on the RAID config before hand.

 

You will also need two SAS 12Gb/s to 4 SATA 6Gb/s breakout cables.

 

Yeah, the thunderbolt (if you have it) is kind of limited (Only thunderbolt things can connect to it). NAS is more open, but you're space constrained (My server chassis is larger than my case too)

[ALwin]

snip

 

Ok thank you for your help.

 

Here is what I will do, as I need to expand my onsite backup/archive storage anyway:

• Disconnect the optical drive and eSata port on my case which will free up 2 SATA ports on my logic board
• Connect a pair of 4-6TB WD Black drives in RAID 1 internally
• And either buy a Synology 12-16 bay NAS system  or
• Build my own network connected file server with a RAID and 10Gbe setup later
• This will let me migrate away from using multiple WD My Book DUO USB RAID 1 drives for onsite backup and I can move those drives to my offsite storage, this will actually free up some room for me to place a large multi-bay storage system

[scottyseng]

-snip-

 

Sounds like a solid plan. I would push towards building your own server because I don't know of many prebuilt NASes that have 10Gb Ethernet on them.

 

Good luck with the move though.

[ALwin]

Sounds like a solid plan. I would push towards building your own server because I don't know of many prebuilt NASes that have 10Gb Ethernet on them.

 

Good luck with the move though.

 

Yeah I've been relying on off the shelf external drives for storage.  I have a few WD DUOs, an older Seagate 4TB NAS, and a LACIE 12TB USB RAID box.  Held off on getting/building a large capacity (20TB+) storage because of budget, space and time.  Now that I am starting to work with 4K footage, I need to streamline and consolidate everything.

 

Synology offers RAID boxes with the possibility to add your own 10Gbe NIC, while I would like to build my own system I also want to be able to get fast support if something goes wrong.  Will decide which one I go with in December.

 

https://www.synology.com/en-us/products/DS3615xs

I'm considering a system like this with my own selection of WD or HGST HDDs.  But thanks for recommending a RAID controller, I still have yet to decide whether to go with this or build my own.  But now I am better prepared.

 

My new setup is going to be:

 

Cameras --> (Macbook+Ext Drive or PC Workstation+Int Drive) --> Multiple Single onsite backup devices --> Offsite backup and Amazon Glacier.

[scottyseng]

-snip-

 

Yeah, surprised you haven't driven yourself crazy having to navigate through so many external drives.

 

I would definitely say weigh the costs and labor for building a NAS versus buying one. Also, keep in mind that server chassis (4U / 20 or 24 drive) are very large. I had no idea what I had bought when I got my SuperMicro 24 bay 4U chassis. It really is larger than my PC. I'd imagine a Synology being much smaller but less expandable.

 

I think the major costs for building your own server would be the chassis (I think a Norco 20 bay is $33) and the RAID card (Usually $500-700). Windows is pretty easy to set up as the storage OS (I don't know much Linux so I didn't go the FreeNAS route).

 

If you don't have one already, I would recommend getting a UPS battery back up for the server as well.

[ALwin]

Yeah, surprised you haven't driven yourself crazy having to navigate through so many external drives.

 

Why do you think I am looking into a solution now... because it's driving me crazy having to connect, disconnect all those drives.  Even though I organize things by year, there are times I can't recall when a particular footage was taken and have to find it.  Recently I started keeping a spreadsheet with an index of projects, footage and locations and on which drives they are stored.

 

Ok if I decide to build my own, I will contact you about choosing some good hardware such as logic board, memory, processor, etc.

 

Synology expands with external boxes of similar sizes.  Sort of like daisy chaining, up to 2-3 external boxes.

[bobhays]

Why do you think I am looking into a solution now... because it's driving me crazy having to connect, disconnect all those drives. Even though I organize things by year, there are times I can't recall when a particular footage was taken and have to find it. Recently I started keeping a spreadsheet with an index of projects, footage and locations and on which drives they are stored.

Ok if I decide to build my own, I will contact you about choosing some good hardware such as logic board, memory, processor, etc.

Synology expands with external boxes of similar sizes. Sort of like daisy chaining, up to 2-3 external boxes.

FreeNas is a great option. You won't even have to buy a RAID card for it which will save a lot of money. I haven't done it myself but the setup looks very straightforward.