Hardware Config Questions

[skimmilk5]

ADMIN: if this belongs in a different category, like photography, please move

Here's the deal. I'm a HUGE photography and videography nerd who insists that everything's better in RAW because I get more flexibility in post. The downside? Massive file sizes and filled drives.

Currently, I have a DAS solution within my computer, with two refurbished Hitachi 4TB drives (don't worry, I have an offline backup regularly made), mirrored in Windows Storage Spaces, for a total of 3.17TB after formatting. As you can imagine, my reason for using refurbished units is because of budget constraints. I can't afford new large capacity drives, like Seagate's enterprise capacity 10TB helium-filled drives or WD's Gold datacenter 10TB helium-filled drives, though that'd be awesome to have. The storage array is completely full, with no where to put anything else. With several large upcoming projects, including some jobs for major league sports teams (100-150GB per session), I'm worried about running out of space to put anything else. Here's where you and your (professional) input comes in.

Would it be better to continue expanding in a DAS environment with larger capacity drives (I only have two 3.5in drive bays)? Which drives should I use? Would it be better for me to build an external DAS box using an older motherboard and SAS expander card?

OR would it be better for me to switch over to a NAS environment with larger capacity drives? Again, which drives should I use? What software operating system would you use? What's the reliability in your opinion?

OR should I start getting archive drives and storing material offline, only pulling those drives and installing them into an external interface when needed? Again, which drives would you recommend for offline storage to minimize data rot? How many drives would you mirror to protect from a frozen drive failure?

Is there a better solution that you can think of? PLEASE let me know as soon as possible!

[Psychopath]

First off, totally agree with keeping RAW files 

But TL;DR is that it really all comes down to how much you are willing/able to spend.

But IMO I would definitely move to a NAS solution as it would be much more flexible, e.g. Access data from anywhere, future expandability, always have access to data, built in redundancy (if done correctly) and much more.

The solution I would suggest is FreeNAS, partly as its all I have ever really used (in home and business applications) as well as it always being 'pretty' rock solid (when I don't f the config up - user error) as well as it having much more features than just a NAS, and well, its free and open source; always a plus. You can also scale it up and down to meet your requirements and budget.

I would definitely shy away from the archive drive option as this seems like a lot of effort for a probable minimum cost saving.

 

If you do have a budget in mind for this, it would help in advising you the best route to take much more. FreeNAS can run on pretty much anything but does benefit in some ways from more expensive hardware, but if you want a slightly more expensive but 'out of the box' soloution a NAS array from vendors like Synology, Qnap, Netgear, WD etc. could be the better option for you.

 

Let us know your thoughts,  hope I can be of help and make some sense in my (probably) incoherent rambling above.

[Pixel5]

you will have to invest some money to make this work and the big question is what do you want from the system and how much are you willing to spend?

 

if you want the easy plug and play solution that just works get yourself a synology system possibly 4bay and start with two 10TB drives so you have spare drive bays to expand later.

 

even if you build something by yourself you will be spending about 800-1000 bucks total for a working PC plus two high capacity drives.

[NelizMastr]

Throwing more harddrives at it isn't really the way to go as you'll run into the physical limit your PC can handle. Internal storage will always be faster than networked storage, though, unless you splurge out on 10GbE equipment. 

 

In terms of scalability, a dedicated NAS with large capacity drives is the most cost effective. Whether this is a prebuilt NAS (Synology, Thecus, QNAP) or something you cooked up yourself isn't too relevant. If it's just basic storage, you'd be fine with a Celeron or Pentium and a few gigs of unbuffered ECC memory. Boards with embedded CPUs are an easy solution for this. ASrock and Supermicro have some offerings in this regard.

 

Examples:

 

Supermicro X10SBA

Supermicro X11SAA

ASRock D1521D4I

 

Add memory, a PSU, a flash drive for the OS and a case, and just chuck a bunch of drives in there. 

[skimmilk5]

On 5/22/2018 at 5:38 AM, Pixel5 said:

you will have to invest some money to make this work and the big question is what do you want from the system and how much are you willing to spend?

 

if you want the easy plug and play solution that just works get yourself a synology system possibly 4bay and start with two 10TB drives so you have spare drive bays to expand later.

 

even if you build something by yourself you will be spending about 800-1000 bucks total for a working PC plus two high capacity drives.

Follow-up question: what do you think about getting a used HP Proliant or Dell PowerEdge server from eBay and stocking it with new drives?

[skimmilk5]

On 5/21/2018 at 8:34 PM, Psychopath said:

First off, totally agree with keeping RAW files 

But TL;DR is that it really all comes down to how much you are willing/able to spend.

But IMO I would definitely move to a NAS solution as it would be much more flexible, e.g. Access data from anywhere, future expandability, always have access to data, built in redundancy (if done correctly) and much more.

The solution I would suggest is FreeNAS, partly as its all I have ever really used (in home and business applications) as well as it always being 'pretty' rock solid (when I don't f the config up - user error) as well as it having much more features than just a NAS, and well, its free and open source; always a plus. You can also scale it up and down to meet your requirements and budget.

I would definitely shy away from the archive drive option as this seems like a lot of effort for a probable minimum cost saving.

 

If you do have a budget in mind for this, it would help in advising you the best route to take much more. FreeNAS can run on pretty much anything but does benefit in some ways from more expensive hardware, but if you want a slightly more expensive but 'out of the box' soloution a NAS array from vendors like Synology, Qnap, Netgear, WD etc. could be the better option for you.

 

Let us know your thoughts,  hope I can be of help and make some sense in my (probably) incoherent rambling above.

I'm pretty familiar with FreeNAS (I've also set up a few FreeNAS boxes too! Great platform! ), so I'm not too worried about setting it up, but what do you think about getting a platform like a Dell PowerEdge or HP Proliant from eBay and kitting it out with new drives?

[Pixel5]

2 minutes ago, kimsejin5 said:

Follow-up question: what do you think about getting a used HP Proliant or Dell PowerEdge server from eBay and stocking it with new drives?

im personally not a big fan of buying old Server hardware because you will often run into problems and will need to search for one with many SATA ports in this case.

 

also for me power consumption is extremely important because electricity is expensive here, having a device that just uses 50 watts more means it will cost me about 130€ a year extra to operate so i rather invest in more efficient hardware and save the money in the long run.

[Psychopath]

On 5/27/2018 at 9:04 PM, kimsejin5 said:

I'm pretty familiar with FreeNAS (I've also set up a few FreeNAS boxes too! Great platform! ), so I'm not too worried about setting it up, but what do you think about getting a platform like a Dell PowerEdge or HP Proliant from eBay and kitting it out with new drives?

That could be worth it, depending you get a good deal, but could also end up costing more down the road through power cost as old (and therefore low cost) servers + gear is usually being sold off from old age and old gear draws significantly more power, especially enterprise/server gear.

 

Personally, I would only go for a PowerEdge or Proliant server if 1) cost was a major factor (or if it works out cheaper when taking power cost into account) and 2) if I intended to use it for other things that multiple cores and a large amount of RAM would benefit from; such as video rendering/encoding, multiple VM's, etc.

[Jarsky]

The biggest issue you have, is that the drives themselves are the largest investment for large storage solutions. 

There are a number of ways you can go, but mirroring is the most expensive since you only have 50% capacity. 

My opinion, would be a custom built with 4 of the most affordable GB/$ drives ( which I think are still 4TB) in a RAID6 (2 drive parity), so that its expandable, but not as costly as mirroring every single drive. Especially if you're making offline backups. 

 

FreeNAS will soon be updated with the ability to add drives to ZFS Raid's and expand the zpool.

 

I'd probably look at something like below, and then add an HBA for more disks as you need to. 

It's also much more power efficient than a PowerEdge. 

 

PCPartPicker part list / Price breakdown by merchant

CPU: AMD - Ryzen 3 2200G 3.5GHz Quad-Core Processor  ($94.99 @ SuperBiiz) 
Motherboard: Gigabyte - GA-AB350M-DS3H Micro ATX AM4 Motherboard  ($59.99 @ Amazon) 
Memory: Team - Elite Plus 16GB (2 x 8GB) DDR4-2400 Memory  ($149.99 @ Newegg) 
Case: Fractal Design - Node 804 MicroATX Mid Tower Case  ($104.99 @ SuperBiiz) 
Power Supply: Corsair - CX (2017) 550W 80+ Bronze Certified ATX Power Supply  ($39.99 @ Newegg) 
Total: $449.95
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2018-05-30 17:46 EDT-0400

 

 

 

[jde3]

To add to this with raw format you might benefit from ZFS compression in FreeNAS.

 

LZ4 is standard but ZSTD is coming really soon. This depends on how well this stuff compresses and I have no idea if it will.