Data storage array solution

[ALwin]

Hello,

I would like to seek some tips on assembling a mass data storage solution and a suitable RAID setup.

Background:
My office uses Windows desktops as our primary machines for normal office use, however we also have 5 Apple desktop machines for media editing. Totaling around 30 machines. Currently we are using some USB RAID storage and swapping drives with media files for our work.

We are looking to build a shared data storage solution, approximately around 50-100TB, with fast access via either copper or optical LAN. The main machines that will be accessing the storage will be the 5 Apple machines using Final Cut or Adobe products, and perhaps at most 10 of the various windows machines connected at any one time if they are even given access.

The primary purpose of the storage array will be for the video editors to have access to the media files, edit directly on the storage array and collaborate.

Our main requirements are of course,
Speed
Reliability
Some sort of data safety (I am considering RAID 6 for the storage array, our office has a contract with an external company for offsite backup of all our data)

So what components would be recommanded, and in what sizes (HDDs, SSDS, memory)?

(Written on my phone, so if I forgot to mention something or need to clarify anything, I will edit later.)

 

EDIT:

Examples of hardware I am considering.  I wont mention specific model numbers if I am not sure if of their availability.

 

Motherboard: Asus Px79-E WS

Processor: Xeon E5 2620 v2

Memory: As much as the motherboard and cpu can support, not certain if I want to go ECC or non-ECC

GPU: whatever is the bare minimum necessary to connect a display.

Recommendation for a hardware RAID controller?

Power supply: 600w PSU?

Storage bay: ?

HDD: HGST NAS drives, 2, 3 or 4TB capacity.

SSD: 1x 120GB Samsung Evo

[Bittenfleax]

@ALwin Why not get a NAS? They are expensive though, especially for a 50-100TB one, like I have never seen one before.

 

For safety, you could use an online backup solution?!

[ALwin]

@ALwin Why not get a NAS? They are expensive though, especially for a 50-100TB one, like I have never seen one before.

For safety, you could use an online backup solution?!

Hi Bittenfleax, we are not looking for a backup solution, we are looking for something we can collaborate and work from directly. Yes we could purchase a solution from a company, however we would like to explore the cost/benefits of a custom solution first.

Additionally we have space on our video editing tools rack to install a few multi-bay storage racks.

[Bittenfleax]

Hi Bittenfleax, we are not looking for a backup solution, we are looking for something we can collaborate and work from directly. Yes we could purchase a solution from a company, however we would like to explore the cost/benefits of a custom solution first.

Additionally we have space on our video editing tools rack to install a few multi-bay storage racks.

Hi, getting a NAS would provide that. It will allow storage across your entire network. With high-end NAS's, you could reach high speeds. However, there is a bottleneck with the drive speed if you are looking for a lot of performance. 

 

However, I don't think that this would be a problem due to only a small number of PC's being on your network.

 

PS. The 3rd party company solution, was just for backing up the NAS.

[Bittenfleax]

@ALwin If you wanted to go balls to the wall and had loads of money, you could get a SAN that provide fibre connection or Gigabit Ethernet for your network. But you are talking about £10,000 +.

[ALwin]

As I mentioned we already have an external company handling offsite data backup for everything. I just want to have something like RAID 6 for the data storage array in case of drive failure.

Also I read that NAS is slower for multi user access compared to a SAN. We want our editors to be able to work on files directly on the storage setup without having to download/upload files and projects to/from local storage.

[Bittenfleax]

-snip-

Sorry, misread it.

 

I think SAN would be the way to go, but an expensive one. SAN's can cost thousands. You can get small (non-rack mounted) SAN's for £800ish.

[ALwin]

@ALwin If you wanted to go balls to the wall and had loads of money, you could get a SAN that provide fibre connection or Gigabit Ethernet for your network. But you are talking about £10,000 +.

Hence I am here asking for some tips, and to get an idea of hardware, to submit a plan of action. Our head video producer is already asking for $50,000, which includes buying a couple of new cameras and accessories.

We need to replace our aging gear, consolidate 10+ years of media files from various storages, including old tape drives,we are already using and have ample storage or expansion ability for the future.

If a custom SAN can be built with off the shelf parts for $10,000 or less it may beat buying one from a vendor and we can configure it as we want.

[Bittenfleax]

-snip-

 Ahhh right, if money is not really a problem, you could go with HP (or DELL). But HP have their own SAN software which is great. The HP Store Easy 1000 and up series is great and is in your price range. (rather old though) You should look into that.

[dalekphalm]

@ALwin Who is going to be managing this File Server? You? An in house IT?

 

Also, what kind of technical degree of knowledge are we talking about? Ideally, you'd use ZFS File System on either FreeBSD (or variant) or Linux (Ubuntu, Debian, etc).

 

Parts:

 

Case - Norco 16 bay case

http://www.amazon.com/NORCO-16-Bays-Server-Chassis-RPC-3216/dp/B00BQY38CG/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094702&sr=1-1&keywords=norco+rpc-3216

 

Drives

EITHER

16 x WD Red Pro 4TB - 64TB of RAW Storage

http://www.amazon.com/Red-Pro-4TB-Hard-Drive/dp/B00LO3KRM8/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438093404&sr=1-1&keywords=wd+red+pro+6tb

 

OR

16 x WD SE 6TB  -  96TB of RAW Storage

http://www.amazon.com/Western-Digital-Enterprise-3-5-Inch-WD6005FRPZ/dp/B00UHJJG5O/ref=sr_1_2?s=electronics&ie=UTF8&qid=1438093383&sr=1-2&keywords=wd+se+6tb

 

Depending on if you want the lower range or higher range of Data Storage.

 

You'll also want:

RAID Card

LSI 9260-16i

http://www.amazon.com/Megaraid-9260-16I-Single-16-PORT-512MB/dp/B003VP2EJQ/ref=sr_1_3?s=electronics&ie=UTF8&qid=1438094762&sr=1-3&keywords=lsi+16+port+raid+card

w/ battery backup

http://www.amazon.com/Megaraid-9260-9280-Raid-LSLIBBU08/dp/B004L62KIY/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094892&sr=1-1&keywords=lsi+bbu08

 

OR

 

HBA

http://www.amazon.com/16-PORT-Int-6GB-Sata-Pcie/dp/B003UNP05O/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094930&sr=1-1&keywords=lsi+9201-16i

 

Whether you choose an HBA (Host Bus Adapter) or a RAID Card depends on the OS you choose, and whether you want to go Hardware RAID or a software RAID solution.

 

If you decide to use ZFS or mdadm, then definitely use the HBA. If you go with Windows Server, you may want to consider the hardware RAID card. ZFS is the superior solution though.

 

For the rest of your hardware, it depends on load. I would suggest a workstation/server motherboard with a low-mid tier Xeon processor, with ECC RAM, 1GB per TB is usually recommended for ZFS, but is not strictly necessary. Basically get as much RAM as you can afford.

 

For the OS Drive, I recommend just a simple 120 GB SSD (I'd get a pro level one like Samsung 850 Pro, or Intel SSD).

 

No need for a video card, unless the CPU and motherboard doesn't support onboard video. If so, buy the cheapest video card you can find.

 

No need for an optical drive either.

 

Make sure the Motherboard has at least Gigabit Ethernet built in. Having 10Gbit Ethernet would be even better. You could add a 10GigE NIC in a PCIe slot if needed.

 

Make sure to pair that with a 10GigE Switch, and if possible, 10GigE NIC's on each of the Macs.

[dalekphalm]

@ALwin Why not get a NAS? They are expensive though, especially for a 50-100TB one, like I have never seen one before.

 

For safety, you could use an online backup solution?!

For the storage he is looking for (50-100TB) a NAS is likely not a good solution. The chassis alone (With no HDD's) is going to cost multiple thousand dollars.

 

Eg:

http://www.amazon.com/Synology-America-Station-Attached-DS2415/dp/B00SWEM4DW/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438095479&sr=1-1&keywords=%09DS2415%2B

12-Bay Synology NAS - $1700 USD with no drives.

 

You could take that $1700 and put it towards building a proper, expandable File Server, and come out way ahead. Plus most NAS's are limited to Gigabit Ethernet.

[Bittenfleax]

-snip-

That's why I said SAN after  :lol:

[dalekphalm]

That's why I said SAN after  :lol:

A full blown SAN is likely unnecessary for their needs. A mid-to-high end File Server is likely good enough. Although the lines between File Server/SAN/NAS can get blurry at times.

[ALwin]

@ALwin Who is going to be managing this File Server? You? An in house IT?

 

Also, what kind of technical degree of knowledge are we talking about? Ideally, you'd use ZFS File System on either FreeBSD (or variant) or Linux (Ubuntu, Debian, etc).

 

Parts:

 

Case - Norco 16 bay case

http://www.amazon.com/NORCO-16-Bays-Server-Chassis-RPC-3216/dp/B00BQY38CG/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094702&sr=1-1&keywords=norco+rpc-3216

 

Drives

EITHER

16 x WD Red Pro 4TB - 64TB of RAW Storage

http://www.amazon.com/Red-Pro-4TB-Hard-Drive/dp/B00LO3KRM8/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438093404&sr=1-1&keywords=wd+red+pro+6tb

 

OR

16 x WD SE 6TB  -  96TB of RAW Storage

http://www.amazon.com/Western-Digital-Enterprise-3-5-Inch-WD6005FRPZ/dp/B00UHJJG5O/ref=sr_1_2?s=electronics&ie=UTF8&qid=1438093383&sr=1-2&keywords=wd+se+6tb

 

Depending on if you want the lower range or higher range of Data Storage.

 

You'll also want:

RAID Card

LSI 9260-16i

http://www.amazon.com/Megaraid-9260-16I-Single-16-PORT-512MB/dp/B003VP2EJQ/ref=sr_1_3?s=electronics&ie=UTF8&qid=1438094762&sr=1-3&keywords=lsi+16+port+raid+card

w/ battery backup

http://www.amazon.com/Megaraid-9260-9280-Raid-LSLIBBU08/dp/B004L62KIY/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094892&sr=1-1&keywords=lsi+bbu08

 

OR

 

HBA

http://www.amazon.com/16-PORT-Int-6GB-Sata-Pcie/dp/B003UNP05O/ref=sr_1_1?s=electronics&ie=UTF8&qid=1438094930&sr=1-1&keywords=lsi+9201-16i

 

Whether you choose an HBA (Host Bus Adapter) or a RAID Card depends on the OS you choose, and whether you want to go Hardware RAID or a software RAID solution.

 

If you decide to use ZFS or mdadm, then definitely use the HBA. If you go with Windows Server, you may want to consider the hardware RAID card. ZFS is the superior solution though.

 

For the rest of your hardware, it depends on load. I would suggest a workstation/server motherboard with a low-mid tier Xeon processor, with ECC RAM, 1GB per TB is usually recommended for ZFS, but is not strictly necessary. Basically get as much RAM as you can afford.

 

For the OS Drive, I recommend just a simple 120 GB SSD (I'd get a pro level one like Samsung 850 Pro, or Intel SSD).

 

No need for a video card, unless the CPU and motherboard doesn't support onboard video. If so, buy the cheapest video card you can find.

 

No need for an optical drive either.

Thank you, you've pointed out some stuff I have been considering. When I get home I will post some of the hardware components I am considering.

But certainly I am considering a x79 series workstation motherboard from Asus, a low cpu clock Xeon processor with more cores instead of a higher cpu clock processor with fewer cores. Low end video card if there isn't any integrated graphics. Hardware RAID.

Depending on the HDD capacity a multi-bay container supports and the rack size, about 24 x 4TB HDDs or 40+ x 2TB HDDs. Certainly a small capacity SSD for the OS.

[Bittenfleax]

A full blown SAN is likely unnecessary for their needs. A mid-to-high end File Server is likely good enough. Although the lines between File Server/SAN/NAS can get blurry at times.

Yeah, I guess they are more large enterprise (or if you are loaded and don't care what you do with your money :lol:  :lol:) 

[dalekphalm]

Thank you, you've pointed out some stuff I have been considering. When I get home I will post some of the hardware components I am considering.

But certainly I am considering a x79 series workstation motherboard from Asus, a low cpu clock Xeon processor with more cores instead of a higher cpu clock processor with fewer cores. Low end video card if there isn't any integrated graphics. Hardware RAID.

Depending on the HDD capacity a multi-bay container supports and the rack size, about 24 x 4TB HDDs or 40+ x 2TB HDDs. Certainly a small capacity SSD for the OS.

Keep in mind that with that number of HDD's, the choice of HDD becomes very important. If you're going to be using 24 or 40+ HDD's, you're now in Enterprise drive territory, since they are rated for that amount of heat and vibration caused by being so close to so many drives.

 

The drives that I selected above are based on a number of criteria:

 

The WD Red Pro drives (Available in up to 4TB capacity) are rated in up to 16 bay enclosures. You could probably get away with putting 24 of them in a good quality file server case, but you'll have to accept the risk that you're using them out of spec.

 

The WD SE drives that I linked to, are available in up to 6TB Capacity, and are rated for enclosures that surpass 16 bays (but can also be used in smaller quantities) - these are true Enterprise grade drives, which the price reflects. The SE drives do come in 4TB Capacity, so you could do a 24 x 4TB setup if you wanted. I simply selected the 6TB ones to maximize your price per GB, but you'll want to do a comprehensive price comparison of the different varieties.

 

Whatever you do, DO NOT use consumer drives in this setup. At all. Ever. That's just a disaster waiting to happen

 

x79 + low clocked, higher core Xeon is a good match. If you go with Hardware RAID, then the amount of RAM matters less, but I would still go with ECC just to be on the safe side.

 

If you go Hardware RAID, then the host OS matters less, but I would personally suggest using Windows Server 2012 R2 Standard. I would normally suggest Essentials, but that version has a 25 user limit for the license. Since you have around 30 users, you'll need to use Standard edition.

 

This will allow you to setup proper network shares, and even configure user/group permissions for access to various shared folders as needed.

[ALwin]

Ok for the amount of HDDs, the amount installed will be multiplied by the capacity, so if I can use fewer number of drives with higher capacity, I would certainly go for it.  I was calculating the amount of drives based on the fact that it would be great to try and get as close to 100TB as possible.  Certainly 16-20 drives with 4TB capacity sounds like a reasonable amount.  I am considering WD Red drives or HGST NAS drives.

 

Apart from OS and hardware considerations you have mentioned, another consideration is that both Windows and OSX workstation users need to be able to read/write to the storage array.

[dalekphalm]

Ok for the amount of HDDs, the amount installed will be multiplied by the capacity, so if I can use fewer number of drives with higher capacity, I would certainly go for it.  I was calculating the amount of drives based on the fact that it would be great to try and get as close to 100TB as possible.  Certainly 16-20 drives with 4TB capacity sounds like a reasonable amount.  I am considering WD Red drives or HGST NAS drives.

 

Apart from OS and hardware considerations you have mentioned, another consideration is that both Windows and OSX workstation users need to be able to read/write to the storage array.

Windows Server can facilitate both OS X and Windows users.

 

I would highly suggest not using WD Red drives for anything remotely approaching the capacity of 16-20 drives. WD Red Pro drives, on the other hand, should be fine.

 

HGST NAS drives should be fine in those capacities. They come in up to 6TB as well.

 

If you wanted to reach as close as possible to 100TB, you have several options.

 

24 x 4TB = 96TB

20 x 5TB = 100TB

16 x 6TB = 96TB

 

Red Pro are available in up to 4TB, HGST NAS are available in 4, 5, or 6TB, and WD SE are available in 4 or 6TB as well.

 

I would recommend less drives that are larger. Especially if you plan on sticking them all into a RAID 6 array.

 

Having a 16 disk RAID 6 array is far better than a 24 disk RAID 6 array. Especially for rebuild times, but also for reliability.

[ALwin]

Just going to do a rough calculation of total cost of hardware based on some of the parts you recommended, rounding up the prices a bit:

 

1. 16 Bay Chasis: $400

2. x79 motherboard with LGA 2011 socket: $350

3. intel Xeon 6core LGA 2011 socket: $400

4. 64GB of ECC memory (8x8GB): Approximately $800? (I couldn't find reliable prices for 8GB or 8x8GB kit modules so I assumed $100/8GB module)

5. 16x 4TB NAS grade HDD: Approximately $200-300 each depending on whether they are SATA or SAS: $3200-$4800

6. 120GB SSD: $60

7. 800W PSU: $200

8. Hardware RAID solution: $500-$1000

9. Cheap GPU: $50

10. Quad port Gigabit Ethernet card: $100?

Total: Roughly $8160

 

Am I missing anything?

 

 

OS and other software, the office's IT dept. can get licenses for us.

[dalekphalm]

Just going to do a rough calculation of total cost of hardware based on some of the parts you recommended, rounding up the prices a bit:

 

1. 16 Bay Chasis: $400

2. x79 motherboard with LGA 2011 socket: $350

3. intel Xeon 6core LGA 2011 socket: $400

4. 64GB of ECC memory (8x8GB): Approximately $800? (I couldn't find reliable prices for 8GB or 8x8GB kit modules so I assumed $100/8GB module)

5. 16x 4TB NAS grade HDD: Approximately $200-300 each depending on whether they are SATA or SAS: $3200-$4800

6. 120GB SSD: $60

7. 800W PSU: $200

8. Hardware RAID solution: $500-$1000

9. Cheap GPU: $50

10. Gigabit Ethernet card: $100?

Total: Roughly $8160

 

Am I missing anything?

 

 

OS and other software, the office's IT dept. can get licenses for us.

Sounds good. The only other thing I might recommend is either:

4-port 1GigE NIC or

1-Port 10GigE NIC

 

The 4-port NIC you could use Teaming (Or SMB 3.0 - or both) to allow 4 concurrent users to each use a full 1GigE pipe, or even potentially up to 4 Gbps if one user uses all 4 pipes - this last bit would only work if that user also had multiple Gigabit connections.

 

With the 10GigE NIC you could have other users using 10GigE NIC's to have a full blown 10GigE network, or have the 10GigE NIC in the server simply plug into a 1GigE switch that has 10GigE ports for the backbone (Essentially the same idea as the 4-port setup but without multiple cables).

 

Having a 10GigE backbone capable switch means that all your users are connected via Gigabit, but 10 users can link up simultaneously to your File Server without the File Server's NIC becoming bottlenecked. The Switch would handle all the load balancing, etc. You'd need to make sure the switch is capable of these functions.

 

Otherwise, a single 1GigE connection is gonna bottleneck fast with even just 5 graphics users hitting it at once - especially if you're using the File Server as an active workspace as well.

 

Alternative connection methods include using straight up Fibre, Infiniband, etc, but 10GigE should save on overall costs and equipment.

[ALwin]

You're right, I was checking the price for a 4 port ethernet card on amazon and thought the ones they listed were all 10 gigabit ports.  I guess the cost will go up a bit for having a multiple port 10 gigabit ethernet card.

 

Something like this I guess.

http://www.amazon.com/Intel-E10G42BT-Gigabit-Ethernet-PCI-Express/dp/B0096F8C3W/ref=sr_1_12?s=electronics&ie=UTF8&qid=1438102562&sr=1-12&keywords=10+gigabit+ethernet+card&refinements=p_89%3AIntel

[dalekphalm]

You're right, I was checking the price for a 4 port ethernet card on amazon and thought the ones they listed were all 10 gigabit ports.  I guess the cost will go up a bit for having a multiple port 10 gigabit ethernet card.

 

Something like this I guess.

http://www.amazon.com/Intel-E10G42BT-Gigabit-Ethernet-PCI-Express/dp/B0096F8C3W/ref=sr_1_12?s=electronics&ie=UTF8&qid=1438102562&sr=1-12&keywords=10+gigabit+ethernet+card&refinements=p_89%3AIntel

You could get away with a Single Port 10GigE NIC, for the Server, and definitely for any clients that may want full speed.

 

If you really wanted a multi-port 10GigE NIC for the server, then the one you linked is not a bad choice.

[brob]

Upgrade network infrastructure as much as possible. The best performing storage system will never be faster than the slowest part of the network connecting the user(s).

 

An o/s like FreeNAS should be considered.

 

On the high end, one could build a system something like:

• Dual Xeon motherboard like the Asus ZPE-D16 (quad-port Intel Ethernet controller),
• Two Xeon E5-26xx
• Two compatible cpu cooler
• 64GB - 512GB ECC memory (32GB - 256GB per cpu, quad channel), more means better caching
• System ssd (can be mirrored)
• High performance ssd for additional read caching (1TB+)
• 7200 SAS or SATA hdd designed for use in RAID arrays, the number depends on RAID scheme used

(FreeNAS implements RAID so hardware RAID controllers are not needed or desirable.)

[dalekphalm]

Upgrade network infrastructure as much as possible. The best performing storage system will never be faster than the slowest part of the network connecting the user(s).

 

An o/s like FreeNAS should be considered.

 

On the high end, one could build a system something like:

• Dual Xeon motherboard like the Asus ZPE-D16 (quad-port Intel Ethernet controller),
• Two Xeon E5-26xx
• Two compatible cpu cooler
• 64GB - 512GB ECC memory (32GB - 256GB per cpu, quad channel), more means better caching
• System ssd (can be mirrored)
• High performance ssd for additional read caching (1TB+)
• 7200 SAS or SATA hdd designed for use in RAID arrays, the number depends on RAID scheme used

(FreeNAS implements RAID so hardware RAID controllers are not needed or desirable.)

I agree with network infrastructure. Definitely make sure all switches are minimum 1GigE compatible. And possibly have a 10GigE backbone switch with one or two 10GigE ports with the rest being 1GigE ports for the clients.

 

As for that server. Holy fuck, while yes, that's a possible high end server, but man that's overkill for his needs. We're talking 30 users total, with 5 users doing heavier multimedia storage for editing, content creation, etc.

 

Just because LMG does it because they're getting all or most of their Hardware for free via Sponsorship deals does not mean everyone should follow suit

[ALwin]

The entire building has a minimum 1 gigabit network infrastructure for all 2000+ office spaces.  The unit I work for is only 20-30 people max, with 5 people doing the heavy video editing.  The others in our unit may just need to get some finalized media files to upload or share.

 

The IT department can handle the network infrastructure, but they are slow to meet the high data storage needs of the small multimedia team.  They even shut down the media server that was running for over 15 years (well we can't really blame them, it was using Realmedia) and won't provide a replacement.  But having our own media server is of secondary priority at the moment.

 

Thats why I am here trying to get some tips to see how we can meet our needs.  We don't need the best or top of the line most expensive solution, we just need one that works.  So I doubt we need dual processor motherboards or 512GB memory.  We're not working with 4K videos, our video files are 1080p at best, but we do need a large capacity storage solution to consolidate so many years worth of video files, audio files, image files and still have room for at least several more years before expanding storage space.

If we ever get the budget approved to either build or buy a storage solution, then we will be busy transfering all the files from the various HDDs, RAID bays, optical storage, tape drives, various forms of memory cards, etc.