High Performance NAS Build

[Nimrod859]

Hi,

 

i'm planing to build a high performance NAS. It should be hold up to 100TB+ on.

The Case would be the Storinator 60XL. The problem is which Raidcards, CPU, Board, RAM and PSU

did you you guys recommed. Were are planing to use this as an Editing Storage for our Epic RED Cinema Camera.

Network will be 10 GBit/s!

 

I hope you can help me.

 

Thanks from Germany

 

Nimrod

[blu4]

Get an NVME 1.2TB Drive for cache and the rest an WD Greens Linus would be overjoyed to help you with this since he goes overkill on NAS storage too.

[Nimrod859]

This Build is for a FullFeatured Movie Production, so the Data must be safe! There would be two of these Machines only for redundance. I doesn't think the Western Digital Green is the best choice. The System would be the Heart of our Datastorage, so when there is anything faulty we are screwed

 

Operating System will be Freenas

[scottyseng]

Hi,

 

i'm planing to build a high performance NAS. It should be hold up to 100TB+ on.

The Case would be the Storinator 60XL. The problem is which Raidcards, CPU, Board, RAM and PSU

did you you guys recommed. Were are planing to use this as an Editing Storage for our Epic RED Cinema Camera.

Network will be 10 GBit/s!

 

I hope you can help me.

 

Thanks from Germany

 

Nimrod

 

Keep in mind the storinator is more for archival storage than a live full load storage. The density of the hard drives, lack of proper cooling, and lack of vibration protection of any kind makes the storinator a bad idea for live full load use. Not to mention if any drive dies, you are going to have a hell of a time trying to replace it.

 

If you are using FreeNAS, you do not need a RAID card, but a LSI SAS HBA (SAS requirement depends on your chassis). I would recommend a 24 Bay 4U Hot Swap Chassis by Norco or if you have the money, one from SuperMicro (They make a really good chassis).

 

You can get by with a i3 or low end Xeon (Haswell). Probably 16GB of DDR4 ECC. You need some enterprise chipset motherboard (Cxxx series for Intel...depends on CPU socket). If you get a SuperMicro chassis, PSUs are built in. Otherwise, you should get something in the high wattage range (900+W), check out Seasonic or Cooler Master Vseries.

[Nimrod859]

Ok,

 

i would go with the SuperMicro 

SC846BE16-R1K28B

But i've questions in fact of the HDD Configuration. Which Harddrives should i use in fact of Performance and reliability? I want to use an Intel 750 Series SSD as cache. Mainboard would be also supermicro with an 6 Core Xeon and 128gb DDR4 ECC. Because Freenas 1gig RAM for each TB space. is that right?? The Storage must be fast. So 10G/Bit Ethernet is a must have.

 

Greetings

 

Daniel

[scottyseng]

Ok,

 

i would go with the SuperMicro 

SC846BE16-R1K28B

But i've questions in fact of the HDD Configuration. Which Harddrives should i use in fact of Performance and reliability? I want to use an Intel 750 Series SSD as cache. Mainboard would be also supermicro with an 6 Core Xeon and 128gb DDR4 ECC. Because Freenas 1gig RAM for each TB space. is that right?? The Storage must be fast. So 10G/Bit Ethernet is a must have.

 

Greetings

 

Daniel

 

Hmm, you do know that is SAS 2 (6Gb/s) backplane right? Since you're building new, I would heavily try to push you to get the SAS3 (12Gb/s) version here:

http://www.supermicro.com/products/chassis/4U/846/SC846XE2C-R1K23B.cfm

 

It will give you the ability to be future proof with your system / really reduce any bottleneck you may have.

 

As for the HBA, you need something like the LSI SAS 9300-8i Host Bus Adapter.

 

I would recommend the WD RE drives (SAS or SATA). As usual, please stress test all drives you buy at least once (I use the WD lifeguard tool's extended test) before dedicating any of them to a array (I usually extended test once then let the drives little for a week and test again). So far, two of these 4TB RE drives in RAID 0 is about 350MB/s seq red / write. I also have six 4TB Reds (Will be removed when I get the other 22 WD RE drives since Reds can't take the abuse) in RAID 10, and they get 550MB/s seq read/write. Sadly I'm bottlenecked to bits by the 1Gb/s onboard NIC on my PC / the server.

 

I don't know too much about FreeNAS, so I'm not sure if you can use the 750 as a cache for the hard drive array. I know you can use it as a separate drive though (Though it won't have any ZFS protection with just one PCIe drive). 128 GB seems a little overkill to me, but I haven't worked with FreeNAS much so I don't know what the real requirements are.

 

You will need some kind of flash drive as the OS drive or tiny SSD drive.

 

Make sure to get SAS cables as needed (You'll need at least two...if the SAS ports on the HBA are near the rear of the chassis, you will need longer SAS cables than the ones that come with the chassis).

 

As for 10Gb/s Ethernet, look at Solarflare, Mellanox, or Intel NICs. You only need one. However, the question is if your existing structure can handle 10Gb/s (For it to be worth it, you need a 10Gb/s switch and all of the connected PCs need 10Gb/s NICs as well).

 

The cheapest 10Gb/s switch I know of is the 8 port one by Netgear.

 

The SuperMicro Chassis will run quite loud, just so you know.

 

Also, make sure to get a proper UPS battery backup, I recommend a rackmount APC one.

[Nimrod859]

Hi thanks a lot. The Infrastructure get completly new. So Storage and workstations will get 10G/bit Connection the normal office PCs 1Gbit. Storage gets an own Room in a Server Rack, with APC UPS. I would go with Intel NIC. Money is not the Problem

[scottyseng]

Hi thanks a lot. The Infrastructure get completly new. So Storage and workstations will get 10G/bit Connection the normal office PCs 1Gbit. Storage gets an own Room in a Server Rack, with APC UPS. I would go with Intel NIC. Money is not the Problem

 

Yeah, make sure to wire the building with CAT6A or higher since it's all new.

 

Though if money is not a problem...I really would say to go with a server that's all SSDs (The active server) and one server that's all hard drive (The archive / backup server). Or a Intel DC P3608 PCIe NVMe SSD instead of a 750 series.

[Nimrod859]

CAT6A is a must have. The Intel DC SSD looks good. No thats to expensive one server with ssds one server with harddrives

[45Drives]

Keep in mind the storinator is more for archival storage than a live full load storage. The density of the hard drives, lack of proper cooling, and lack of vibration protection of any kind makes the storinator a bad idea for live full load use. Not to mention if any drive dies, you are going to have a hell of a time trying to replace it.

 

If you are using FreeNAS, you do not need a RAID card, but a LSI SAS HBA (SAS requirement depends on your chassis). I would recommend a 24 Bay 4U Hot Swap Chassis by Norco or if you have the money, one from SuperMicro (They make a really good chassis).

 

You can get by with a i3 or low end Xeon (Haswell). Probably 16GB of DDR4 ECC. You need some enterprise chipset motherboard (Cxxx series for Intel...depends on CPU socket). If you get a SuperMicro chassis, PSUs are built in. Otherwise, you should get something in the high wattage range (900+W), check out Seasonic or Cooler Master Vseries.

 

Hi Scotty,

 

Wanted to chime in here on a few points that you brought up to clarify:

 

Vibration - Seagate studied our pod for vibration, and we scored the highest rating possible. You can read the full report here.

 

Thermal properties - It's a pretty big misconception that our pods get exceptionally hot and lack proper cooling. You can check out the full thermal analysis we've done here. 

 

Replacing drives - We've made it easier than ever to replace drives by making our chassis lid tool-less. So now you can slide the pod from the rack, remove the top cover and vibration dampening covers with no tools required. Drives can be hot-swapped and any degraded array can be rebuilt without shutting down the unit. 

 

Archival storage versus live full-load storage - While the Storinator is kick-ass for archival storage, it performs excellently for live, data-intensive applications (especially when enterprise-class drives are used). With 100TB of usable space of 6TB drives in a RAID 10 (34 drives in total), Nimrod859 would have excellent speeds that exceed far more than archival storage. We have information on our wiki that describes raid levels and tested speeds. Feel free to take a look at our graphs for a full breakdown on achievable speeds.

 

PSUs - Our PSU is a server-grade 950W redundant 2+1 Supply that has 1500W of active power on startup. Our product team has gotten lots of positive feedback on this from our customers' deployments in the field.

 

HBA - Rocket 750 is standard on Storinators, but the LSI is available (and popular) with customers as well. 

 

We appreciate any and all feedback that we receive from the community. So if you have any questions for our product team or R&D folks, let me know.

[45Drives]

This Build is for a FullFeatured Movie Production, so the Data must be safe! There would be two of these Machines only for redundance. I doesn't think the Western Digital Green is the best choice. The System would be the Heart of our Datastorage, so when there is anything faulty we are screwed

 

Operating System will be Freenas

 

Thanks for checking out the Storinator!

 

Based on the info you supplied, it sounds like it would be an excellent fit for your needs. As you probably know, we offer FreeNAS, as well as Debian, Ubuntu, CentOS and Rockstor (we have quite a few customers running Windows as well). FreeNAS is extremely popular with good reason - it's established and works really well.

 

Re: hard drives, we don't recommend Western Digital green (more info here on our wiki). Scottyseng recommended WD Re drives - we'd echo this recommendation (in fact, Brett from our R&D team did some power-draw testing using them in a blog post, which you can read here). We offer both the 4TB and 6TB model, with a five-year warranty. 

 

If you have any additional questions about the XL60, or the set-up we'd recommend for you, just let me know here, or send a DM with your contact info.

[45Drives]

Hi,

 

i'm planing to build a high performance NAS. It should be hold up to 100TB+ on.

The Case would be the Storinator 60XL. The problem is which Raidcards, CPU, Board, RAM and PSU

did you you guys recommed. Were are planing to use this as an Editing Storage for our Epic RED Cinema Camera.

Network will be 10 GBit/s!

 

I hope you can help me.

 

Thanks from Germany

 

Nimrod

 

One more thing I forgot to mention before - if you want to hear from a 45 Drives customer that's using Storinators for video post-production, let me know. We have a few that would be happy to give a referral, including one based out of North Carolina that does reality TV productions and commercial contracts. 

[scottyseng]

-snip-

 

Ah, sorry, I was comparing you guys to the backblaze pod (I thought the storinator and that were the same). Yeah, I was looking at the reports you linked and was like, that doesn't look like the same pod I saw on backblaze.  

 

Alright, with that cleared up, the storinator looks a lot better.

[45Drives]

Ah, sorry, I was comparing you guys to the backblaze pod (I thought the storinator and that were the same). Yeah, I was looking at the reports you linked and was like, that doesn't look like the same pod I saw on backblaze.  

 

Alright, with that cleared up, the storinator looks a lot better.

 

All good! It's a bit confusing - we were one and the same for awhile in terms of pod design, when we first started 45 Drives a few years ago. However, over the past year especially, through R&D development and a lot of feedback from our customers, we've tweaked the design and its components from the original Backblaze design (more info on that here if you're curious). Now we find the performance is not only more reliable, but also able to withstand really data-heavy applications.

 

Thanks for checking out the links!