Building a storage server 80TB

[System Error Message]

with 4TB drives you will need 20 to get 80TB, your case cant fit all. Consider a bigger case to fit more drives, a SAS controller so you can use SAS to sata (some server boards have them integrated).

 

ZFS is nice as long as you use the same drives as different size drives can get complicated. You can have zfs handle raid or mdadm. I suggest either brtfs or zfs.

 

RAID isnt dead, its really great. You can do more such as instead of raid 6 you can use more drives for hashing for more redundancy.

 

For performance reasons, a dedicated controller is helpful, otherwise motherboard SAS with a faster CPU.

 

4U case will fit all the drives you need cheaply.

[oxzhor]

13 hours ago, Mikensan said:

You could do 12 disks per vdev, there isn't any limitation just recommendations.

 

12 x4TB Raidz2 = 36.38TiB, so you'll have ~72TiB usable. 8TB drop isn't terrible for 2 disk failover per vdev.

This is a good option to, i gonna thinking about it

[oxzhor]

6 hours ago, System Error Message said:

with 4TB drives you will need 20 to get 80TB, your case cant fit all. Consider a bigger case to fit more drives, a SAS controller so you can use SAS to sata (some server boards have them integrated).

 

ZFS is nice as long as you use the same drives as different size drives can get complicated. You can have zfs handle raid or mdadm. I suggest either brtfs or zfs.

 

RAID isnt dead, its really great. You can do more such as instead of raid 6 you can use more drives for hashing for more redundancy.

 

For performance reasons, a dedicated controller is helpful, otherwise motherboard SAS with a faster CPU.

 

4U case will fit all the drives you need cheaply.

Yes i want to scale up to a 4U with 24 disks. I am still looking for what for supermicro board i will go in combination with the 3x IBM-M1015

[scottyseng]

5 minutes ago, oxzhor said:

Yes i want to scale up to a 4U with 24 disks. I am still looking for what for supermicro board i will go in combination with the 3x IBM-M1015

If you're getting a SuperMicro chassis, they have LSI SAS expanders on the backplanes. You just need to plug two SAS ports from a SAS HBA card to it. All 24 drives will be available. You do not need three HBA cards for this.

 

I have a LSI MegaRAID 9260-8i powering my NAS, and it is able to drive all 24 drive bays (15 in use at the moment) with two SAS ports.

 

Though as mentioned above, you probably should get a 24 bay 4U chassis.

 

I have a SuperMicro 4U chassis with 15 4TB drives, and in RAID6, I'm getting about 37.5TB in space. I think your 80TB goal is definitely going to be hard to reach with 16 bay for sure (Unless you get 8TB drives...which are pretty pricey at the moment).

[oxzhor]

3 minutes ago, scottyseng said:

If you're getting a SuperMicro chassis, they have LSI SAS expanders on the backplanes. You just need to plug two SAS ports from a SAS HBA card to it. All 24 drives will be available. You do not need three HBA cards for this.

 

I have a LSI MegaRAID 9260-8i powering my NAS, and it is able to drive all 24 drive bays (15 in use at the moment) with two SAS ports.

 

Though as mentioned above, you probably should get a 24 bay 4U chassis.

 

I have a SuperMicro 4U chassis with 15 4TB drives, and in RAID6, I'm getting about 37.5TB in space. I think your 80TB goal is definitely going to be hard to reach with 16 bay for sure (Unless you get 8TB drives...which are pretty pricey at the moment).

Hi Scot, i thinking to get a 24 bay case, i found a cheap one at x-case see https://www.xcase.co.uk/products/x-case-extra-value-rm-424-24-hotswap-bays

[scottyseng]

3 minutes ago, oxzhor said:

Hi Scot, i thinking to get a 24 bay case, i found a cheap one at x-case see https://www.xcase.co.uk/products/x-case-extra-value-rm-424-24-hotswap-bays

Ah, okay, for that case, you would need multiple HBA cards (No expander on the backplane). Or get a single HBA card and a SAS expander.

 

That would work for your needs. Do note though, that if you have SAS 6Gb/s cards, you will need a converter from SAS 12Gb/s ports to SAS 6Gb/s to match the newer connector on that server chassis.

 

SuperMicro server chassis as you probably could tell, cost quite a lot...I actually bought a used one myself and just put my own parts into it.

 

Be prepared as well, these servers get extremely heavy, I usually have to take all of the drives out to attempt to move mine for cleaning.

[oxzhor]

52 minutes ago, scottyseng said:

Ah, okay, for that case, you would need multiple HBA cards (No expander on the backplane). Or get a single HBA card and a SAS expander.

 

That would work for your needs. Do note though, that if you have SAS 6Gb/s cards, you will need a converter from SAS 12Gb/s ports to SAS 6Gb/s to match the newer connector on that server chassis.

 

SuperMicro server chassis as you probably could tell, cost quite a lot...I actually bought a used one myself and just put my own parts into it.

 

Be prepared as well, these servers get extremely heavy, I usually have to take all of the drives out to attempt to move mine for cleaning.

The case look really solid for that price, here a video from it

 

Also an interested solution is a case with a sas backplate expander like you say scot

 

 

[scottyseng]

23 minutes ago, oxzhor said:

-snip-

Yeah, that case looks really good. If you can afford it, I would go with the backplane with the SAS expander built in. It also will let you expand out to another chassis.

[oxzhor]

3 minutes ago, scottyseng said:

Yeah, that case looks really good. If you can afford it, I would go with the backplane with the SAS expander built in. It also will let you expand out to another chassis.

For off-site backups it will be perfect this setup, with running freenas into raidz2.

Any people see complications with my whishes ?

[leadeater]

1 hour ago, scottyseng said:

That would work for your needs. Do note though, that if you have SAS 6Gb/s cards, you will need a converter from SAS 12Gb/s ports to SAS 6Gb/s to match the newer connector on that server chassis.

They look like 6Gb ports to me not 12Gb, specs also say 6Gb backplane.

 

29 minutes ago, oxzhor said:

For off-site backups it will be perfect this setup, with running freenas into raidz2.

Any people see complications with my whishes ?

Nope looks fine to me.

[scottyseng]

10 minutes ago, leadeater said:

They look like 6Gb ports to me not 12Gb, specs also say 6Gb backplane.

Ah, I misread the description (12Gb compatible). That tells me I should probably go to sleep...

[oxzhor]

12 minutes ago, leadeater said:

They look like 6Gb ports to me not 12Gb, specs also say 6Gb backplane.

 

Nope looks fine to me.

There say:

Quote

Backplane

6GB Expander SAS backplanes compatible with SATA & SAS   - 12GB Card Compatible

Also i checked the card he use LSI MegaRAID SAS 9361-8i, 12Gb/s, 1GB Cache

[leadeater]

2 minutes ago, oxzhor said:

There say:

Also i checked the card he use LSI MegaRAID SAS 9361-8i, 12Gb/s, 1GB Cache

Yea was just pointing out the case backplane was 6Gb not 12Gb, the cards you have or are going to get will also be 6Gb. So you'll be wanting to use SFF-8087 to SFF-8087 SAS cables.

 

I actually have an LSI 9361-8i with the 1GB cache upgrade and FastPath + CacheCade, it's a sweet card but anything 12Gb is still rather expensive and you don't need it for FreeNAS.

[oxzhor]

24 minutes ago, leadeater said:

Yea was just pointing out the case backplane was 6Gb not 12Gb, the cards you have or are going to get will also be 6Gb. So you'll be wanting to use SFF-8087 to SFF-8087 SAS cables.

 

I actually have an LSI 9361-8i with the 1GB cache upgrade and FastPath + CacheCade, it's a sweet card but anything 12Gb is still rather expensive and you don't need it for FreeNAS.

Which card you recommend with this case? the LSI 9240-8i?

[leadeater]

4 minutes ago, oxzhor said:

Which card you recommend with this case? the LSI 9240-8i?

For FreeNAS basically any SAS card that has been flashed to IT mode, IBM M1015-IT/LSI 9240-8i-IT, or a true HBA.

 

A RAID card flashed to IT mode is an HBA but there are cards that were HBA's from the start e.g HP H220.

[oxzhor]

3 minutes ago, leadeater said:

For FreeNAS basically any SAS card that has been flashed to IT mode, IBM M1015-IT/LSI 9240-8i-IT, or a true HBA.

This cased give possibilities and save two IBM-M1015 cards, and yes he is more expensive then the one without the sas backplane include.

I will start looking for hardware for this build.

[oxzhor]

Hi All,

 

Ok after the really good advices from the people here i change the system setup:

 

X-Case RM424-EX Pro 24 Bay Hotswap Storage Server Case- Expander Backplane https://www.xcase.co.uk/products/x-case-rm424-ex-pro-24-bay-hotswap-storage-server-case-expander-backplane?variant=9314376069

Supermicro X10SRL-F <- I choose this board so i can add later more memory if needed with freenas.

Intel Xeon E5-2603 v3

16GB Samsung M393A2K40BB1-CRC <- for starting later i will upgrade it with more volume

6x HGST Deskstar NAS 4TB HDN724040ALE640

1x IBM-M1015

Intel 10G Ethernet Server Adapter 10Gbps Dual Port PCI-E X520-DA2

Seasonic G-series 650 watt 80 PLUS® Gold efficiency

OS: Freenas with raid-z2

Edited September 20, 2016 by oxzhor
Change memory adviced by supermicro

[Rison]

If it's just a storage server, you're probably not going to need more than 16G of memory.. that in itself is overkill. Only if your server is production with multiple users constantly using it you would consider upgrading. 

I'm not familiar with that HBA card, so long as it's supported in FreeNAS/BSD you'll be fine for the first 16 drives.

 

As mentioned previously, using raid Z2 you will have to rebuild the array from scratch (basically) in order to expand, so you'll have to put your data somewhere while you do the expansion. 

 

When I read "80TB" array, I really didn't expect 4TB drives, as mentioned 12 drives is prob the most i'd put in one array. Perhaps by the time you go to expand, 10TB drives will be more affordable, who knows.

[Mikensan]

9 minutes ago, Rison said:

If it's just a storage server, you're probably not going to need more than 16G of memory.. that in itself is overkill. Only if your server is production with multiple users constantly using it you would consider upgrading. 

I'm not familiar with that HBA card, so long as it's supported in FreeNAS/BSD you'll be fine for the first 16 drives.

 

As mentioned previously, using raid Z2 you will have to rebuild the array from scratch (basically) in order to expand, so you'll have to put your data somewhere while you do the expansion. 

 

When I read "80TB" array, I really didn't expect 4TB drives, as mentioned 12 drives is prob the most i'd put in one array. Perhaps by the time you go to expand, 10TB drives will be more affordable, who knows.

Destroying the raidz2 vdev to expand only applies if you want to add one or two drives directly to that vdev. Any more than that you'd just create a new vdev and add it to the pool or replace each disk in the vdev.

 

i also thought at first he'd be using 6tb disks but that was when he wanted the 12 bay chassis (or was it 16? Too lazy to go back lol). Think you get better $/gb with 4tb as well as better performance(kind of) with more disks. 

 

With 80TiB 16gb will work but certainly not as well as 32gb. I imagine with storage that big he will be moving some large files, 32gb will come in handy.

[oxzhor]

1 hour ago, Mikensan said:

Destroying the raidz2 vdev to expand only applies if you want to add one or two drives directly to that vdev. Any more than that you'd just create a new vdev and add it to the pool or replace each disk in the vdev.

 

i also thought at first he'd be using 6tb disks but that was when he wanted the 12 bay chassis (or was it 16? Too lazy to go back lol). Think you get better $/gb with 4tb as well as better performance(kind of) with more disks. 

 

With 80TiB 16gb will work but certainly not as well as 32gb. I imagine with storage that big he will be moving some large files, 32gb will come in handy.

Totally true mikesan, my build will grow with the amount of storage needed.