Help with new build TrueNAS CORE which raidz and number of drives to use

[ageekhere]

Hi all, i am planning on building a new home storage server and have been researching on which tech to use (i think i have worked out all the hardware I would need for the build), I have about 20TB of data atm and i want to aim for about 50TB to 70TB. I am thinking so far on using TrueNAS CORE with zfs (though not fully decided yet). I want to use 18TB NAS CMR drives but i cannot work out which raidz that I would need to use.

 

My first idea is 4x18TB drives in raidz1 (which would give me about 50TB of usable storage) however though my research it seems that this is a very risky idea.

So the second idea is 5x18TB drives in raidz2(which is about 50TB) however i am losing 2 drives out of the 5.

So maybe then 6x18TB drives in raidz2(which is about 70TB) though that would be quite costly.

I am planning on buying a second hand LTO5 tape library for creating backups.

 

After going down the rabbit hole of rebuild times and risk of failure i am not sure on what hard drive capacity is safe to use, what raidz to use.

Also i am not sure if TrueNAS CORE is the best option for me as i am not sure how long scrub times would take (i think you have to run them every two weeks? would 50TB takes days to scrub?).

 

Help a fellow geek out on deciding which raidz and hard drive capacity to use to achieve somewhere between 50TB to 70TB of capacity.

Thanks

 

 

[Electronics Wizardy]

What hardware are you using?

 

Id probalby go 6x18tb if you can here, otheriwse 4x18tb isn't that raisky and would work fine.

 

How heavy is the io, scrubs should only take a day or two here.

 

 

[ageekhere]

8 minutes ago, Electronics Wizardy said:

What hardware are you using?

 

Id probalby go 6x18tb if you can here, otheriwse 4x18tb isn't that raisky and would work fine.

 

How heavy is the io, scrubs should only take a day or two here.

 

 

Still deciding on the hardware, but something like
AMD RYZEN CPU 

AMD Motherboard with intel gigabit Ethernet controller
32GB ECC memory (can always upgrade to 64GB if needed)

 

With the scrubs taking a day is that only for the first scrub or would i need to do that every 2 weeks? Can i use the server still while scrubbing? 

[Electronics Wizardy]

Just now, ageekhere said:

Still deciding on the hardware, but something like
AMD RYZEN CPU 

AMD Motherboard with intel gigabit Ethernet controller
32GB ECC memory (can always upgrade to 64GB if needed)

 

With the scrubs taking a day is that only for the first scrub or would i need to do that every 2 weeks? Can i use the server still while scrubbing? 

All the scrubs would take a day. You can still use the server when a scrub is running.

 

Id probalby get 10gbe with that size, copying all those files over gigabit is gonna take a while.

 

How about a board like thie x570d4u-2l2t. That way you get 10gbe, impi, garanteed ecc support.

[ageekhere]

The x570d4u-2l2t is about $850 AUD motherboard, i kind of am looking at maybe a $150 AUD motherboard (as it is, it is going to cost about $1000 AUD per hard drive)
Maybe something like B550M-PLUS supports ECC, has 2.5 ethernet (though not intel) and an extra pcie slot for extra sata expansion cards.

Though i still need to do some more research in this area.

[Electronics Wizardy]

2 minutes ago, ageekhere said:

The x570d4u-2l2t is about $850 AUD motherboard, i kind of am looking at maybe a $150 AUD motherboard (as it is, it is going to cost about $1000 AUD per hard drive)
Maybe something like B550M-PLUS supports ECC, has 2.5 ethernet (though not intel) and an extra pcie slot for extra sata expansion cards.

Though i still need to do some more research in this area.

Can you get used parts? Id look at a used server board. Something like 2011-3 should be pretty cheap and have all the features you need.

 

Id avoid those cheaper 2.5gbe nics. What network speed setup do you have now?

 

If you want cheap drives, shuck externals, normally much cheaper than new hdds.

[ageekhere]

Just normal home 1GB Gigabyte Network (10 gigabit switches are still quite costly), will leave an upgrade option so i can add in a 10gbe nic.

The drives i am looking at are Seagate IronWolf Pro NAS 18TB HDD, ST18000NE000.
Can always look at used server motherboards.

 

The main issue was working out which raidz and the supported hard drive size.

Thanks for the help. 

[Electronics Wizardy]

1 minute ago, ageekhere said:

Just normal home 1GB Gigabyte Network (10 gigabit switches are still quite costly), will leave an upgrade option so i can add in a 10gbe nic.

The drives i am looking at are Seagate IronWolf Pro NAS 18TB HDD, ST18000NE000.
Can always look at used server motherboards.

 

The main issue was working out which raidz and the supported hard drive size.

Thanks for the help. 

What sites are you buying from? id get shucked external drives as there much cheaper per tb.

 

Since you have a good backups plan, a raid failure isn't the end of the world, and the risk is much lower than some make it out to be.

[ageekhere]

$1149 - ST18000NE000 https://www.scorptec.com.au/product/hard-drives-&-ssds/hdd-3.5-drives/85058-st18000ne000
$2199 - EXTERNAL DESKTOP DRIVE https://www.scorptec.com.au/product/hard-drives-&-ssds/external-desktop-drives/89291-sthj36000800

$99 cheaper to get the external drive, hmm.

 

[Electronics Wizardy]

24 minutes ago, ageekhere said:

$1149 - ST18000NE000 https://www.scorptec.com.au/product/hard-drives-&-ssds/hdd-3.5-drives/85058-st18000ne000
$2199 - EXTERNAL DESKTOP DRIVE https://www.scorptec.com.au/product/hard-drives-&-ssds/external-desktop-drives/89291-sthj36000800

$99 cheaper to get the external drive, hmm.

 

how about this guy https://www.scorptec.com.au/product/hard-drives-&-ssds/external-desktop-drives/90255-wdbbkg0180hbk-aesn

 

$330 cheaper per hdd

[Thirdgen89GTA]

8TB and 14TB drives are the sweet spots right now.  You'll save yourself big bucks if you don't try for the highest capacity platters.

 

There are cases you can buy, both rack mount, and mid-tower ATX style that can fit a significant amount of drives. Once you get past 6-8 drives you need a HBA to support more with most mainboards.

 

And as you probably won't have access to a LTO tape backup system, and most likely don't have enough money to build two for twice the price, I would not go with anything less than RaidZ2 for double redundancy.  All of my arrays use RaidZ2, its no replacement for a true backup, but this was a Cost vs Failure comparison.

 

Lastly, my scrubs are 100% unnoticeable.  I can't tell when the server is doing them because they are scheduled start at midnight and only on Sundays. The longest scrub is 8 hours, the shortest is 8 minutes.  I stagger the scrubs so they don't occur on the same day, and I only scrub monthly.

 

My vDevs consist of 6 disks in RaidZ2.  I have 1x SSD, and 2x 8TB Spinning rust.

 

Total raw disk capacity is over 96TB.

 

 

 

[ageekhere]

1 hour ago, Thirdgen89GTA said:

8TB and 14TB drives are the sweet spots right now.  You'll save yourself big bucks if you don't try for the highest capacity platters.

 

There are cases you can buy, both rack mount, and mid-tower ATX style that can fit a significant amount of drives. Once you get past 6-8 drives you need a HBA to support more with most mainboards.

 

And as you probably won't have access to a LTO tape backup system, and most likely don't have enough money to build two for twice the price, I would not go with anything less than RaidZ2 for double redundancy.  All of my arrays use RaidZ2, its no replacement for a true backup, but this was a Cost vs Failure comparison.

 

Lastly, my scrubs are 100% unnoticeable.  I can't tell when the server is doing them because they are scheduled start at midnight and only on Sundays. The longest scrub is 8 hours, the shortest is 8 minutes.  I stagger the scrubs so they don't occur on the same day, and I only scrub monthly.

 

My vDevs consist of 6 disks in RaidZ2.  I have 1x SSD, and 2x 8TB Spinning rust.

 

Total raw disk capacity is over 96TB.

 

 

 

Good info, thanks