Do any NAS solutions do everything I want?

[UncleFunkle]

I'm looking to either setup a new NAS server or upgrade storage on my existing one. Currently I'm running TrueNAS Scale with 2x 3TB drives and 4x 2TB drives all configured in the same pool as RAID-Z2. I think ZFS is a great filesystem, but it's also a pain for home use. Mismatched drive capacities cause me to give up space on those drives (currently have 2TB unusable) and pools can't easily be expanded if drives are later replaced with larger drives. Before I start copying the data to whatever spare storage I have so I can recreate the pool and copy it back, I want to know if another NAS option would be better.

 

Features I'm looking for:

• Mismatched drive support (with minimal unused space)
• Expandable storage by replacing one or more drives with larger ones
• Per-folder compression (I don't want to apply compression to things that are already compressed)
• Full Windows ACL support
• Time Machine support with ability to limit share size so macOS doesn't fill the entire NAS
• VM or Docker support (would just be running a couple very small things)
• rclone or other encrypted backup support (must work with Backblaze B2)
• Bitrot detection (automatically recovering from that is nice, but alerting me to restore the file from a backup is fine)

The only places where TrueNAS fails are the first 2 bullet points. To my current understanding, Unraid fails on the compression and bitrot detection points (correct me if I'm wrong). Do any other options like Synology, QNAP, or Asustor cover this list better?

[LIGISTX]

3 minutes ago, UncleFunkle said:

Features I'm looking for:

• Mismatched drive support (with minimal unused space)
• Expandable storage by replacing one or more drives with larger ones

The only places where TrueNAS fails are the first 2 bullet points. To my current understanding, Unraid fails on the compression and bitrot detection points (correct me if I'm wrong). Do any other options like Synology, QNAP, or Asustor cover this list better?

ZFS has these issues as you are experiencing. It was not designed for “cost cutting rollouts”, it was designed for enterprise. That said, once all of the drives are replaced with the larger capacity drives (if you 1 by one replace all of the drives with 3TB), you will have a vdev of the total capacity of those drives… but only once they are ALL that size. 
 

unraid is likely the “best” alternative, with windows storage spaces being another option which I do think will do some of what you want, but you’d then need to get windows programs to do some of what you want like replication and backup. 
 

Truenas is likely the best answer… but you can’t really peace meal it. Need to invest in the storage you need up front, and when you start to run out, need to save to buy a new vdev. 

[Electronics Wizardy]

Synology should be able to do all of this for you, but you need to buy their hardware.

 

Windows server should be able to do all of this too.

 

1 hour ago, UncleFunkle said:

Unraid fails on the compression and bitrot detection points (correct me if I'm wrong).

You could do compression on unraid with btrfs if you want to with a bit of mount option tweaking.

 

HDDs have checksumming so its pretty rare for a hdd to have incorrect data read. THey will throw an error if a bad block if found.

 

 

[UncleFunkle]

4 hours ago, LIGISTX said:

ZFS has these issues as you are experiencing. It was not designed for “cost cutting rollouts”, it was designed for enterprise. That said, once all of the drives are replaced with the larger capacity drives (if you 1 by one replace all of the drives with 3TB), you will have a vdev of the total capacity of those drives… but only once they are ALL that size.

Yeah I figured as much about ZFS. I think it's excellent for that purpose. Just doesn't work well for home use.

2 hours ago, Electronics Wizardy said:

Synology should be able to do all of this for you, but you need to buy their hardware.

 

Windows server should be able to do all of this too.

 

You could do compression on unraid with btrfs if you want to with a bit of mount option tweaking.

 

HDDs have checksumming so its pretty rare for a hdd to have incorrect data read. THey will throw an error if a bad block if found.

I'll look into Windows server. I hadn't thought of that as an option. Though if you're saying Synology should meet my needs, it's probably a better choice for me. Would take up less space and use less power than my current tower I'm using with an old Xeon inside.

 

But are you sure the checksums within the HDD are enough? Didn't Linus nearly lose a bunch of video because of bit rot by not scheduling the ZFS scrub task to run? I know bit rot is nasty because if you don't catch it fast enough, the backups will be poisoned too.

[LIGISTX]

1 hour ago, UncleFunkle said:

Yeah I figured as much about ZFS. I think it's excellent for that purpose. Just doesn't work well for home use.

4 hours ago, Electronics Wizardy said:

Define not good for home use.... many people use it for home use, myself and my homelab friends included. You just need to go into it understanding what it is and what its limitations are. 

 

What is your end goal? If you plan to add more space, start buying 3 TB drives to replace the 2 TB drives... once you get all of them replaced, you will magically gain more space, AND you will have a bunch of 2 TB drives you can set up another vdev with (maybe add a few larger drives to that new 2TB vdev as well). Then when you are starting to need more space again, start replacing the 2 TB drives in vdev 2 with whatever size of larger drive you picked, and start replacing them.

 

Ideally, you just buy what you need up front and go from there, but that is not cheap. I spent ~1500 bucks to get my 10 4 TB drives back in 2015, but it is all I need, and I sit at about ~55% used. Just need to plan ahead, and potentially save up to buy drives.

 

1 hour ago, UncleFunkle said:

But are you sure the checksums within the HDD are enough? Didn't Linus nearly lose a bunch of video because of bit rot by not scheduling the ZFS scrub task to run? I know bit rot is nasty because if you don't catch it fast enough, the backups will be poisoned too.

No, SMART isn't good enough. SMART doesn't checksum everything, it just tries to catch errors, but it doesn't really actively fix anything and clearly doesn't catch everything (you can have drive outright fail without ever getting a SMART error).

 

Data rot is a thing, but for *most* home user data, it's not that big of an issue. That said... I personally consider it something I work towards not being affected by, and thus do run ZFS and run ECC RAM to really give myself the best shot at data security. Is this needed - no. Do I do it because I can - yup.

[UncleFunkle]

7 hours ago, LIGISTX said:

Ideally, you just buy what you need up front and go from there, but that is not cheap. I spent ~1500 bucks to get my 10 4 TB drives back in 2015, but it is all I need, and I sit at about ~55% used. Just need to plan ahead, and potentially save up to buy drives.

Thinking on it overnight, I'm convinced now that this is the way to go. Keep using TrueNAS and just drop the money up front to get what I expect to be plenty of storage for the next 10 years.

[Electronics Wizardy]

9 hours ago, LIGISTX said:

No, SMART isn't good enough. SMART doesn't checksum everything, it just tries to catch errors, but it doesn't really actively fix anything and clearly doesn't catch everything (you can have drive outright fail without ever getting a SMART error).

 

The drives checksum all the data that is stored on them. This is separate from smart, and is why drives very rarely read bad data, as the drives know if there reading the correct data.

 

11 hours ago, UncleFunkle said:

But are you sure the checksums within the HDD are enough? Didn't Linus nearly lose a bunch of video because of bit rot by not scheduling the ZFS scrub task to run? I know bit rot is nasty because if you don't catch it fast enough, the backups will be poisoned too.

I think the problem there was a bad backplane. Normally when ZFS shows lots of checksum errors you would notice it in a diff

 

It would also fail a parity check that most raid controllers /raid cards run. If you use something like unraid you can do checksumming with btrfs if you want.

[LIGISTX]

1 hour ago, Electronics Wizardy said:

The drives checksum all the data that is stored on them. This is separate from smart, and is why drives very rarely read bad data, as the drives know if there reading the correct data.

From my understanding, the drive knows it’s reading bad data, but has no idea what the correct data is… so it’s helpful insofar as it’ll spit out a SMART error, but then your SOL. Plus, if that was the case, wouldn’t it be impossible for a scrub to fix bad data if no SMART error was ever seen? Is that always the case that a drive will throw an error prior to a scrub going and fixing if?

[Electronics Wizardy]

38 minutes ago, LIGISTX said:

From my understanding, the drive knows it’s reading bad data, but has no idea what the correct data is… so it’s helpful insofar as it’ll spit out a SMART error, but then your SOL. Plus, if that was the case, wouldn’t it be impossible for a scrub to fix bad data if no SMART error was ever seen? Is that always the case that a drive will throw an error prior to a scrub going and fixing if?

To my knowledge, ZFS doesn't look at or read the smart data at all. ZFS does its own checksumming, and will know if a read is incorrect and try to fix the error if it can.

 

HDDs will often slow down and throw read errors if the drive can't read the data, Many of the times I've seen a bad hdd will be much slower as it keeps trying to read a sector correctly.

[LIGISTX]

13 minutes ago, Electronics Wizardy said:

To my knowledge, ZFS doesn't look at or read the smart data at all. ZFS does its own checksumming, and will know if a read is incorrect and try to fix the error if it can.

I am almost certain ZFS doesn’t look at SMART. It will just scan for errors and fix them if it has enough parity data to do so. 
 

I am just saying, I don’t think SMART can be replied upon at all. It will not actually do anything besides (hopefully) warn you of an issue; it can’t actually fix data integrity issues.