Data Integrity Tools For Storage Spaces?

[Looby]

Hello All,

 

Currently running storage spaces ntfs in parity and have concerns just in general about data integrity.  

 

 

What are my best options to ensure my data doesnt suffer degredation?  Are there third party scanning tools or anything that might assist with this?

 

* ReFS has a lot of mixed reviews and dont think bog standard w10pro allows that anymore.

 

* Drivepool + stablebit scanner + snapraid sounds like an "ok" solution but im iffy on the largest disk is parity drive thing as my biggest drives are almost double the other drive sizes.  Scanner tool seems nice for drive health/integrity checks and notifications sound amazing but Im more interested in data checking.

 

* Move to zfs type os...  Yeah Im happy on windows and it does everything else I need as is.

 

 

What are realistic options?  Ive got backups of backups but none of them will do me any good if im backing up corrupted data....  Yes in theory if I have multiple backups I can just go back to an uncorrupted one BUT if its not a file I touch for years (old pics, docs, videos, etc) I might get to a point where the rotation overwrites the last uncorrupted copy.

 

*Edit - My concerns stem from noticing old music files and recordings with errors that they didnt use to have.  Some are 15-20 year old recordings of my band and are irreplaceable.  Again yes I have backups but see above comment about backing up corrupted files without knowing.

 

 

Thanks,

Looby

 

Edited July 22, 2022 by Looby
Adding more detail

[AbydosOne]

Welcome to my personal (self-inflicted) hell.

 

tl;dr: Set up a ZFS-based NAS (with scrubs!) and don't think too hard about it. Zeros issues over seven years (and I do actually check). Also, bit rot is very rare on HDDs (in my experience); flash is where the trouble lies.

 

29 minutes ago, Looby said:

I might get to a point where the rotation overwrites the last uncorrupted copy.

The thing about backups is that they need to know that a byte/page/sector was modified in order to back it up (when doing incremental backups). Bit rot is a) random and b) doesn't inform the file system that it changed something (that would be too easy). Bit rot shouldn't propagate unless the file was read/modified/rewritten with error in place, causing the FS to set it as "modified".

 

36 minutes ago, Looby said:

Are there third party scanning tools or anything that might assist with this?

Scrubbing basically reads the whole file system on a hardware level and ensures all the data blocks still come to the same checksum/hash as when they were written (which is stored with the data). The FS can then repair blocks based on other copies in the disk array. One can also do this on a OS file level (save hashes for each file then compare periodically), but I can't recommend a single piece of software to achieve this (mostly because I wrote my own so that it behaved exactly how I needed for how I set up my backups).

[Looby]

Ive heard great things about zfs file system but from my definitelty limited knowledge you cant pool different size disks and add others as needed.  That is a basic need of mine as Ive got disks of all sizes and GOD how I dont want to take everything down every time I need to expand storage.

 

Ive got everything from 16tb to 2tb drives deployed right now.  Hell most I dont even have in even numbers...

 

Im *knock on wood* lucky to have never experienced a drive failure but also worry about storage spaces recovery ability should one die and another dies during rebuild but I backup regularly so unless multiple drives in multiple pools in multiple servers die I SHOULD be ok...

 

Mostly Im using Aomei Backupper in a sync configruation when I need to update files on a non connected backup drive.  Idk if that helps or hurts my problems though.