I need education: RAID, backups, images

[johnt]

I'm confused. I thought I did enough research, but I'm coming up short. I have two 2 TB drives in RAID1-ish mode as simple redundancy. I use Disk Management in Windows 11 Pro to mirror the two drives using "volume mirroring." I've tested it and I can remove a drive, connect it to another computer, and it works. It's a perfect duplicate. So the benefit here is that I don't have to worry about a RAID array of any sorts or recovery if an issue comes up in case of a single drive failure. I can just replace it, re-mirror, and move on.

 

But I'm running into an issue where my drives are resynching after every restart. While this isn't a huge pain for 2 TB, my goal is to move onto dual 20 TB drives. My current method is not practical in the long run. The 2 TB drives can take like 8 hours sometimes, so 20 TB is not going to work. So I started looking into back up or cloning software thinking that it would just copy the contents of one drive to another. But it turns out most generate back up images on another drive and it's not a very straightforward recovery if something happens. I also don't understand how and when images are created. Say I have 10 files and the software has created a backup image, if I change 1 of the ten files, does it create a brand new image of all ten files, a partial image of incremental changes? I don't understand it. My recent research has brought me to EaseUS ToDo Backup, but I am not sure if that is going to do what I want. The concept of "imaging" comes up often with back up software and I'm not a fan. I just want my files in two locations. What am I misunderstanding about back up? The concept of imaging seems like it would just eat space on the back up drive.

 

Help! I'm looking for a "OneDrive" method solution, where changes can be quickly detected and the initiation of copying to a backup location begins nearly instantaneously. The nice part about my current setup is that I have no overhead from saving a file in two locations. Windows treats it as a single drive. But that's something I can overlook if I don't have this resynching issue after EVERY restart.

 

I'm posting this here because the drives are on my NAS/server. I have shares that other home devices can access. So we store files on it and some media. This was just supposed to be a basic backup.

[leadeater]

1 hour ago, johnt said:

I'm confused. I thought I did enough research, but I'm coming up short. I have two 2 TB drives in RAID1-ish mode as simple redundancy. I use Disk Management in Windows 11 Pro to mirror the two drives using "volume mirroring." I've tested it and I can remove a drive, connect it to another computer, and it works. It's a perfect duplicate. So the benefit here is that I don't have to worry about a RAID array of any sorts or recovery if an issue comes up in case of a single drive failure. I can just replace it, re-mirror, and move on.

 

But I'm running into an issue where my drives are resynching after every restart.

Change over to Windows Storage Spaces Two-Way Mirror. You can move between computers etc and you won't have that resync issue.

 

1 hour ago, johnt said:

Say I have 10 files and the software has created a backup image, if I change 1 of the ten files, does it create a brand new image of all ten files, a partial image of incremental changes? I don't understand it.

That is just a limitation of the backup software you have used, most don't do it that way. At least not paid backup solutions don't. Windows Backup also does not do that either.

[Needfuldoer]

6 hours ago, johnt said:

I've tested it and I can remove a drive, connect it to another computer, and it works.

You don't want to do this, that's not what mirroring is for. As soon as you plug one drive into another computer, that drive doesn't exactly match the one that stayed behind anymore. Next time they're both connected to the same computer, the "array" will have to resilver no matter what solution built it.

 

Use a separate drive for shuffling files around, and leave your big drives installed in your server. 

 

RAID is uptime insurance against hardware failure so you don't have to recover from your backups, not a synchronization scheme like ZFS replication or rsync.

[johnt]

7 hours ago, leadeater said:

Change over to Windows Storage Spaces Two-Way Mirror. You can move between computers etc and you won't have that resync issue.

I thought I tried this during my testing and the drives were not recognizable by the second computer no matter what I did. They came up in Disk Management but I couldn't do anything to them, and the dedicated software wasn't able to rebuild the array either. I will have to give this another shot I suppose.

 

5 hours ago, Needfuldoer said:

As soon as you plug one drive into another computer, that drive doesn't exactly match the one that stayed behind anymore

This is 100% correct. The reason I did tried this during my testing was to see what would happen to the data on each drive in case one failed, and also. How would I be able to recover my information? I also tested this with Windows Storage Spaces and the second PC would not let me touch the drives at all. Moving them to another PC was also to test the case where I had a PC failure and I needed to move the mirrored drives/RAID array to another device. My bad if it sounds like I was trying to move stuff around... not at all.

 

I've been wondering if I need to try trueNAS or Linux in general to make this happen. The Windows based software seems iffy at best. Although I've heard RAID using trueNAS is not necessarily reliable if a drive fails. I don't mind paying for unraid if I have to. I don't have have enough experience with any of these OSes to make a confident leap (which is why I tried Windows first).

[Kilrah]

38 minutes ago, johnt said:

and also. How would I be able to recover my information?

Say your PC has 2 drives in RAID1, 1 fails, you don't need to recover your information since the pool is still available, just degraded. At that point you replace the failed drive and it rebuilds.

 

38 minutes ago, johnt said:

I also tested this with Windows Storage Spaces and the second PC would not let me touch the drives at all.

Did you move both drives? If so it should work. Moving only one would be importing a degraded array, not sure how that impacts it.

 

As mentioned RAID's only for availability precisely so you can continue to work uninterrupted if one drive fails, it's not a backup in any sense of the word.

The backup should be to a completely separate device, and usually not realtime but regular/scheduled unless it's also versioned. Can be an external drive with something like freefilesync for a file-based backup which it seems you want in this case, you usually only want images for an OS drive. For images Macrium Reflect is pretty good and can do incrementals, sadly they killed the free version recently.

 

I use urbackup for both, it's free but it's a client-server model.

 

[johnt]

3 hours ago, Kilrah said:

Did you move both drives? If so it should work. Moving only one would be importing a degraded array, not sure how that impacts it.

Correct. I moved both drives... sort of. I had two NVME drives and two SATA drives installed on one motherboard. The NVME drives had different Windows 11 OS installed, and the SATA drives were configured in RAID1 in the OS on NVME drive 1. I would switch between the NVME drives to test "different computers." I really thought the volume mirroring was a great alternative to RAID, but the constant resynching is annoying. I wouldn't be worried if it was a quick scan to identify items that are out of sync, and maybe that's a part of it, but it's taking hours to complete after each restart.

 

3 hours ago, Kilrah said:

Can be an external drive with something like freefilesync for a file-based backup which it seems you want in this case

3 hours ago, Kilrah said:

I use urbackup for both, it's free but it's a client-server model.

Holy moly I think these two programs can do exactly what I want, and maybe urbackup can do a bit more than I originally expected from my "client" devices. I liked that RAID1 was more or less realtime, but scheduling an overnight backup is hardly a big deal. Thank you for naming both. This is exactly the education I was looking for. I may not be using these concepts in their ideal methods (secondary device, off site back up, etc.), but my biggest gripe with starting a backup of all my DVDs and blu-rays, and possibly ending up with 10 TB or more of data, and losing it all from one drive failure. Having a back up, even in the same machine, is a huge relief.

 

Do you know if urbackup server and client be installed on the same device?

[leadeater]

4 hours ago, johnt said:

I thought I tried this during my testing and the drives were not recognizable by the second computer no matter what I did. They came up in Disk Management but I couldn't do anything to them, and the dedicated software wasn't able to rebuild the array either. I will have to give this another shot I suppose.

Once a disk is put in to a Storage Spaces pool then you can only manage it via Storage Spaces. If you move the disk to another computer then you have to import the pool using the metadata on the disk, I usually do that using PowerShell but I suspect the Windows Storage Spaces GUI can also do it.

[Blue4130]

1 hour ago, johnt said:

Having a back up, even in the same machine, is a huge relief.

Having the backup in the machine is a big no-no. Way too many things can kill your primary and back up. Always disconnect it after the backup job is finished.

[johnt]

6 hours ago, Blue4130 said:

Having the backup in the machine is a big no-no. Way too many things can kill your primary and back up. Always disconnect it after the backup job is finished.

I knowww but my wife sees me shaking my head after every attempt I've made so far (RAID, volume mirror, copy and paste). I need something that works before improving my back up game. I'm actually considering using my previous NAS and connecting it to my LAN-only (no internet) to perform weekly back ups. It's a slow slow slow Pentium G4400 from the 2015 era with 8 GB of memory. But honestly it was still playing 4k movies for me up to two years ago when I replaced it, so it shouldn't have any trouble transferring files over my LAN for back up. That will be my next great adventure. I'll have to learn how to do wake on LAN. All my networking/server gear is in the garage. You've given me ideas!

[johnt]

7 hours ago, leadeater said:

Once a disk is put in to a Storage Spaces pool then you can only manage it via Storage Spaces. If you move the disk to another computer then you have to import the pool using the metadata on the disk, I usually do that using PowerShell but I suspect the Windows Storage Spaces GUI can also do it.

I guess I didn't realize RAID was like a gang... once you're in... it's almost impossible to leave. My hope and dream, during a drive failure, was to move the remaining drive to another PC and transfer the files to another drive or RAID. I don't know if I feel comfortable using a RAID array for the rest of my life... Maybe I am overthinking it.

[johnt]

@Kilrah dude. THANK YOU. urbackup and freefilesync are exactly and more than what I was expecting. I've been playing with them all day.

 

I thought urbackup was the solution, and while I like the overall server/client method a lot, I really dislike how the back ups are stored. It's way better than other back up software, but it's still weird and kind of hard to navigate. But I'm definitely saving that program for the future. It's way too cool.

 

Now freefilesync on the other hand is exactly what I was looking for. How is this software free? I already donated. Between that and realtimesync (software to check for changes in the source folder and automatically initiate a batch from ffs). It's crazy. I don't understand why Microsoft doesn't just include something this simple in Windows Pro. I guess it's called OneDrive. But that's not practical for a media server and way too expensive.

 

Thank you again.

[leadeater]

4 hours ago, johnt said:

I guess I didn't realize RAID was like a gang... once you're in... it's almost impossible to leave. My hope and dream, during a drive failure, was to move the remaining drive to another PC and transfer the files to another drive or RAID. I don't know if I feel comfortable using a RAID array for the rest of my life... Maybe I am overthinking it.

I'd recommend not and go with data backups

[johnt]

4 hours ago, leadeater said:

I'd recommend not and go with data backups

This is sound advice.

[msknight]

I think it may be worth going back to the drawing board and thinking through what you want to achieve. Personally, I gave up on using Windows for RAID when I was using NT; (it didn't store metadata on the drives, so unless you had the definition file from within Windows, then no other system would see the drive) and if I used a custom RAID chip, there was no guarantee that the chip on another system would recognise the set either. I then moved to Solaris and stuck with ZFS ever since... but that's another story.

 

I have two servers, each have three drives in RAID 5 configuration. Balance of cost, risk, performance, resilience. Three 8tb drives give me 16tb usable space (not really, but no use quibbling for this kind of discussion) If one drive fails, I have time to replace it and resilver.

 

If the system fails, I can plug the drives into any other system using ZFS (version dependent) and "import" the set... as long as it's got enough nodes to be functional.

 

However, I do run backups. Being xNix, I use a tool called "rsync" - I hook up a large drive via USB 3 (I have a few of them so I'm not relying on one backup set) and while the first rsync takes many, many hours... subsequent rsync runs take less than an hour because it only copies the files that have changed. Using the --delete option, if a file has been deleted from the source... then it is deleted from the destination as well. 

eg. This command takes all the files from a data set mounted on /mnt/panther and copies it to a drive mounted on /mnt/movies ...

rsync -rt --delete /mnt/panther/ /mnt/movies

...and when it's done I simply unmount the USB drive and store it safely.

All this software is free... however if you are not used to xNix and ZFS, then it will be a learning curve.

 

There is something that ZFS does (via Scrub) that other RAID solutions won't do... which is to check that the data is still viable. But that's another issue.

 

As for your idea of having a second, "live," system that is backed up to automatically... there's nothing wrong with this. The core issue is what risks you're facing and which you want to mitigate against. Having another, "live," system means that the drives inside it are always running and thus wearing down; the up side to this is that you're able to run nightly copies if you actually want to ... which might mitigate the need to have RAID on your original server to begin with. (all my desktop systems that are running 24/7 are running on a single SSD but rsync themselves to the main server once a week.) ... however you're also then running the risks of the backup server suffering a drive failure, and also power failures (which themselves can be mitigated to a point by UPS and surge/brownout protection)

 

So... my 24/7 PC's run rsync to the main server once a week. The main server has RAID 5 which gives me balanced resilience/storage/cost ... and that is backed up to two different sets of USB removable hard drives every month.

 

I was looking in to a CM3588 as a live backup server in addition... but the CM3588 committed hari kiri a few hours after I started using it, and the company have not been very responsive to the failure... but hey ho.

 

You state that your dream was to take the remaining drives to another system if it failed... and then use them. I believe you are over thinking this... for one simple reason... if a RAID system fails due to drive failure, then the system itself stays running ... there should be no reason to need to move the surviving drive to another system.

 

However... I will sound this warning ... you need a mechanism to alert you to the fact that one drive has failed. In a home environment, some people have no mechanism for this. NAS boxes have LED's on the front to notify of drive failure, and you can make a point of looking at the NAS every morning. Some can also e-mail you if there's a failure. However... many standard OS's actually won't, out of the box, notify you if a drive has failed. The only time you'll know, is when catastrophic failure occurs and by that time it's way too late.

 

In my opinion... that's what you need to test for.

 

Sit down and have a think.

[leadeater]

10 hours ago, msknight said:

Personally, I gave up on using Windows for RAID when I was using NT; (it didn't store metadata on the drives, so unless you had the definition file from within Windows, then no other system would see the drive)

That's reallllly old now, both Storage Spaces and Disk Manager RAID put metadata on the disks are and fully transferable between operating systems.

[Needfuldoer]

The most important thing to remember is that RAID is not a backup. RAID is uptime insurance, keeping your system online and (hopefully) saving you the effort of manually restoring from your backups in the event of a drive failure. 

 

Features that act as version control (like ZFS snapshots, Volume Shadow Copy, and Time Machine) can protect your data from accidental deletion and ransomware, but they're not a substitute for a real backup either.

 

Any data you care about should have offline backups that follow the 3-2-1 rule:

 

3 copies, on 2 different kinds of media, 1 of which is stored off-site.