Home Server Upgrade - Lessons Learned - Converting Live RAID and Resizing Encrypted Partitions

Gerowen · December 31, 2020

So I just finished upgrading the storage in my home server and I learned a few lessons that I thought I would share with you guys, so that hopefully some of you might find it useful at a later date. I recorded a lot of it and plan on writing it up in a blog and making a YouTube video about it. If/when I do, I'll edit this post with links to those items.

First off, my server is running Debian 10 Linux in a headless state; no GUI installed, which means 99% of my management is done via SSH unless there's some kind of a problem that prevents it from booting so that I have to physically go connect a monitor to it, so everything I did was done via the command line. It "was" using two 12TB WD Gold drives in Linux software RAID 1 (mdadm). I created a partition with a specific sector count on each of those drives about 1GB less than the total capacity of the drives, and then added that partition as the RAID member. That way if I ever replaced a drive with one of a different model and the capacities were "slightly" off, it wouldn't be an issue. When creating the actual data partition on the RAID device, I encrypted the partition so that if I ever had to RMA a drive, nobody would be able to retrieve sensitive or personal information from the drive at a later date, especially since RAID 1 has a complete copy of everything on both drives, so one drive is enough to recover everything. The partition didn't used to be encrypted until I had my first drive failure and realized I was about to ship a drive with lots of personal files through the US Postal Service, and that in all likelihood they would refurbish the drive and resell it with the same platters inside since I'm pretty sure it was just the read head that died. I managed to get the drive working long enough to run shred on it for about 90% of the drive before it quit for good (which was fine because it wasn't 90% full) before shipping it off, but after that I decided to encrypt the partition.

We started approaching the 12TB limit recently, and I decided that the easiest course of action would be to buy one additional drive and switch to RAID 5; which would double our storage capacity to 24TB and still allow us to lose one drive without losing any data. I have a separate off-site backup drive, so we're talking strictly about what's physically inside the server. My only concern was, I really didn't want to delete my RAID group and re-make it if I didn't have to. I have the backup, but it would have taken over 24 hours to restore from the backup since it's a USB external drive; and that would have left us without Nextcloud and Plex for that entire time (I make heavy use of Nextcloud and my kids and nieces/nephews love Plex), plus that would mean that until it was finished, I would only have one complete copy of my data, and I'd just have to hope my backup drive didn't die during the restoration.

As it turns out, if you're using Linux software RAID (mdadm), a two disk RAID 1 can be directly converted to a degraded two disk RAID 5, at which point you just add the third disk so that it's no longer in a degraded state. All you have to do is run:

mdadm /dev/mdfoo --grow --level=5

mdadm --grow /dev/mdfoo --add /dev/foo --raid-devices=3

The cool thing about this was that even though the reshape took almost two days, my data was accessible during that time, so we were still able to continue using the various services on the server. On top of that, once it was done, it automatically grew the array as a whole to the proper new size of 24TB, although I had read that in some cases you might need to do that part manually. The server obviously runs on an UPS, so if there had been a power outage or flicker during the process, it wouldn't have bothered anything. I even rebooted the server once during the reshaping process and it just picked up where it left off once it was back up and running.

My next issue was that, even after the many hours it took to reshape the array into a proper RAID 5 that spanned all three disks, the actual data partition on it was still only 12TB. The partition was encrypted, so my next task was to resize that partition to take advantage of the increased capacity of the RAID pool.

After doing some reading and experimenting in a VM, I discovered that once the reshape was done, all I had to do was unmount and luksClose the encrypted partition, then unlock it again with luksOpen, and LUKS automatically extended the LUKS device to fill the new capacity of the RAID array. Then I just had to do a filesystem check on the data partition within the LUKS device by running:

e2fsck -f /dev/mapper/somedrive

Then resize the partition with:

resize2fs /dev/mapper/somedrive

Resizing the partition itself only took a couple of minutes.

So what I've learned from all this is that, if you don't care to do some reading, Linux disk management via the terminal is incredibly powerful and flexible. I was able to convert a software RAID 1 directly to a software RAID 5, and then resize an encrypted partition on that RAID 5 group, all without losing any data, and with minimal actual downtime. Obviously I updated my backup drive before starting the process, just in case. I'm just incredibly grateful that it all went off without a hitch, which isn't always the case when it comes to technology, so I thought I would share my little learning experience in case it helps or entertains some of you.

The info I found about converting the RAID array was here:

https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm#Upgrading_a_mirror_raid_to_a_parity_raid

And the help I got for resizing the encrypted partition came from "jelly" on the Debian IRC on Freenode.