raid5 How to fix RAID5 on QNAP TS-451+

[Anima Tech]

Hello!

 

Due to an extended power loss my NAS lost power during raid scrubbing or shutdown process (dunno which). Side note: I have an UPS hooked up to it, but my auto shutdown timer was set a bit too high value and my UPS ran out of juice.

Resulting in something getting scrambled with my hardware RAID5 and NAS not booting up. I have PuTTY access to my NAS and the status LED is blinking green, after i somehow managed to get it a little less messed up (At least i hope so.) .

 

I don't want to use something like ReclaiMe RAID software to rescue my files. Because, it takes forever for it to even find all my files and then reinstall my NAS from nothing if that's at all possible to avoid.

Before i even knew i could accidentally trigger resync, i ran "dmesg" and it triggered it and completed successfully as far as i know.

Before resync, 1/4 HDDs was showing "Active" status in "mdadm -E /dev/sdc3" and "clean" in the rest of them. Now all 4 show as "clean".

 

This is what i have tried so far, in the order i tried things(Pastebin links to results):

After i somehow got the damn thing to boot without blinking red status LED, i pressed the "Reset to factory defaults" button in web GUI, in hopes of it allowing me to get to QOS to fix the RAID array.

No such luck, since QNAP in their infinite wisdom requires at least 1 working HDD with QOS installed on it, which is on the broken RAID5 array -.-*. I can't just use some random HDD for QOS either because the RAID tool in QOS requires all RAID discs to be installed. Which is impossible with my 4 drive setup, unless someone knows how to fool the system to think that an USB HDD is an internal one and thus allow installing QOS on it.

 

dmesg (Right after it no longer booted up and was flashing red status LED)

The rest of these are after resync.

mdadm -E (HDD 1-4)

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 - This errors always.

mkdir /share/NAS2 - No problem there

mount -t ext4 /dev/md1 /share/NAS2 - Got error again: wrong fs type, bad option, bad superblock on /dev/md1, missing codepage or other error. In some cases useful info is found in syslog - try dmesg | tail or so.

cat /proc/mdstat + mdadm --detail --scan + mdadm -D /dev/md???

umount - Tried that for md322 and md256 after i still didn't get the array to mount, no luck.

cat /proc/mdstat - I used that to check for the resync progress.

tail - I have no idea how to use that or it doesn't work on my NAS

I forgot the commands for these, but i also tried to use backup superblocks and to stop all services before using "assemble" again after it errored out.

Edit1: lsblk: command not found

Edit1: fdisk -l

Edit1: fsck: command not found

Edit3: Ran lsblk on Ubuntu machine with the drives from my NAS - Results

Edit3: Ran fsck on Ubuntu machine with the drives from my NAS - Results

Edit3: Ran fdisk -l on Ubuntu machine with the drives from my NAS - Results

Edit4: ReclaiMe - Result

 

After my Google-fu failed to get me detailed enough answers to my specific problem, this is what i have gathered about the problem so far:

- I have no hardware failures or problems.

- I possibly have something wrong with the superblock and i haven't been able to use the backup ones.

- There are 2 RAID1 arrays, which get created on each startup. This may have something to do with factory reset. Do i need to delete those and if so, how do i do it?

- The resync probably helped, but according to Google it may have messed up things beyond repair?

- I have little to no knowledge about Linux, so using common sense and according to info gathered with mdadm everything looks good, but is it?

 

My NAS (for fast reference):

Model: QNAP TS-451+

HDDs: WD Red 6TB x4 RAID5 + External USB3 enclosure with some 128Gb Kingston SSD with Windows 7 installed thru Virtual Station for TeamSpeak3 server and for easy NAS management via TeamViewer.

Firmware: 4.3.3.0.4.0.4

Factory reset done after RAID failure (The option specifically said that no data would be lost.)

PuTTY access only

Edit2: I May have messed up the drive order when i tried out ReclaiMe RAID software. Is there a way to verify which order the drives should be?

 

If any info is missing from the post, i'll add it as requested.

 

If i have understood everything correctly, all i need to do is to get the array assembled and mounted without errors and then save changes to mdadm.conf. (I don't know how to do that last part at all.)

Seems simple but as i have mentioned above, it just isn't happening for me. SO then, how do i get this thing working again, without losing any data or having to buy enough HDDs to dump all my data into and reinstalling my NAS from scratch?

Again, im an a total Linux NOOB! So please explain everything in detail! Thank you!

[Electronics Wizardy]

4 minutes ago, Anima Tech said:

Resulting in something getting scrambled with my hardware RAID5 and NAS not booting up. I

Its not hardware raid, its using mdadm in linux

 

Restore backups? Id just delete the array and restore backups on it.

 

what does lsblk get you?

 

 

[Anima Tech]

17 minutes ago, Electronics Wizardy said:

Its not hardware raid, its using mdadm in linux

 

Restore backups? Id just delete the array and restore backups on it.

 

what does lsblk get you?

 

 

Oh, my bad. Going to have to edit that bit.

 

I have no idea how to restore the backups with PuTTY, im not even sure if i have any recent ones.

 

-sh: lsblk: command not found

[Electronics Wizardy]

1 minute ago, Anima Tech said:

Oh, my bad. Going to have to edit that bit.

 

I have no idea how to restore the backups with thru PuTTY, im not even sure if i have any recent ones.

 

-sh: lsblk: command not found

You have backups on a externall hdd or a cloud service right? Just remake the array and restore thise.

 

How much do you care about the data, the best option is to send this to a data recovery service, there is a good chance your just making it worse here.

 

If you want to continue, id put these drives in a desktop running linux, with anouther drive to copy the data too.

[Anima Tech]

5 minutes ago, Electronics Wizardy said:

You have backups on a externall hdd or a cloud service right? Just remake the array and restore thise.

 

How much do you care about the data, the best option is to send this to a data recovery service, there is a good chance your just making it worse here.

 

If you want to continue, id put these drives in a desktop running linux, with anouther drive to copy the data too.

 I do have full backups of all the important data, but there are files that i never had enough space for in my backup NAS. It's these files that i even bother to try and fix this for, since there is only 1 copy of them and it's on the NAS which i have problems with.

 

The data that i might lose is not so important that i'd wanna spent money to have it recovered by some service.

 

I suppose i could install Linux on a spare HDD, but i still haven't got enough space to put all that data to. But if that's the final option, i'll do it and use the new HDDs to upgrade my backup NAS to have enough space for full backups of the main NAS after all that.

[Electronics Wizardy]

6 minutes ago, Anima Tech said:

 I do have full backups of all the important data, but there are files that i never had enough space for in my backup NAS. It's these files that i even bother to try and fix this for, since there is only 1 copy of them and it's on the NAS which i have problems with.

 

The data that i might lose is not so important that i'd wanna spent money to have it recovered by some service.

 

I suppose i could install Linux on a spare HDD, but i still haven't got enough space to put all that data to. But if that's the final option, i'll do it and use the new HDDs to upgrade my backup NAS to have enough space for full backups of the main NAS after all that.

you can boot linux from a live usb, so you don't need a hdd to install it on.

 

The distro on the qnap is limit, so getting data will be a pain, esp since you don't know much about linux.

 

Whats the output of fdisk -l?

[Anima Tech]

9 minutes ago, Electronics Wizardy said:

you can boot linux from a live usb, so you don't need a hdd to install it on.

 

The distro on the qnap is limit, so getting data will be a pain, esp since you don't know much about linux.

 

Whats the output of fdisk -l?

Oh, that's handy. I assume there is a tool to make such an USB stick? Also which distro should i put on a stick?

 

It sure is.. And that's why it's been such a royal pain in the butt to Google how to fix my issue.

 

https://pastebin.com/5XRAsFC0 I will add that to the main post as well.

Should i have added my RAID5 array to that as well? With assemble? It always forgets everything about it when i reboot.

[Electronics Wizardy]

1 hour ago, Anima Tech said:

/dev/md1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3 - This errors always.

what error?

3 minutes ago, Anima Tech said:

Oh, that's handy. I assume there is a tool to make such an USB stick? Also which distro should i put on a stick?

 

It sure is.. And that's why it's been such a royal pain in the butt to Google how to fix my issue.

 

https://pastebin.com/5XRAsFC0 I will add that to the main post as well.

why are you using /dev/sda3. How do you know its on the third partition?

 

What does cat /proc/mdstat show?

[Anima Tech]

2 minutes ago, Electronics Wizardy said:

what error?

why are you using /dev/sda3. How do you know its on the third partition?

 

What does cat /proc/mdstat show?

failed to get exclusive lock on mapfile - continue anyway...

I just did as Google told me and it told me to use "mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3" when reassembling any RAID5 array

 

[~] # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md1 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]

      17551702848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

 

md322 : active raid1 sdc5[3](S) sdd5[2](S) sda5[1] sdb5[0]

      7235136 blocks super 1.0 [2/2] [UU]

      bitmap: 0/1 pages [0KB], 65536KB chunk

 

md256 : active raid1 sdc2[3](S) sdd2[2](S) sda2[1] sdb2[0]

      530112 blocks super 1.0 [2/2] [UU]

      bitmap: 0/1 pages [0KB], 65536KB chunk

 

unused devices: <none>

 

 

 

I updated the fdisk -l to show md1:

 

Disk /dev/md1: 17972.9 GB, 17972943716352 bytes
2 heads, 4 sectors/track, -1 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

[Electronics Wizardy]

24 minutes ago, Anima Tech said:

failed to get exclusive lock on mapfile - continue anyway...

I just did as Google told me and it told me to use "mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3" when reassembling any RAID5 array

 

[~] # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md1 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]

      17551702848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

 

md322 : active raid1 sdc5[3](S) sdd5[2](S) sda5[1] sdb5[0]

      7235136 blocks super 1.0 [2/2] [UU]

      bitmap: 0/1 pages [0KB], 65536KB chunk

 

md256 : active raid1 sdc2[3](S) sdd2[2](S) sda2[1] sdb2[0]

      530112 blocks super 1.0 [2/2] [UU]

      bitmap: 0/1 pages [0KB], 65536KB chunk

 

unused devices: <none>

 

 

 

I updated the fdisk -l to show md1:

 

Disk /dev/md1: 17972.9 GB, 17972943716352 bytes
2 heads, 4 sectors/track, -1 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

how about fsck /dev/md1?

[Anima Tech]

9 hours ago, Electronics Wizardy said:

how about fsck /dev/md1?

-sh: fsck: command not found

[Electronics Wizardy]

Just now, Anima Tech said:

-sh: fsck: command not found

yea id just plug it into a live linux system then, this terminal seems very limited.

 

Id just put ubuntu on a usb drive.

[Jarsky]

Have a read through this thread, and see if it helps. 

Plugging it into a full Linux distro would be helpful though for troubleshooting. 

 

 

[Jarsky]

11 hours ago, Anima Tech said:

 

[~] # cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]

md1 : active raid5 sda3[0] sdd3[3] sdc3[2] sdb3[1]

      17551702848 blocks super 1.0 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

So this shows that your RAID assembled......

 

11 hours ago, Anima Tech said:

I updated the fdisk -l to show md1:

 

Disk /dev/md1: 17972.9 GB, 17972943716352 bytes
2 heads, 4 sectors/track, -1 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Disk /dev/md1 doesn't contain a valid partition table

 

But this shows it cant read the partition table...

 

Maybe try stopping MDADM and force reassembling it? 

 

mdadm --stop /dev/md1
mdadm --assemble --scan --force

 

[Anima Tech]

6 hours ago, Jarsky said:

Maybe try stopping MDADM and force reassembling it? 

mdadm --stop /dev/md1
mdadm --assemble --scan --force

 

Nothing changed. Im plugging the drives into a machine running Ubuntu, in order to get the full set of Linux tools to use for this problem.

Quote

Edit: I May have messed up the drive order when i tried out ReclaiMe RAID software. Is there a way to verify which order the drives should be?

Also when i plug them into the Ubuntu machine, is there a tool to find out the correct order? Or is it just trial and error?

[Jarsky]

Using the --scan function, it should be placing them in the correct order. Only if you destroy the raid and try to re-create it should you really have order issues. Have you checked the UUID in your mdadm.conf? This thread he has the same issue which

[Anima Tech]

On 4/20/2018 at 10:20 PM, Anima Tech said:

Edit: Ran lsblk on Ubuntu machine with the drives from my NAS - Results

Edit: Ran fsck on Ubuntu machine with the drives from my NAS - Results

Im having problems getting mdadm to show anything. Do i need to set my SATA controller into RAID mode(RAID5 isn't supported on my mobo.) in Bios or something to get this working? Current setting is ACHI.

 

It just shows:

Personalities:

unused devices: <none>

On 4/21/2018 at 11:00 AM, Jarsky said:

Plugging it into a full Linux distro would be helpful though for troubleshooting. 

On 4/21/2018 at 9:40 AM, Electronics Wizardy said:

yea id just plug it into a live linux system then, this terminal seems very limited.

Done. Now what should i do?

 

I also ran fdisk -l on the Ubuntu machine - Results

Here is the ReclaiMe array layout info as well

[KasperHJ]

Auch, been where you are! :-/

My Qnap also lost its RAID5 three times, each time I got help from the Qnap Helpdesk, who remoted my Qnap and got my RAID5 back....

 

If your Qnap is up and running but "just" can't load the RAID I suggest you add the Helpdesk QPKG from App Center and create yourself a support ticket.

This will enable Qnap to help you remotely...

 

Best of luck!

 

/Kasper

[Anima Tech]

1 hour ago, KasperHJ said:

Auch, been where you are! :-/

My Qnap also lost its RAID5 three times, each time I got help from the Qnap Helpdesk, who remoted my Qnap and got my RAID5 back....

 

If your Qnap is up and running but "just" can't load the RAID I suggest you add the Helpdesk QPKG from App Center and create yourself a support ticket.

This will enable Qnap to help you remotely...

 

Best of luck!

 

/Kasper

Well, that's the thing.. I have:

On 4/20/2018 at 10:20 PM, Anima Tech said:

PuTTY access only

The only way to get to Helpdesk is to put another drive in without my RAID5 array to have access to QNAP's web GUI.

[KasperHJ]

On 25/04/2018 at 3:55 PM, Anima Tech said:

Well, that's the thing.. I have:

The only way to get to Helpdesk is to put another drive in without my RAID5 array to have access to QNAP's web GUI.

... Now that you mention it, I think the first (and maybe second) attempt to salvage my data by Qnap when my RAID5 broke was actually done using TeamViewer to my own computer, and then remote a Putty session into my NAS ...
I don't know if they've completely scrapped this type of support after they got their Helpdesk-QPKG-service, but maybe you're lucky they still can help you out! :-)

 

/Kasper