HP P410i - Not completeing rebuild

[svmlegacy]

Hi,

 

So, long story short. RAID 1+0, drive failed a few months ago. No big deal, swapped it out for a new one. It seems to rebuild while the system is on, slow blinking between the paired drives. Usually takes ~ 6hrs, then all four drives light up green, and blink with I/O. Anytime the system is powered off or rebooted, it again goes to rebuild. (Pic related)

 

What do I need to do to complete the rebuild, so that I don't have to worry of a 2nd drive failure, short of remaking the RAID array?

 

System: HP DL380 G6, Ubuntu 20.04 x86_64, headless.

 

Edit:

BIOS: P62 (Latest) iLO2: 2.33 (Latest) Raid Firmware: 6.64 (Latest)

[Radium_Angel]

13 minutes ago, svmlegacy said:

swapped it out for a new one

You didn't swap it out for a WD Red NAS drive did you?

[svmlegacy]

Just now, Radium_Angel said:

You didn't swap it out for a WD Red NAS drive did you?

Negative. It's a Toshiba 2.5" drive.

[Radium_Angel]

1 minute ago, svmlegacy said:

Negative. It's a Toshiba 2.5" drive.

Ok, I'm assuming you let the system finish rebuilding the RAID. If the process is interrupted, it'll start from fresh.

Also, if this is a server, why are you rebooting it or powering it off?

[svmlegacy]

Just now, Radium_Angel said:

Ok, I'm assuming you let the system finish rebuilding the RAID. If the process is interrupted, it'll start from fresh.

Also, if this is a server, why are you rebooting it or powering it off?

Sometimes it has an uptime of multiple weeks, but I do take it down occasionally for updates on the OS. Most recently I had rebooted it to take pictures of the setup process for a friend. It's not running anything mission-critical. We also had a wicked storm a few days ago that caused a power outage.

 

I've just updated the RAID controllers firmware from 2.74 to 6.64, which should bring the system fully up to date.

[Radium_Angel]

3 minutes ago, svmlegacy said:

Sometimes it has an uptime of multiple weeks, but I do take it down occasionally for updates on the OS. Most recently I had rebooted it to take pictures of the setup process for a friend. It's not running anything mission-critical. We also had a wicked storm a few days ago that caused a power outage.

 

I've just updated the RAID controllers firmware from 2.74 to 6.64, which should bring the system fully up to date.

I understand about storms, etc.

Hopefully the FW update will solve things. I know some HP servers insist on a low-level format of drives before rebuilding the RAID. We had one at work which took 3 days to rebuild because of this.

[Nick7]

Did you check status from OS using hpacucli?

[svmlegacy]

28 minutes ago, Nick7 said:

Did you check status from OS using hpacucli?

I appreciate the advice. hpacucli doesn't work on my system atm, and won't pick up the controller (Pic related)

 

 

[Dutch_Master]

If you have another spare drive of a suitable size, I'd suggest using dd to copy over the data from the intact side of the RAID1 so you have at least a working copy in case of another failure. Next, consider turning that RAID1+0 into a RAID6. In RAID6 you can loose any 2 drives, whereas in your current setup, loosing the wrong drive in an already degraded array could cost you dearly.

[svmlegacy]

2 minutes ago, Dutch_Master said:

If you have another spare drive of a suitable size, I'd suggest using dd to copy over the data from the intact side of the RAID1 so you have at least a working copy in case of another failure. Next, consider turning that RAID1+0 into a RAID6. In RAID6 you can loose any 2 drives, whereas in your current setup, loosing the wrong drive in an already degraded array could cost you dearly.

Much appreciated. I've been keeping my data backed up, as I'm a bit paranoid of losing this array. Unfortunately my controller can only do RAID 5 or 1+0.

[svmlegacy]

I was able to get hpssacli working, and can confirm that it finished rebuilding when on.

 

 

However, during the last reboot after the firmware upgrade, it still goes into a rebuild. We'll have to see next time it goes down, I guess.

[Dutch_Master]

11 hours ago, svmlegacy said:

Much appreciated. I've been keeping my data backed up, as I'm a bit paranoid of losing this array. Unfortunately my controller can only do RAID 5 or 1+0.

In that case I'd recommend RAID5 with a hot-spare disk. Still the same storage capacity, but in case of a failure the array will start rebuilding immediately (with the hot-spare) and you have time to swap out the defective drive (replacement becomes new hot-spare).

 

Another consideration: I'm not aware what OS you use, but for Linux I'd suggest putting the card in HBA mode (if it can) and let the kernel deal with RAID issues, the mdadm tool is perfect for that. If you're not on Linux, could the problem stem from an interrupt flag not set (correctly) and Ubuntu interpreting it as a degraded array?