Jump to content

Advice about rescuing degraded RAID 5 array

So I have been running a home file server that is based on a Pentium III and a 3ware 9500S 4 port sata raid controller with four 500GB WD Black drives in RAID5 config for the past number of years with little to no issues. The OS of choice has been ClarkConnect followed by ClearOS Enterprise edition.  In the last few months I have noticed that one of the drives was reporting in ClearOS as being degraded, so I was limiting use of the server although it was still functioning fine on the remaining 3 drives. More recently I decided to finally try and tackle fixing it and started to copy data off to an external drive as a backup (possibly the wrong deicison).  The first ~500GB chunk of data was copied slowly but with no other issues. However when I tried to start copying a second chunk of data over, the machine suddenly went unresponsive and the array was unbootable after a reboot. Looking into the 3ware manager software on the card itself after post, i could see the array with a degraded status and one drive reporting as "not in use". I selected the option to rebuild the "unit" in the manager and it was saying that rebuild would occur after exiting (pressing F8) however when I did that it would continue trying to boot as normal and fail. There was no indication of a rebuild in porgress.

 

Next we tried replacing the unused drive with a similar working drive, thinking that the rebuild would occur after a fresh drive has been added.  Only later did it become apparent that the "problem" drive had to be removed from the unit before physically disconnecting and replacing it. With the fresh drive attached the manager was reporting as one drive being available and the raid unit being unusable. There was seemingly no way of adding the new drive into the unit in order to rebuild. We reattached the original drive to the controller and tried to find a way to still rebuild. At some point through all the restarts and attempts at problem solving we ended up with the original 4 drives giving the same sort of unusuable unit message with one available drive. 

 

We considered RAID recovery software but were unable to boot linux live CD's on the original computer. We then moved the card and drives over to a windows Vista machine. The controller is visible in Windows and installed a driver for the correct model (although it is old) but the drives are not visible in disk manager or anywhere else.

 

Is there any way of somehow forcing the unit back into the "degraded" state so that we can again attempt a rebuild?

 

Any suggestions or help in the right direction would be greatly appreciated!

Laptop: Lenovo T430
- Intel Core i5 3320M @ 2.60GHz - HD4000 Graphics - 12GB DDR3 RAM Dual Channel - SanDisk SSD 128GB - Pop!_OS -

Desktop: self-built
CPU: Ryzen 7 2700 - Motherboard: MSI Tomahawk B450 MAX II - RAM: 32GB DDR4 Kingston HyperX Fury @ 3200- GPU: Gigabyte GTX 1660Ti Windforce OC 6GB
Case: Corsair 275R Airflow - Storage: Samsung 970 EVO Plus & Crucial MX300 - PSU: Seasonic Focus GX-750 - Displays: Iiyama ProLite X2783HSU & EIZO FlexScan1921
Cooling: AMD Wraith Spire - Keyboard: Keychron K3 Pro - Mouse: Logitech MX Ergo & Steelseries Rival 3 Wireless - Sound: Steelseries Arctis 7 - OS: Pop!_OS 22.04 LTS (6.2.6)

Wife's Desktop: self-built

CPU: AMD Ryzen 5 1600 @ 3.8GHz - Motherboard: MSI Mortar B350M - RAM: 32GB DDR4 G.skill Ripjaws V @ 2866MHz - GPU: Sapphire RX 5700XT

CaseCorsair 275R Airflow - Storage: Samsung 970 EVO Plus & 870 EVO - PSU: Seasonic Focus PX-750 - Displays: Massdrop VAST & EIZO FlexScan1921

Cooling: DeepCool Captain EX with Corsair Light Loops - Keyboard: Keychron K8 Pro - Mouse: Keychron M1 Wireless - Sound: Steelseries Arctis 7 - OS: Linux Mint 20.3 (6.2.4)

Link to comment
Share on other sites

Link to post
Share on other sites

My guess would be the RAID card is faulty, I've had about three 3ware cards do the exact same thing.

 

As far as RAID recovery goes you actually shouldn't use the current RAID card to do it, use an HBA or onboard SATA controllers. You want to treat the disks as non-RAID disks then run up a recovery tool such as Recover My Files point it to the 4 disks and tell it there was a RAID 5 array using these disks with X stripe size.

Link to comment
Share on other sites

Link to post
Share on other sites

I don't have knowledge on this topic to give Next Step advice but I do have a question. This old Pentium III RAID server. Not counting the 1 drive RAID redundancy involved in a RAID5 configuration. There was no backup for the server?

 

In my own servers configuration my server provides two major services. Backup & remote access. I keep all my files in a RAID on my main computer and the server is an entire copy with its own built-in redundancy. If the RAID in the server fails and is found in the same degraded state as yours I can simply dump the configuration and all the data. Wipe the drives, recreate the RAID, format the newly build array and copy all the information back from my desktop RAID. I've never needed to do this but if it happens I can. So their's no backup to the server I take it?

Link to comment
Share on other sites

Link to post
Share on other sites

I have to admid that I never worked with 3ware cards but as leadeater mentioned your card is very probalby buggy. My approach would be to get a new (identical) RAID card (short look on ebay shows some used 98000-8 ports cards from 10$ up).  Importatnt is to use a similar manufacturer and the card should not be older.

You can then attach the drives to the new controller and it is possible that the card will recognize the array and start a rebuild.

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

Thanks for the replies so far. We are currently waiting for a replacement controller. Will report back with results and eventual next steps.

Laptop: Lenovo T430
- Intel Core i5 3320M @ 2.60GHz - HD4000 Graphics - 12GB DDR3 RAM Dual Channel - SanDisk SSD 128GB - Pop!_OS -

Desktop: self-built
CPU: Ryzen 7 2700 - Motherboard: MSI Tomahawk B450 MAX II - RAM: 32GB DDR4 Kingston HyperX Fury @ 3200- GPU: Gigabyte GTX 1660Ti Windforce OC 6GB
Case: Corsair 275R Airflow - Storage: Samsung 970 EVO Plus & Crucial MX300 - PSU: Seasonic Focus GX-750 - Displays: Iiyama ProLite X2783HSU & EIZO FlexScan1921
Cooling: AMD Wraith Spire - Keyboard: Keychron K3 Pro - Mouse: Logitech MX Ergo & Steelseries Rival 3 Wireless - Sound: Steelseries Arctis 7 - OS: Pop!_OS 22.04 LTS (6.2.6)

Wife's Desktop: self-built

CPU: AMD Ryzen 5 1600 @ 3.8GHz - Motherboard: MSI Mortar B350M - RAM: 32GB DDR4 G.skill Ripjaws V @ 2866MHz - GPU: Sapphire RX 5700XT

CaseCorsair 275R Airflow - Storage: Samsung 970 EVO Plus & 870 EVO - PSU: Seasonic Focus PX-750 - Displays: Massdrop VAST & EIZO FlexScan1921

Cooling: DeepCool Captain EX with Corsair Light Loops - Keyboard: Keychron K8 Pro - Mouse: Keychron M1 Wireless - Sound: Steelseries Arctis 7 - OS: Linux Mint 20.3 (6.2.4)

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

Status update:

Getting another of the same RAID controller did not solve the issue. Everything stayed unchanged. So of to the next step we ordered a plain and simple SATA controller with enough ports, since we only have laptops that are current and thus no other option than to retrofit an older machine on the cheap. We use a recovery software we found that rebuilds the filesystem currently to a USB drive. It is painfully slow and will only be able to be read by Linux, but that should be fine to rescue files. Next report will follow.

Laptop: Lenovo T430
- Intel Core i5 3320M @ 2.60GHz - HD4000 Graphics - 12GB DDR3 RAM Dual Channel - SanDisk SSD 128GB - Pop!_OS -

Desktop: self-built
CPU: Ryzen 7 2700 - Motherboard: MSI Tomahawk B450 MAX II - RAM: 32GB DDR4 Kingston HyperX Fury @ 3200- GPU: Gigabyte GTX 1660Ti Windforce OC 6GB
Case: Corsair 275R Airflow - Storage: Samsung 970 EVO Plus & Crucial MX300 - PSU: Seasonic Focus GX-750 - Displays: Iiyama ProLite X2783HSU & EIZO FlexScan1921
Cooling: AMD Wraith Spire - Keyboard: Keychron K3 Pro - Mouse: Logitech MX Ergo & Steelseries Rival 3 Wireless - Sound: Steelseries Arctis 7 - OS: Pop!_OS 22.04 LTS (6.2.6)

Wife's Desktop: self-built

CPU: AMD Ryzen 5 1600 @ 3.8GHz - Motherboard: MSI Mortar B350M - RAM: 32GB DDR4 G.skill Ripjaws V @ 2866MHz - GPU: Sapphire RX 5700XT

CaseCorsair 275R Airflow - Storage: Samsung 970 EVO Plus & 870 EVO - PSU: Seasonic Focus PX-750 - Displays: Massdrop VAST & EIZO FlexScan1921

Cooling: DeepCool Captain EX with Corsair Light Loops - Keyboard: Keychron K8 Pro - Mouse: Keychron M1 Wireless - Sound: Steelseries Arctis 7 - OS: Linux Mint 20.3 (6.2.4)

Link to comment
Share on other sites

Link to post
Share on other sites

Status update and conclusion:

After a four day long rebuild to an external USB drive we now have the original partitions back on it! Data can be recovered; everything seems to be readable just fine from my usual Linux live systems.
One of the files i copied off of it, m4a i think was having trouble and skipped, so there might be a small percentage of data being corrupted. All other random files we tested however were fine.
We are gonna plan out a new server set up in the coming months. I will probably post about it in a kind of a build log thread.

Laptop: Lenovo T430
- Intel Core i5 3320M @ 2.60GHz - HD4000 Graphics - 12GB DDR3 RAM Dual Channel - SanDisk SSD 128GB - Pop!_OS -

Desktop: self-built
CPU: Ryzen 7 2700 - Motherboard: MSI Tomahawk B450 MAX II - RAM: 32GB DDR4 Kingston HyperX Fury @ 3200- GPU: Gigabyte GTX 1660Ti Windforce OC 6GB
Case: Corsair 275R Airflow - Storage: Samsung 970 EVO Plus & Crucial MX300 - PSU: Seasonic Focus GX-750 - Displays: Iiyama ProLite X2783HSU & EIZO FlexScan1921
Cooling: AMD Wraith Spire - Keyboard: Keychron K3 Pro - Mouse: Logitech MX Ergo & Steelseries Rival 3 Wireless - Sound: Steelseries Arctis 7 - OS: Pop!_OS 22.04 LTS (6.2.6)

Wife's Desktop: self-built

CPU: AMD Ryzen 5 1600 @ 3.8GHz - Motherboard: MSI Mortar B350M - RAM: 32GB DDR4 G.skill Ripjaws V @ 2866MHz - GPU: Sapphire RX 5700XT

CaseCorsair 275R Airflow - Storage: Samsung 970 EVO Plus & 870 EVO - PSU: Seasonic Focus PX-750 - Displays: Massdrop VAST & EIZO FlexScan1921

Cooling: DeepCool Captain EX with Corsair Light Loops - Keyboard: Keychron K8 Pro - Mouse: Keychron M1 Wireless - Sound: Steelseries Arctis 7 - OS: Linux Mint 20.3 (6.2.4)

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

Side note about data rescue... I am sure most people who are more geeky than us and run a Linux system permanently as their dailies will not find this helpful, but to us and some that are still on windows despite all the privacy issues because we wanna game, there is a software from paragon that allows you to mount extfs on windows. also another that does the same with hfs. i have them both since i used macs a lot years ago and now we got the other for our rescued linux partition.

 

TL;DR Use Linux partitions on Windows with ExtFS for Windows (they offer a free trial).

Laptop: Lenovo T430
- Intel Core i5 3320M @ 2.60GHz - HD4000 Graphics - 12GB DDR3 RAM Dual Channel - SanDisk SSD 128GB - Pop!_OS -

Desktop: self-built
CPU: Ryzen 7 2700 - Motherboard: MSI Tomahawk B450 MAX II - RAM: 32GB DDR4 Kingston HyperX Fury @ 3200- GPU: Gigabyte GTX 1660Ti Windforce OC 6GB
Case: Corsair 275R Airflow - Storage: Samsung 970 EVO Plus & Crucial MX300 - PSU: Seasonic Focus GX-750 - Displays: Iiyama ProLite X2783HSU & EIZO FlexScan1921
Cooling: AMD Wraith Spire - Keyboard: Keychron K3 Pro - Mouse: Logitech MX Ergo & Steelseries Rival 3 Wireless - Sound: Steelseries Arctis 7 - OS: Pop!_OS 22.04 LTS (6.2.6)

Wife's Desktop: self-built

CPU: AMD Ryzen 5 1600 @ 3.8GHz - Motherboard: MSI Mortar B350M - RAM: 32GB DDR4 G.skill Ripjaws V @ 2866MHz - GPU: Sapphire RX 5700XT

CaseCorsair 275R Airflow - Storage: Samsung 970 EVO Plus & 870 EVO - PSU: Seasonic Focus PX-750 - Displays: Massdrop VAST & EIZO FlexScan1921

Cooling: DeepCool Captain EX with Corsair Light Loops - Keyboard: Keychron K8 Pro - Mouse: Keychron M1 Wireless - Sound: Steelseries Arctis 7 - OS: Linux Mint 20.3 (6.2.4)

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×