Windows Storage Spaces = Useless ?

[TheEnglishGasMan]

Hi There,

 

Looking for some help please from one of the many gurus on here : 

 

Context

I have recently upgraded my home server by increasing overall storage capacity from 20 TB to 84 TB by adding a bunch of brand new Seagate 16TB disks that were on sale @ Newegg in April. 

 

My total drive pool now looks like this : 

 

1 x Samsung 960 PRO NVMe M.2 SSD (1 TB) Boot Drive 

2 x Seagate IronWolf Pro 10TB Disks (20 TB) Original Pool

4 x Seagate Exos 16TB Disks (64 TB) New Disks

Total useable pool = 76.3 TB 

 

My original 20 TB setup was configured in Windows Storage Spaces and in Two-way mirror format. The boot drive was not part of the storage pool. 

 

I added all 4 Exos disks to the pool at once (which I now regret) and for a month had no issues. Unfortunately, one of the Exos drives has since failed (My first hard disk failure in over 10 years and on a brand new disk). Thankfully, I had not migrated all data from various external and other hard disks to the server yet - so total consumption of the pool is only 9.08 TB (I have provided a screenshot below). 

 

Proposed Resolution

• To diagnose/ confirm the issue with the failed disk (and to send it back to Seagate for replacement) I need to delete the storage pool.
• First, I need to transfer my data from the pool. 
• I bought another Seagate IronWolf Pro 10TB disk - connected it up and began migrating the data from the pool.
• My plan was to do the above, then recreate the pool with the 2 x 10 TB hard disks, then 2 x 16 TB hard disks. Then finally when Seagate sends me a replacement drive - to upgrade the pool with the other 2 drives. 

 

The Problem

• 7 TB of the 9.08 TB transferred without any issues.
• Today while transferring the remaining 2.08 TB - I noticed the transfer speed dropping to 0 and the suspected failed drive making some horrible noises. 
• The transfer is now stuck and it seems the remaining data on the pool is unreadable. 

 

My Questions

• How can the data on the pool be unreadable? I have 6 disks in there - 1 set of 3 for useage and 1 set for mirror. Even with one drive down, the data should be mirrored on the other drives and fully readable. Am I missing something? 
• Is there any other proposed resolution?
• To prevent this from happening again - would you have any suggestions? This is my first experience with Software RAID or RAID of any kind actually. Historically I have used a Drobo DAS for the last 6 years or so and never had a problem. 

 

Yours faithfully,

The Gas Man

 

 

 

 

 

[leadeater]

7 minutes ago, TheEnglishGasMan said:

To diagnose/ confirm the issue with the failed disk (and to send it back to Seagate for replacement) I need to delete the storage pool

Why? You shouldn't have to delete the pool.

 

7 minutes ago, TheEnglishGasMan said:

The transfer is now stuck and it seems the remaining data on the pool is unreadable

You should unplugged the failed disk, not doing that can interfere.

 

Also you should upgrade the Pool version to the latest possible, you originally created it on an older version of Windows and upgrading the pool version is a good idea.

[TheEnglishGasMan]

@leadeater

 

Thanks for your reply. My apologies for not including in the original post.

 

I had originally attempted to the remove the drive from the pool and received the following error : 

 

Can't remove drive from the pool

Drive could not be removed because not all data could be reallocated. Add an additional drive to this pool to reattempt this operation. 

 

Should I go ahead and physically unplug the drive to force removal? 

[leadeater]

5 minutes ago, TheEnglishGasMan said:

Should I go ahead and physically unplug the drive to force removal? 

Yes, if it keeps coming back online it will stall the OS and the storage sub system while it tries to probe the disk and if it's faulty it could just cycle between showing up, locking, going offline showing up etc etc. If it's bad just disconnect it.

 

If that doesn't work then you'll likely have to give Repair-VirtualDisk a shot.

 

The way Storage Spaces is designed is to prevent operations that could cause data loss which is why you can't remove the disk from the pool. You're supposed to attach a new disks to the system then click remove from pool and it'll bring in the new disk and copy the data off the failed one or from other copies in the pool if the disk being replaced is offline. Problem is not everyone has a spare disk or can get a replacement before removing the faulty disk.

[brwainer]

30 minutes ago, TheEnglishGasMan said:

@leadeater

 

Thanks for your reply. My apologies for not including in the original post.

 

I had originally attempted to the remove the drive from the pool and received the following error : 

 

Can't remove drive from the pool

Drive could not be removed because not all data could be reallocated. Add an additional drive to this pool to reattempt this operation. 

 

Should I go ahead and physically unplug the drive to force removal? 

In this case its better to unplug the failed drive so Windows doesn’t keep trying to use it. You’ll leave the pool as-is, complaining about a missing drive, until the replacement arrives. Then you’ll add the new one, which will allow you to (logically) remove the failed one. But as @leadeater said, you should do the upgrade on it. 

[TheEnglishGasMan]

@leadeater & @brwainer

 

Thank you guys for your quick responses. It's much appreciated - as you can tell, a little out of my depth here. 

 

Update  : 

 

Bad News

• After trying to migrate the data manually (before posting this) and it failing. I cancelled the transfer. It took a good two hours for it to finally cancel. 
• I rebooted the machine and went back into storage spaces.
• I now have 2 of the 16TB drives with warnings. For some reason, serial numbers are not displayed when the drives have warnings. Not sure why this is. 

Good News (maybe?) 

• Fed up, I booted into Safe mode and went back into storage spaces - not sure why but maybe hoping the 2nd drive didn't have the warning, but it did. 
• I then tried to remove the two drives via the remove option (there was no prepare for removal option). 
• Same as this earlier today - I received the Can't remove drive from the pool error as mentioned above. 
• Shut down and booted back into normal windows with the plan of doing one more check then physically removing the drives as mentioned above. 
• This time when I entered storage spaces - to my surprise - the two drives with warnings are now under status "Preparing for removal" and over the space of the last few hours their usage has gone from 22% & 24.3% to 13%.
• I'm assuming this means that storage spaces is reallocating the data to the other disks in the pool? 

If this works.... would you advise to remove the two "warning" drives from the pool, re-format them and try to reuse them? Tough times getting any sort of delivery services here at the moment and would love to avoid sending the drive back to Seagate.

 

 

[leadeater]

54 minutes ago, TheEnglishGasMan said:

I'm assuming this means that storage spaces is reallocating the data to the other disks in the pool? 

Yes

 

54 minutes ago, TheEnglishGasMan said:

If this works.... would you advise to remove the two "warning" drives from the pool, re-format them and try to reuse them? Tough times getting any sort of delivery services here at the moment and would love to avoid sending the drive back to Seagate.

Format them outside of the pool first and run some tests on them to check they are actually ok

[WereCatf]

19 minutes ago, TheEnglishGasMan said:

If this works.... would you advise to remove the two "warning" drives from the pool, re-format them and try to reuse them?

Just formatting them and then reusing them would be stupid. You need to actually test them before reusing them, like e.g. first do a SMART surface-level self-test followed by a software-test. If those don't return any errors, then you may attempt to proceed with re-attaching them to the pool.

[TheEnglishGasMan]

@leadeater & @WereCatf

 

Thanks both.

 

The two drives are now at 10% and 9.8% and the horrific noises from before seem to have stopped (fingers crossed). 

 

Once done, I'll format and run them through SeaTools to check health before attempting to reintroduce. 

[brwainer]

2 hours ago, TheEnglishGasMan said:

@leadeater & @WereCatf

 

Thanks both.

 

The two drives are now at 10% and 9.8% and the horrific noises from before seem to have stopped (fingers crossed). 

 

Once done, I'll format and run them through SeaTools to check health before attempting to reintroduce. 

Horrific noises mean you shouldn’t use it (them?) anymore. Hopefully you only have one failed drive, but hard drives tend to exhibit batch-level failures - i.e. all the drives produced in one day tend to be affected by the same problem.

[TheEnglishGasMan]

@brwainer 

 

Thanks for the reply. Yes, that's where I'm leaning towards - to send both back to Seagate but will still run SeaTools once Im done with the pool.

 

Update : 

• Unfortunately it seems that the warning drives are stuck at 6.56% during Preparing for removal. They've been at 6.56% for about 6 hours now. 
• I tried to "Remove" on the warning drives again and I get the same error as above "Can't remove drive from the pool - data cannot be reallocated, add a new drive to the pool and try again"

What is the best way to proceed?

 

Thanks in advance for the support.

[brwainer]

4 hours ago, TheEnglishGasMan said:

@brwainer 

 

Thanks for the reply. Yes, that's where I'm leaning towards - to send both back to Seagate but will still run SeaTools once Im done with the pool.

 

Update : 

• Unfortunately it seems that the warning drives are stuck at 6.56% during Preparing for removal. They've been at 6.56% for about 6 hours now. 
• I tried to "Remove" on the warning drives again and I get the same error as above "Can't remove drive from the pool - data cannot be reallocated, add a new drive to the pool and try again"

What is the best way to proceed?

 

Thanks in advance for the support.

Wait 24-48 hours. Don’t do anything more which might interrupt it and lose your data. 

[TheEnglishGasMan]

Thanks again for the support rendered.

 

Update : 

• After 36 hours of waiting, the drives did not move from 6.56% so I decided to cut my losses. 
• I went through my files (the last 2TB as outlined in first post) and manually copied them onto the new 10TB disk. From 880 or so files left, I managed to salvage around 280 with 600 being unreadable / unmoveable. All in all, from 9.08TB of data - I managed to salvage 7.2TB.
• I then deleted the storage pool and one by one removed all 6 drives from the pool.

Drive Testing : 

• I downloaded SeaTools and ran a combination of 3 basic tests across the drives (SMART Check, Short Drive Self Test and Short Generic Read). 
• Surprisingly all drives passed SMART and Short Drive Self Tests but unsurprisingly two drives failed the Short Generic Read tests. The two drives with warnings in earlier screenshots. 
• These drives will be RMA'd back to Seagate. Still can't believe that I got two brand new drives that failed within one month of operation, never happend to me before in 20 years of being an enthusiast. Guess I'll be more careful with Neweggs hard disk sales moving forward. 

Recreating the Pool & A New Issue : 

• I formatted the good drives with Windows Disk Manager and ran the SeaTools tests again. 
• I tried to recreate my original 20TB storage pool with the 2 x 10TB hard disks and have begun encountering a new error : 

 

 

• I then tried to create a new pool with the 2 remaining 16TB disks and encountered the same error. 
• Interestingly - When the storage pool creation fails, the drives selected for the pool become "unallocated" in drive management and need to be recreated as simple volumes every time. 

Spent a few hours researching this error and have done the following troubleshooting :

1. Switched all SATA Data Cables around physically (on drives & on motherboard). 
2. Disabled / Re-enabled disks in Device Manager 
3. Tried creating a pool with a single drive with the plan to then add a drive once the pool was created (found a thread on another TechForum that said this resolved the issue for them). 
4. Booted into Safe Mode and tried creating the pool there (another TechForum had mentioned this had worked for them). 

Unfortunately none of the above have been successful. 

 

I'm yet to try the below : 

 

1. SeaTools Overwrite / Sanitize Erase to format the drives - My thinking here is that maybe Windows can tell that these drives used to be part of a previous pool (maybe some residual metadata) and is not letting me create a new pool. 
2. Fresh Windows install - Can't see how this would work though as boot drive was not part of the pool.  
3. Powershell - I understand you can execute storage space instructions through Powershell but again my experience with PS is very limited. 
4. Hardware RAID - My MSI Z270 board has onboard Hardware Raid and I'm thinking of going down this route. Worried I would lose the Storage Space flexibility of creating a single drive letter with multiple drives in the future. 

Anyone experienced this issue before?

 

*DEEP SIGH* 

 

Thanks again in advance for any support.