Need some Help: blk_update_request I/O error

[chilicheeseburger]

Alright, so I ran into a problem with my Server (running Ubuntu 16.04 LTS). It disappeared from my network today so I went to see what was going on. The server was running but it didn't accept any input from mouse or keyboard, so I decided to manually do a hard reset.

 

It tried to boot up but before it got to the login screen it started displaying these error messages:

Spoiler

This message is still updating all the time, I don't know what it means exactly and what will happen after that.

I'm running (or was running?) a software RAID 5 with 4x 3TB WD RED drives and a SanDisk 128GB SSD for the OS.

I take it sde is the SSD and it has somehow died?

[leadeater]

It'll be a dead disk/array, probably the SSD since the OS went down. Can you take it out and plug it in to another PC just to see if it shows up correctly? Also check if the firmware has locked it in Read-Only mode due to wear.

[chilicheeseburger]

12 minutes ago, leadeater said:

It'll be a dead disk/array, probably the SSD since the OS went down. Can you take it out and plug it in to another PC just to see if it shows up correctly? Also check if the firmware has locked it in Read-Only mode due to wear.

I also figured it was the SSD since otherwise it should boot. sda, sdb, sdc, sdd should be the HDDs that make up the RAID 5 array, sde should be the SSD but I don't remember 100% since it's been 2.5 years since I set it all up... I could take it out and plug it in another PC but should I do that right now or wait to see what happens? It's still constantly updating those error messages and I'm not too sure what the first block of numbers means.

 

If I put it in another PC, how would I go about checking if the firmware locked it in read-only mode? SMART Tools?

[unijab]

Live CD boot

[chilicheeseburger]

4 minutes ago, unijab said:

Live CD boot

Going to try that now. 

 

Do you know what the numbers in front of the blk_update_request mean? 

[unijab]

its time

[brwainer]

38 minutes ago, chilicheeseburger said:

Going to try that now. 

 

Do you know what the numbers in front of the blk_update_request mean? 

that is the time since the system booted when the message was generated

[chilicheeseburger]

3 minutes ago, unijab said:

its time

 

3 minutes ago, brwainer said:

that is the time since the system booted when the message was generated

Okay thanks. I made a bootable USB drive, trying that now.

[chilicheeseburger]

Okay so I booted from the live USB drive I created. It lists all my drives and as I already suspected, the sde drive is in fact the SSD.

 

I installed mdadm, scanned for my RAID array. It found it, I remounted it and all the data is there so that's awesome.

 

Inside the disk utility, my SSD shows up and this is the SMART data it initially reported:

I'm running another self test now. The strange thing is that it says "Overall Assessment: Disk is OK"

Should I trust that, format the SSD (maybe try and back it up before that) and do a clean reinstall of ubuntu? Or play it safe and buy a new SSD?

 

Edit: Maybe I should note that I ran into some problems back when I updated from Ubuntu 14. Could that also cause the errors?

[brwainer]

16 minutes ago, chilicheeseburger said:

Okay so I booted from the live USB drive I created. It lists all my drives and as I already suspected, the sde drive is in fact the SSD.

 

I installed mdadm, scanned for my RAID array. It found it, I remounted it and all the data is there so that's awesome.

 

Inside the disk utility, my SSD shows up and this is the SMART data it initially reported:

I'm running another self test now. The strange thing is that it says "Overall Assessment: Disk is OK"

Should I trust that, format the SSD (maybe try and back it up before that) and do a clean reinstall of ubuntu? Or play it safe and buy a new SSD?

 

Edit: Maybe I should note that I ran into some problems back when I updated from Ubuntu 14. Could that also cause the errors?

It is very common for SMART to show no issues when a drive is failing - particularly for write problems (which is what the blk_update_request I/O Error indicates). I recommend making a backup of this disk (should probably be able to do this since you haven't shown any indications of read problems yet) and replacing the SSD. Even if you can make it work again by formatting and reinstalling, I wouldn't trust it anymore.

[chilicheeseburger]

This is what the second Self-Test came up with, pretty much the same...

Just now, brwainer said:

It is very common for SMART to show no issues when a drive is failing - particularly for write problems (which is what the blk_update_request I/O Error indicates). I recommend making a backup of this disk (should probably be able to do this since you haven't shown any indications of read problems yet) and replacing the SSD. Even if you can make it work again by formatting and reinstalling, I wouldn't trust it anymore.

I thought that was the entire point of SMART... Anyway is there a program in particular you suggest to back it up (maybe to make an image of the entire drive) or should I just copy the contents?

[brwainer]

6 minutes ago, chilicheeseburger said:

I thought that was the entire point of SMART... Anyway is there a program in particular you suggest to back it up (maybe to make an image of the entire drive) or should I just copy the contents?

yes, that is/was the point of SMART, but there is still many causes of failure that don't get captured by either the metrics, or the self tests. The self tests in particular can't detect certain write problems.

 

If you can connect both the new and old drives to the computer at the same time, then you can use gparted (should be on your live CD) to copy (clone) the drive over. Otherwise, Clonezilla works well for either drive cloning, or backup and restore (you "clone" the old drive to a file on another disk to backup, and then "clone" that file onto the new drive to restore - but the Clonezilla menus will guide you through it)

[chilicheeseburger]

1 minute ago, brwainer said:

yes, that is/was the point of SMART, but there is still many causes of failure that don't get captured by either the metrics, or the self tests. The self tests in particular can't detect certain write problems.

 

If you can connect both the new and old drives to the computer at the same time, then you can use gparted (should be on your live CD) to copy (clone) the drive over. Otherwise, Clonezilla works well for either drive cloning, or backup and restore (you "clone" the old drive to a file on another disk to backup, and then "clone" that file onto the new drive to restore - but the Clonezilla menus will guide you through it)

Okay, I'll try that once I have a new SSD. Just noticed it's getting hard to find a decent 128GB SSD and it seems prices went up suddenly... can't find any Samsung and only the BX300 from Crucial on Amazon (Germany) for 62€... Those are my two favorite manufacturers of SSDs... Not sure if I wanna go with SanDisk again, then again it was running for 2.5 years pretty much 24/7... Anyway, I'll figure something out.

 

In the mean time I'm running the server from the USB drive, only using it for PLEX and Samba anyway (for lack of better ideas ) and I've set that all up now.

I'm just so glad I didn't lose any data from the RAID and that it was so easy to reassemble that. I appreciate Linux so much more now .

 

Thanks for all the help! I'm not too knowledgeable when it comes to Linux and server/RAID stuff. Actually, that was part of the reason why I built that server, to gain some experience.

 

 

 

[brwainer]

2 minutes ago, chilicheeseburger said:

Okay, I'll try that once I have a new SSD. Just noticed it's getting hard to find a decent 128GB SSD and it seems prices went up suddenly... can't find any Samsung and only the BX300 from Crucial on Amazon (Germany) for 62€... Those are my two favorite manufacturers of SSDs... Not sure if I wanna go with SanDisk again, then again it was running for 2.5 years pretty much 24/7... Anyway, I'll figure something out.

 

In the mean time I'm running the server from the USB drive, only using it for PLEX and Samba anyway (for lack of better ideas ) and I've set that all up now.

I'm just so glad I didn't lose any data from the RAID and that it was so easy to reassemble that. I appreciate Linux so much more now .

 

Thanks for all the help! I'm not too knowledgeable when it comes to Linux and server/RAID stuff. Actually, that was part of the reason why I built that server, to gain some experience.

 

 

 

That is odd about SSDs being scarce and expensive. Did you check other european websites? I recall the UK having quite a few good tech sites. Not sure how bad shipping (and tax?) is between countries in the EU, I'm in the US. Also don't be afraid to use a 120GB drive either, if that's all you can get, you can adjust the partition sizes to match using gparted (CloneZilla might also be able to do this during the recovery/clone, don't remember) - either one will make sure that they aren't corrupting any data when shrinking, they will move data as needed.

[chilicheeseburger]

5 minutes ago, brwainer said:

That is odd about SSDs being scarce and expensive. Did you check other european websites? I recall the UK having quite a few good tech sites. Not sure how bad shipping (and tax?) is between countries in the EU, I'm in the US. Also don't be afraid to use a 120GB drive either, if that's all you can get, you can adjust the partition sizes to match using gparted (CloneZilla might also be able to do this during the recovery/clone, don't remember) - either one will make sure that they aren't corrupting any data when shrinking, they will move data as needed.

I checked but didn't find anything better. I'm probably just remembering the prices wrong and it was never significantly cheaper. Shipping from other EU countries depends a lot on the store you order from but in most cases if you can get the item locally it defeats the purpose. As for tax, with physical items you always pay the VAT of the country you ordered from.

 

My old drive was also 128GB so cloning shouldn't be a problem. A larger SSD wouldn't make sense since it's only supposed to hold the OS and some programs.

[brwainer]

Just now, chilicheeseburger said:

My old drive was also 128GB so cloning shouldn't be a problem. A larger SSD wouldn't make sense since it's only supposed to hold the OS and some programs.

I'm saying that you can also move down to a 120GB drive if needed - just need to shrink your main partition by ~8GB

[chilicheeseburger]

5 minutes ago, brwainer said:

I'm saying that you can also move down to a 120GB drive if needed - just need to shrink your main partition by ~8GB

Okay, sorry didn't quite catch that the first time  Thanks for the tip!

 

I'm considering two options now:

1. Directly replacing the 'dead' SSD with a 120GB Crucial BX300 (62,90€)
2. Getting a 525GB Crucial MX300 (146,88€) to replace the 128GB ADATA SP900 from my main rig and use that to replace the 'dead' SSD in my server.

Option 2 would give me more bang for the buck, but I'd have to go thru more trouble since the ADATA SSD is my main rig's OS drive. 

[brwainer]

41 minutes ago, chilicheeseburger said:

Okay, sorry didn't quite catch that the first time  Thanks for the tip!

 

I'm considering two options now:

1. Directly replacing the 'dead' SSD with a 120GB Crucial BX300 (62,90€)
2. Getting a 525GB Crucial MX300 (146,88€) to replace the 128GB ADATA SP900 from my main rig and use that to replace the 'dead' SSD in my server.

Option 2 would give me more bang for the buck, but I'd have to go thru more trouble since the ADATA SSD is my main rig's OS drive. 

Either way would be about the same amount of work in my opinion - sure with option 2 you will end up doing 2 clones, but in either case you are going to have to deal with resizing partitions. The actual cloning is just set-up-and-wait, and the resizing requires you to do some more managing (barely any, but still more work than setting up a clone).

[chilicheeseburger]

On 10/11/2017 at 3:41 AM, brwainer said:

Either way would be about the same amount of work in my opinion - sure with option 2 you will end up doing 2 clones, but in either case you are going to have to deal with resizing partitions. The actual cloning is just set-up-and-wait, and the resizing requires you to do some more managing (barely any, but still more work than setting up a clone).

Thanks for all the help! Quick Follow-Up on this whole issue:

 

I chose to go with the Crucial BX300 120GB to just replace the old SanDisk SSD in my server. Saved some money that way.

I did create an image of  the old drive, however I then chose to go with a fresh install of Ubuntu 17.04 because 

• There's not really much to set up. Reassemble and mount the RAID 5 array, set up PLEX, a VNC server, SFTP access and automatic backups. I did all that within a couple of hours.
• There have been some issues with the previous install and I suspect they came from either the update from 14.04 to 17.04 or me messing around too much. 

My server is back up and running now and I'm kinda glad I went with fresh install of the OS.

 

I'm evaluating what to do with the old SSD now, it still gets recognized, I can read and write from and to it just fine. But with the previous issue and 2.5 years of uptime I wouldn't wanna trust it as a boot drive anymore. I have it hooked up to my main rig now in an USB 3.0 enclosure and I'll use it for some Steam games until I can think of something better to do with it.

 

Here's a benchmark I did for fun. (Connected via USB 3.0)

Spoiler

 

[leadeater]

On 10/11/2017 at 1:06 PM, chilicheeseburger said:

I thought that was the entire point of SMART... Anyway is there a program in particular you suggest to back it up (maybe to make an image of the entire drive) or should I just copy the contents?

SMART also pre-dates SSD existence and most SSDs go in to a firmware locked read-only state when they wear out which makes it very hard for SMART to tell if the SSD has failed.

 

15 hours ago, chilicheeseburger said:

I can read and write from and to it just fine.

Interesting that it still works, sounded to me like it had gone in to read only. Maybe it was just file system corruption or something.