Identifying Failed Disk In A Raid Array

[charlie22911]

Good day all,

 

I've been watching LTT on YouTube for quite some time now, and a common theme in these videos (I'm looking at you unraid projects) is redundancy through RAID.

Something that has been completely glossed over however is what to do should a drive fail, more to the point; how do you physically identifty a failed drive within a drive pool that lacks status indicators?

 

As to why I am asking:

I've got a spare 6850k around which I am in the planning stages of building an unraid box (build log to come once ready), I've got 8x 4TB WD Red HDDs on hand to go in my build.

[zMeul]

I haven't had a drive fail in a RAID matrix before, but I'm guessing you ID it by it's serial number that's also written on the drive

[Captain_WD]

14 hours ago, charlie22911 said:

~snip~

Hey there Welcome to the community! 

 

Could you give more details about your setup such as planned RAID controller, motherboard, OS details, RAID type, etc.

Typically, depending on the Os and the controller you use, you should get a notification about the specific drive that has problems either by port number or by drive's Serial Number. If you suspect that a drive has failed you should have a tool provided by the controller or the OS that should give you an indication about if, why and which drive has failed. From then on it's only a matter of identification by serial number or port number. 

 

Let us know if you need more info on this More specifics should be useful! 

 

Captain_WD.

[dfsdfgfkjsefoiqzemnd]

Usually the RAID controller will give you the serial number. 

 

One of my SSDs was kicked out of my RAID5 array last Friday.  When I rebooted, this is how the PC told me which SSD it was. 

 

[charlie22911]

Thanks for getting back to me.

 

I actually came to the same conclusion via Google shortly after posting, my apologies; I'll turn this into a build log once I have the parts.

 

Also a heads up to anyone wanting a cheap quality PCIe RAID controller, I found the LSI 9211 for $88 on amazon; same card is $248 on Newegg:

https://www.amazon.com/gp/product/B002RL8I7M/

 

Keep in mind it doesn't include SATA breakout cables so you have to get those separately (unless you are using a backplane).

 

So far here is the hardware I plan to build this unRAID 6 box with:

 

Intel Core i7 6850k 6 Core CPU (Have on hand)

ASUS X99A USB 3.1

4x 8GB Mushkin Enhanced Redline DDR4 2666

8x 4TB Western Digital RED Hard Drives (Have on hand)

128GB Samsung SSD (Cache, have on hand)

LSI SAS9211-8i with 2x SAS to 8 port SATA breakout cables (In the mail)

NVidia Geforce GT730 (Basic GUI stuff, Have on hand)

NZXT Kraken x41 140mm closed loop liquid cooler

Antec High Current Pro HCP-1200 (Have on hand)

Fractal Design Define R5 Blackout Silent

 

Feel free to drop some suggestions if you have them!

 

EDIT:

 

As for RAID type, I don't think I am going to utilize any sort of raid, the plan right now is to just do a BTRFS pool with two parity drives and 1 128GB SSD for cache in unRAID. I'm open to suggestions if there is a better way to approach this though!

[Captain_WD]

10 hours ago, charlie22911 said:

~snip~

Hey, glad to see you got a pretty good deal! 

 

The build plan seems quite good! What usage type are you looking at? What are you going to do on this build? 

 

If you plan on running demanding workloads on this build (it seems like you will be with the water cooling loop and general specs) I'd consider some sort of rubber mounts in order to reduce the vibrations and noise from the HDDs inside your case as this could improve the performance of the build and the noise that it produces. 

 

Post back when you start the build or if you have any questions! 

 

Captain_WD.

[scottyseng]

20 hours ago, charlie22911 said:

-snip-

I'm lost...why do you need a RAID card if you mention BTRFS / Unraid? If you just need more ports, I'd use a HBA card. Otherwise, can't you just make the array using BTRFS? Oh wait, I see you mentioned that you weren't going to the RAID route. Nevermind.

 

Also, cheap and quality RAID cards don't go in the same sentence. The entry level LSI cards 9x40 and below are generally just software RAID on a card. Real quality RAID cards have a dedicated backup battery and dedicated RAM cache to speed things up.

 

Yeah, I got a server chassis because tracing drives on a RAID card without lights is really hard. You have to match up the ports exactly and label the drives / double check the serial numbers on the port listing so you know which drive is which.

 

Hmm, you could just use Unraid for everything.

[dfsdfgfkjsefoiqzemnd]

13 hours ago, Captain_WD said:

I'd consider some sort of rubber mounts in order to reduce the vibrations and noise from the HDDs inside your case

The Define R5 has rubber rings on the HDD trays and custom screws so you can mount the HDDs completely on rubber. 

That's why I chose the R5 for my own NAS.  I'm running 8 Reds (WD40EFRX) in there.

 

Still pretty loud though, but that's normal with all those motors spinning away. 

Right now the NAS is the only device here in my household that still uses HDDs.  As soon as 4TB SSDs drop below 400EUR, that'll end too.

[Captain_WD]

I'd second @scottyseng 's suggestion as it seems that you don't really need a high-end RAID controller. 

 

@Captain ChaosI understand. Having that many HDDs in a single case does come with some noise and vibrations. Still, making something like this requires you to look out for the $/GB ratio and currently HDDs are far better than SSDs. Also, if you are doing more writing on that NAS SSDs may not be the best choice out there. 

Nevertheless, I understand your point.

 

Captain_WD. 

[charlie22911]

 

On ‎10‎/‎19‎/‎2016 at 3:29 AM, Captain_WD said:

Hey, glad to see you got a pretty good deal! 

 

The build plan seems quite good! What usage type are you looking at? What are you going to do on this build? 

 

If you plan on running demanding workloads on this build (it seems like you will be with the water cooling loop and general specs) I'd consider some sort of rubber mounts in order to reduce the vibrations and noise from the HDDs inside your case as this could improve the performance of the build and the noise that it produces. 

 

Post back when you start the build or if you have any questions! 

 

Captain_WD.

 

Right now I have the following planned for it:

• Hosting VMs (1 Windows 10 and 1 Debian based guest for now)
• Hosting several dedicated game servers
• Serving as a backup target for my two PCs and Laptop
• Media serving with on the fly transcoding
• Enabling me to digitally hoard data of dubious value

I ordered the silent edition of the Define R5 which has acoustic dampening material inside to help with noise. The box will be sitting in a closet so some noise is not an issue.

 

 

19 hours ago, scottyseng said:

I'm lost...why do you need a RAID card if you mention BTRFS / Unraid? If you just need more ports, I'd use a HBA card. Otherwise, can't you just make the array using BTRFS? Oh wait, I see you mentioned that you weren't going to the RAID route. Nevermind.

 

Also, cheap and quality RAID cards don't go in the same sentence. The entry level LSI cards 9x40 and below are generally just software RAID on a card. Real quality RAID cards have a dedicated backup battery and dedicated RAM cache to speed things up.

 

Yeah, I got a server chassis because tracing drives on a RAID card without lights is really hard. You have to match up the ports exactly and label the drives / double check the serial numbers on the port listing so you know which drive is which.

 

Hmm, you could just use Unraid for everything.

 

On a broader level I can agree with what you are saying, however LSI makes a quality product; their soft RAID implementation is far better than pretty much any integrated RAID solution I've encountered to date.

But I digress, the main purpose of that card is to give this build room to grow; the motherboard has 8 SATA ports which aren't enough for my 8 HDD + 1 SSD cache layout.

 

 

15 hours ago, Captain Chaos said:

The Define R5 has rubber rings on the HDD trays and custom screws so you can mount the HDDs completely on rubber. 

That's why I chose the R5 for my own NAS.  I'm running 8 Reds (WD40EFRX) in there.

 

Still pretty loud though, but that's normal with all those motors spinning away. 

Right now the NAS is the only device here in my household that still uses HDDs.  As soon as 4TB SSDs drop below 400EUR, that'll end too.

 

How are your drives holding up thermally? I don't plan to spin my drives down and I'm not sure how much airflow will get through such tightly packed drives.

Edited October 20, 2016 by charlie22911
Typing is hard

[scottyseng]

3 hours ago, charlie22911 said:

-snip-

I agree with the LSI card soft RAID implementation, but you might have lower speeds / lose data depending on how many times you lose power or need to force shut down the system. You can just use Unraid / BTRFS with a LSI HBA (pass through) and build the array with BTRFS/UnRAID (No need for hardware RAID). It allows the expansion of the same amount of ports via SAS.

 

I have six WD Reds 4TB packed one of top of the other (one column of six drives, in a 24 bay 4U chassis) and they get to 46C under load. I don't have the stock server fans, I replaced them with lower RPM but slower ones. I think you'll be fine for temps.

 

However, I'd be cautious if you were running the 7200RPM drives, I have the WD Re drives, and they run really hot (even hotter than WD Blacks). I had to space them out to keep them cooler. I remember when I had four in one column, they broke 60C. When they're separated, they run at 55C.

[dfsdfgfkjsefoiqzemnd]

4 hours ago, charlie22911 said:

How are your drives holding up thermally? I don't plan to spin my drives down and I'm not sure how much airflow will get through such tightly packed drives.

My drives don't spin down either.

 

I replaced the front fan with a pair of BeQuiet SilentWings2 fans (much more quiet but less pressure-optimized) and initially I had temps between 40 and 46°C (the lower value being with the drives just spinning away, the higher during sustained writing).  Raising or lowering the fan speed made little to no difference.

 

With a pair of 120mm fans (also SilentWings2) zip-tied to the back of the drive cages, temps drop to 34 on idle and 38°C during writing.  So push-pull really helps to draw more air through the drive cages.

 

Again, fan speed doesn't seem to matter much, so I have all my fans running on idle. 

4 fans (or 6 if you count the side panel fan and the rear fan too) is more than Fractal says the R5's built-in controller can handle, so I took no chances and used an old fan controller that I still had in my parts closet.  A couple of 12V->5V or 12V->7V low-noise adapters should do the trick though.

 

I checked Google's report on drive failures in their data centers ( link to pdf ) and found that the ideal temp would be around the 35°C mark, so my temps are just perfect.

 

In all fairness, I didn't come up with that trick with the 120mm fans myself.  @Cheatdeath did the same to his R5 NAS box, but he had a much harder time because he used Noctua fans which have regular mounts.  My BeQuiet fans have rubber eyelet-shaped mounts that make it a lot easier to put zip-ties through them.

 

 

[Captain_WD]

19 hours ago, charlie22911 said:

~snip~

In this case the setup seems pretty good. 

Anywho, continuing with the topic, either the motherboard through BIOS/UEFI (if you are using hardware RAID from the motherboard or a dedicated controller) or the OS itself (if you are doing software RAID over the OS) will inform you if and which drive has issues and point you towards the specific drive via either the port number or the drive's serial number so it's good to have that info on labels on the drives for easier access instead of having to remove each drive in order to check that. 

 

Captain_WD.