LTT Storage Rankings

[KayKun]

Jep i know ist but the term Scary RAID is new to me

When i get more 2 TB Drives i would Like to Change to RAID 5 or RAID 6 to be a little bit Saver.

But i´m happy its a little and nice Project wenn i have monny to throw away B)

And i´m allways out for new ideas for my Server

[TJStamp]

Are you sure you are using the consumer 3TB versions? Those are the drives we're referring to, ST3000DM001

 

You're quite right actually... I assumed that $employer went with their original plan to cheap out and populate our NetApp platform with the barracuda drives, looks like we're using ST3000NM0033 Constellation series instead... which is quite a relief

[Tomasu]

 

 

Keep in mind Backblaze's methodology is extremely flawed, and I would not use it as a reliable reference. 

 

 

In the end, you're just as likely to get dead drives from any company, personally, I've only ever had 1 HDD fail on me and it was a WD Caviar Blue 1TB after 1 year. Does that mean I'll never buy WD again? Nope. I also have 4 Seagate NAS drives that have been running (perfectly) for a total of over 20,000 drive hours (about 9 months of 24/7 use for 2 and 7 months of 24/7 for the other 2) and none have died.

EDIT: No need to paraphrase! Alp's quote is now pasted here.

 

Keep in mind I've had something like 6ish Seagates fail the past year. 2 3TBs out of 5, and 3 2TBs out of 7, and 1-3 (1 for sure, i cant remember when exactly the other 2+ 1TB drives failed, could have been a year prior) 1TB drives.

 

Yes, I just had a second 3TB drive fail. Out of an array that has barely been used the past year, and is purely a backup array which is rsynced to from the main nas array once a day, which its only been doing for the past month or two. Before that, it literally wasn't doing anything.

 

If I could remember the total count of failed drives the past 15 years, it'd be pretty shocking. Its over 1 disk a year on average at this point. A few years ago, it was at about 10 disks, and around 1 disk a year on average. I just assumed it was bad luck, but at this point? I will not be buying another seagate for quite some time.

 

And before anyone asks, my computers are all on UPSs so they get decent power, the controllers for the two big arrays are LSI SAS2008 cards that likely aren't killing drives (can a controller card cause a drive to report uncorrectable sectors and reallocate hundreds or thousands of sectors?), everything is cabled decently with SAS breakout cables, temperatures are reasonable at around 35c all the time. The drives were also not all in the same system, or on the same UPS during the various failures.

 

Some of the drives suffered from seagates power management firmware bugs, causing heads to park and unpark many times a day, even when not in use apparently.

 

I just have no trust left to give them. I used to be a fan for many years. I bought nothing but seagates. Maybe in a few years when they figure out their problems I'll buy more, but so far it seems WD and HGST drives have better customer reviews. Sure, call me a statistical anomaly, but from my point of view, they are currently not worth buying. I think there's a reason Seagate's desktop drives are cheaper than almost everyone else's. You get what you pay for.

 

append: fun story, one of the failed 1TB disks is a previous RMA return. thats at least the second time I had a replacement fail on me. one brand new (from memex), the other an actual rma replacement.

Edited April 28, 2015 by Tomasu

[MrBucket101]

-snip-

You're running the drives 24/7. They aren't meant for it.

I made the same mistake, but regardless, they should last more than 1yr 24/7. Pull up the smart info on your bad drives if you can and see how many hours they were powered on.

The consumer sea gate drives are only meant for light workloads. 8 hours a day, 5 days a week.

Linus is using the enterprise grade 6tb seagates in the storinator. If he trusts the enterprise drives then so can we. It's just a shame that seagates really cheaps out on their consumer drives. It seems as though they make profits off of consumers who dont want to pony up for WD Drives of the same size. Lots of corners are cut to meet that price point.

[Ramaddil]

Update #4 to server http://linustechtips.com/main/topic/21948-ltt-10tb-storage-show-off-topic/?p=414401

 

Added 20TB to Server updated Raid Card

[TJStamp]

I am in the process of a heavy upgrade to my storage system. Instead of running everything virtualized on a single machine I'm going to have a single purpose storage server providing iSCSI over infiniband to a pair of small VMWare hosts. I've already built the 2 hosts which work well over teamed 4x port gigabit NICs but seriously cant wait for the infiniband cards to arrive.

 

The storage server itself is going to be substantially rejigged to provide a high speed SSD datastore and large HDD datastore for the vmware boxes and a NAS datastore for the rest of the storage. Estimating 4x 512GB SSDs, 4x 6TB enterprise drives (with 128GB cache SSD) and 6x 6TB NAS drives should do a nice job.

 

Will post build pics as it all comes together for now here's one of the VMWare esxi mini Hosts:

 

New VMWare ESXi Host by TJ Stamp, on Flickr

[WPanayides]

Why oh why LTT ::: 

 

 

Need to buy another 2Tb now, sitting on 9.67TB :-/

 

Love you guys and gals 

[Jarsky]

The storage server itself is going to be substantially rejigged to provide a high speed SSD datastore and large HDD datastore for the vmware boxes and a NAS datastore for the rest of the storage. Estimating 4x 512GB SSDs, 4x 6TB enterprise drives (with 128GB cache SSD) and 6x 6TB NAS drives should do a nice job.

 

 

My god this is relevant to my interest. Sounds like a much more extreme version of what one of my friends was doing with NUC's and a storage server. I look forward to seeing more on this pictures, setup, configuration, etc...sounds really well planned out.

 

I'm just awaiting a few more drives to build my new array then will be posting an update to mine soon too - config has changed substantially from what's in the LTT 10TB+ list

[Sousuke]

Looking awesome guys!

My Netgear ReadyNAS Is still going strong with 86% of the space still free ... won't be for long.

[Cheatdeath]

Update #1 Click Here

 

This is my first dedicated server/nas. Previously I had been serving the media and file sharing for my home network with my main desktop machine, it has a 17TB Flexraid setup that is still going but now only for redundancy. The new machine is running the latest FreeNAS with various plugins.

 

Specs below.

 

 

24TB Advertised, I am using a Raidz2 8 disk array, for 16.4TB Usable and can lose 2 drives before game over.

 

FreeNas is running off a 16GB USB 2.0 drive

 

CASE: Fractal Design R5 Black No window
PSU: Corsair CS450M GOLD Cert
MB: Supermicro X10SLL-F
CPU: Intel Xeon E3-1231V3 Haswell 3.4GHz 
HS: Noctua NH-U9S
RAM: 32GB Crucial DDR3 ECC 4x8GB
RAID CARD 1: LSI 9211-8i flashed with P16IT Firmware
HDD: 8x TOSHIBA PH3300U-1I72 3TB 7200 RPM

Fans: 2xFractal 140mm / 2x Noctua 120mm / 3x Noctua 140mm

 

 

 

 

[dfsdfgfkjsefoiqzemnd]

@Cheatdeath : Nice idea, those extra fans on the back of the hard drive tray. Do you notice a difference in hard drive temperature?

Mine are 38 to 40°C (100-104°F) under load with 2 front fans (I ditched the default Fractal fans for 140mm BeQuiet SilentWings2) at 5V.

Based on Google's evaluation of the 100k+ hard drives in their datacenters those temps are perfect, but I'm worried what happens if I turn on 8 drives simultaneously instead of 2. (currently doing the burn-in 2 disks at a time while I wait for the NAS' motherboard to arrive)

[Cheatdeath]

@Cheatdeath : Nice idea, those extra fans on the back of the hard drive tray. Do you notice a difference in hard drive temperature?

Mine are 38 to 40°C (100-104°F) under load with 2 front fans (I ditched the default Fractal fans for 140mm BeQuiet SilentWings2) at 5V.

Based on Google's evaluation of the 100k+ hard drives in their datacenters those temps are perfect, but I'm worried what happens if I turn on 8 drives simultaneously instead of 2. (currently doing the burn-in 2 disks at a time while I wait for the NAS' motherboard to arrive)

 

The problem with the R5 is that the front intake fans don't work so good with the filter on and door closed. Adding the extra 2 120mm lowered my drive temps 5C. Drives are now 35C under load. Also there is not a factory mounting setup to add these fans, I had to use zip ties, which sounds fine till you try to do it lol, larger hands and small zip ties....sigh. Now that I think about it more I bet you could drill some holes and use the rubber Noctua things to mount the fans.

[dfsdfgfkjsefoiqzemnd]

The problem with the R5 is that the front intake fans don't work so good with the filter on and door closed. Adding the extra 2 120mm lowered my drive temps 5C. Drives are now 35C under load. Also there is not a factory mounting setup to add these fans, I had to use zip ties, which sounds fine till you try to do it lol, larger hands and small zip ties....sigh. Now that I think about it more I bet you could drill some holes and use the rubber Noctua things to mount the fans.

I know there are no holes, the first thing I did after seeing your pics was to open up my own R5 and check for mounting holes.

Sounds like I'll be adding a couple of 120mm fans to mine too then. Seeing as I have a cat, I like to keep the filter on the front.

I'll probably drill some holes and use the plastic pins that BeQuiet includes. The only tricky part will be the alignment, I don't want the fan mounting points to interfere with the drive caddies.

Do you use low-profile screws to hold the drive caddies in place or didn't you bother with screwing those in? I have the original thumb screws in there right now and that feels a lot more secure than having the caddies move about.

P.S. : did you do anything about the optical bays? If you remove those covers you can see that there's a serious unfiltered gap around those, where dust can come in through the slits in the front-side and get in the case through the bays themselves.

I plan on using duct tape to close that gap even though I'll have positive pressure anyway.

[Cheatdeath]

I know there are no holes, the first thing I did after seeing your pics was to open up my own R5 and check for mounting holes.

Sounds like I'll be adding a couple of 120mm fans to mine too then. Seeing as I have a cat, I like to keep the filter on the front.

I'll probably drill some holes and use the plastic pins that BeQuiet includes. The only tricky part will be the alignment, I don't want the fan mounting points to interfere with the drive caddies.

Do you use low-profile screws to hold the drive caddies in place or didn't you bother with screwing those in? I have the original thumb screws in there right now and that feels a lot more secure than having the caddies move about.

P.S. : did you do anything about the optical bays? If you remove those covers you can see that there's a serious unfiltered gap around those, where dust can come in through the slits in the front-side and get in the case through the bays themselves.

I plan on using duct tape to close that gap even though I'll have positive pressure anyway.

 

The drive cages and thumb screw holes on my case do not line up and are not usable for me. Do you think this is a major problem for my drives? They feel secure and well mounted so I don't know. I have good positive pressure, probably going to pick up another hot swap bay off monoprice to match the top one though. 

 

And I feel your pain in the cat department, for some reason this is now the place to be for our cats....

[MG2R]

And I feel your pain in the cat department, for some reason this is now the place to be for our cats....

Computers give off heat -> cats like heat -> cats like computers -> cats are on the Internet.

 

It all makes sense!

[alpenwasser]

I'm terribly sorry for the delay, things have been rather busy in college in recent months.

 

-snip-

 

Updated.

 

-snip-

Nice, an actual server chassis!

 

 

-snip-

Updated your config accordingly.

 

 

-snip-

Nice and compact, welcome to the list!

 

I know this Config is Very Special but is´t what I need!

Hey, special systems is what this topic is about!

 

-snip-

Nice and clean, I like it.

 

 

Computers give off heat -> cats like heat -> cats like computers -> cats are on the Internet.

 

It all makes sense!

It is the law of the interwebz, indeed.

[Ramaddil]

Update #4 to server http://linustechtips.com/main/topic/21948-ltt-10tb-storage-show-off-topic/?p=414401

 

Added 20TB to Server updated Raid Card

 

[dfsdfgfkjsefoiqzemnd]

<post removed>

[Cheatdeath]

Time to join, methinks.

 

Cool build man, I think we are on the same page when it comes to our Nas boxes lol. I see you went with the extra fans and looks like zip ties which is what I did also.

 

Have been extremely happy with my build as I am sure you will be, it has really been fun and I feel so much better about my data.

[dfsdfgfkjsefoiqzemnd]

Cool build man, I think we are on the same page when it comes to our Nas boxes lol. I see you went with the extra fans and looks like zip ties which is what I did also.

 

Have been extremely happy with my build as I am sure you will be, it has really been fun and I feel so much better about my data.

Yeah, I went out and bought 2 more fans because of your build.  You didn't patent that idea, did you?

I didn't have trouble attaching them, probably because of the different mounts BeQuiet uses.  Those holes were easy to put a zip-tie through.

 

I might add a fan in the side panel over the weekend to get extra cooling for the card. 

I'm also considering putting a 1TB HDD in the top 5.25" bay to replace the backup share, but then it'd have 11 drives and would need a new name.

[MG2R]

but then it'd have 11 drives and would need a new name.

Honestly, Hendecagon doesn't sound too bad

[alpenwasser]

- snip -

 

Nice, welcome to the club!

 

Honestly, Hendecagon doesn't sound too bad

[dfsdfgfkjsefoiqzemnd]

Nice, welcome to the club!

 

Thanks. 

 

You, @MrBucket101 and @Eniqmatic helped out a bunch though.  I'd probably still be comparing RAID cards if it weren't for all your tips and recommendations. 

 

I added a side fan today to help cool the card before I'd start using it extensively.  For now I'm using one of the original Fractal fans.

I now know it fits (the topmost edge of the fan is a few mm from the bottom of the heatsink's cap), so I'll get a SilentWings2 this weekend and wire it up properly.  6 case fans, that thing is almost as crazy as my main PC.

Fractal somehow thought that you wouldn't need a filter in the side, but luckily I had some spare 200mm filters left from my defunct HAF X.  4 new holes and voila, a filtered side intake. 

 

Small remark though : It's 32.5TB, not 32.0

[alpenwasser]

Thanks. 

 

You, @MrBucket101 and @Eniqmatic helped out a bunch though.  I'd probably still be comparing RAID cards if it weren't for all your tips and recommendations. 

 

I added a side fan today to help cool the card before I'd start using it extensively.  For now I'm using one of the original Fractal fans.

I now know it fits (the topmost edge of the fan is a few mm from the bottom of the heatsink's cap), so I'll get a SilentWings2 this weekend and wire it up properly.  6 case fans, that thing is almost as crazy as my main PC.

Fractal somehow thought that you wouldn't need a filter in the side, but luckily I had some spare 200mm filters left from my defunct HAF X.  4 new holes and voila, a filtered side intake.

 

Happy to be of service.

Small remark though : It's 32.5TB, not 32.0

Ah right, I see you actually use them for data, not the OS. Will fix that tomorrow.

[MrBucket101]

 

@Cheatdeath : Nice idea, those extra fans on the back of the hard drive tray. Do you notice a difference in hard drive temperature?
Mine are 38 to 40°C (100-104°F) under load with 2 front fans (I ditched the default Fractal fans for 140mm BeQuiet SilentWings2) at 5V.

Based on Google's evaluation of the 100k+ hard drives in their datacenters those temps are perfect, but I'm worried what happens if I turn on 8 drives simultaneously instead of 2. (currently doing the burn-in 2 disks at a time while I wait for the NAS' motherboard to arrive)

Your temps are fine, I wouldn't worry about having the extra fans. I'm assuming your burn in puts the drives at full load. The drives in my array are 34C on the hotside (nearest my raid card and expander), and 31C on the cool side.

 

Small remark though : It's 32.5TB, not 32.0

WHOA WHOA WHOA, take it easy kimosabe

 

welcome to #22, just remember whose #21  :rolleyes: