LTT Storage Rankings

[dfsdfgfkjsefoiqzemnd]

We survived!  I now have 3 HGST drives, 21 Seagates, with 1 HGST spare on my desk.  4 more Seagates are still working through RMA.   I'm eyeballing adding all 4 of them as hot spares via the SFF-8088 connector, but need to consider if it's really worth the effort to do that.

Personally I wouldn't add them anymore.

In fact I'd buy a stack of HGST or WD drives and start swapping out the Seagates before they kick the bucket, just like Backblaze did. You know they're going to die pretty soon anyway.

Do a full wipe on the removed drives and sell them on to partially fund the next few new drives.

---

I'm still contemplating moving from FreeNAS to Linux, and really should shut down the NAS so I can flash that <censored> raid card again.

The NAS is running just fine, but the risk of corrupting everything due to a firmware/driver mismatch does bother me.

It honestly blows my mind that they just change the driver, thereby forcing everyone who owns that card (LSI 9211-8i) to flash the firmware

If I make the move to Linux, I might as well host my own domain, mail server and cloud on the NAS so I don't need a webhost etc anymore.

My main concern is the whole UI change though, I'm not a fan of CLI really, and that's putting it mildly. I can stand it just fine for 5 minutes when using codes that I know, but when I need to start looking up things online and start typing them in, using CLI quickly makes me want to throw my keyboard at the screen.

[.:MARK:.]

Personally I wouldn't add them anymore.

In fact I'd buy a stack of HGST or WD drives and start swapping out the Seagates before they kick the bucket, just like Backblaze did. You know they're going to die pretty soon anyway.

Do a full wipe on the removed drives and sell them on to partially fund the next few new drives.

---

I'm still contemplating moving from FreeNAS to Linux, and really should shut down the NAS so I can flash that <censored> raid card again.

The NAS is running just fine, but the risk of corrupting everything due to a firmware/driver mismatch does bother me.

It honestly blows my mind that they just change the driver, thereby forcing everyone who owns that card (LSI 9211-8i) to flash the firmware

If I make the move to Linux, I might as well host my own domain, mail server and cloud on the NAS so I don't need a webhost etc anymore.

My main concern is the whole UI change though, I'm not a fan of CLI really, and that's putting it mildly. I can stand it just fine for 5 minutes when using codes that I know, but when I need to start looking up things online and start typing them in, using CLI quickly makes me want to throw my keyboard at the screen.

 

One of the great things about linux is that you can have a program specifically to be a WebUI for most management needs, Check out WebMin.

[KayKun]

Hay I Updated som thinks at my Server

 

 

 

 

Edited February 6, 2016 by alpenwasser
Fixed link after forum upgrade. :)

[scottyseng]

Hardware
CASE: SuperMicro 846E1-R900B (24 3.5" Bays)
PSU: SuperMicro PWS-920P-SQ 920W (Only one for now, will purchase the other one for redundancy later...I do have the two original 900W PSUs that came with the chassis, but they're too loud)
MB: SuperMicro MBD-X10SRA-O
CPU: Intel Xeon E5-2695V3 (14 core, 28 thread, 2.2GHz, 2.5GHz on Turbo)
GPU1: ZOTAC GeForce GT 1030 2GB GDDR5 (ESXI console output)

Cooler: Cooler Master Hyper D92
RAM: 64GB (16GB x 4) Crucial DDR4 ECC 2133MHz Server RAM
HBA CARD 1: LSI 9211-8i
HBA CARD 2: LSI 9311-8i
SSD 1: 2 TB Crucial MX500 (VM Storage)

SSD 2: 1 TB Crucial MX500 (Steam Game Storage)
SSD 3: 480 GB Sandisk Extreme Pro (VM Storage)
Other Storage: 64 GB Samsung Bar 3.0 Drive (ESXi boot drive)
HDD 1: 6 x 4TB Western Digital Red and 1 x 4TB Western Digital Red Pro (Hot Spare)
HDD 2: 6 x 4TB Western Digital RE SAS and 5 x 4TB HGST SAS
HDD 3: 6 x 8TB Western Digital shucked drives out of WD Elements (5 x WD80EDAZ, 1 x WD80EZAZ)
Total Drive Capacity: 120TB
NIC: 10Gb x 2 Solarflare SFN5122F

OS: VMware ESXi 6.7.0 Update 3 (Want to upgrade to 7, but need to replace the SAS2 HBA with a SAS3 one per compatibility matrix for 7.0)

UPS: Eaton 9130 2000VA with 1 external battery module (Complete overkill is even more overkill)

 

Software and Configuration:
I run VMware ESXi after being tired of stability issues with Windows Server 2012 R2. All hard drives run off the SuperMicro BPN-SAS2-846EL1 backplane (I replaced the original SAS1 Backplane). I plan to buy a 45 bay Supermicro disk shelf next.

Two SAS2 cables from the backplane feed into the LSI HBA 9211-8i.

Previous configuration:

Six 4TB WD Reds are in RAID6. The ten 4TB WD RE SAS drives are in RAID6. This used to have a LSI MegaRAID 9260-8i CV.
However, due to my great luck, I lost my 10 drive RAID6 array and so I have made the decision to migrate to TrueNAS Core (Formerly FreeNAS)

New configuration:
TrueNAS Core has the two HBA cards running into it.
Six 4TB WD Reds are in RAIDZ2 with the WD Red Pro as a hot spare. The six WD RE SAS drives and five HGST SAS Drives are in RAID Z2. The six 8TB WD Elements drives are in RAIDZ2 in one vdev in a pool. My goal is to keep expanding this pool by adding more six drive RAIDZ2 vdevs.
 

Usage:
I use the server for my primary storage and for running home labs.

 

Backup:
Now a priority, I regret not having one when my main array failed. I'm planning to buy a perfect mirror to the main RAIDZ2 pool.

Additional info:

The ten 4TB WD Re drives in RAID6 are my new primary storage location

The six 4TB WD Red drives are also a temp array as I save up to buy the other 4TB WD Re SAS drives.

I do have the original 5 hot swap 80mm fans, but they were too loud and the SuperMicro motherboard only has options for 50%, 75%, and 100% fan speed.

I was too broke to buy a fan controller and couldn't deal with the sound, so I dropped in some BeQuiet Pure Wings Fans (Some 120mm, some 140mm)

 

The final amount of drives will be 24 4TB WD Re SAS drives, but I'm saving up funds for those (Might take a year or two). First priority is another SuperMicro PSU. The six WD Reds will be the backup and moved out of the server once the above 24 drives are bought.
 

Photo's:

 

Spoiler

Inside of the chassis:

 

A fan cooling the RAID card:

 

Flat view:

 

Three 120mm fans in the center behind backplane:

 

Front view:

 

The server lives on top of my art drawing supply drawer:

 

Configuration in LSI MegaRAID Storage Manager:

 

 

 

 

I must say I did not expect the RAID6 array to crack the 1GB/s barrier. This is the newly created RAID6 array of the eight WD Re SAS drives:

 

 

 

[b348xyx]

 

CPU Bench in CPU-Z:

 

 

 

Its interesting seeing performance difference between two older physical cpus IE my Dual Xeon X5650's 12core 24 thread vs your single newer Xeon E5-2695V3 14 core 28 Threads.

 

amazing how cpus have come along all that power in a single cpu !

 

[scottyseng]

-snip-

 

Yep, that's very true. I think what I still can't get over is how low the vCore is on the Xeon. At idle it uses 0.764v, under Intel Burn Test, it uses 0.934v, and under normal full load (typical render), it uses 0.890v.

 

On Cinebench R15, single thread performance is 104cb, and multicore is 1608cb.

 

Compared to my i5-2500K in my main PC, it uses 1.114v at idle and 1.398v (OC'd to 4.6GHz). The Xeon absolutely crushes the 2500K though.

 

However, the 2500K wins in single thread performance due to the higher clock.

 

And yes, this CPU is 100% overkill for a NAS. I have yet to ever see it go above 5% use in normal use even with my content creation programs. I still remember when my original plan was to use a Haswell i3 with a server board...

[AnonymousGuy]

I'm contemplating going all YOLO-fukit and adding full array redundancy in the form of 8 x 8 TB Seagate archive drives in JBOD on a "cheap" Adaptec 2xSAS raid card adapted down to PCI-e x1 since that's all I have left in the server (GPU + 10Gbit card + 9280i only leaves the x1 slot open).  Someone talk me out of spending another $2k on my server.

[RandomNOOB]

Someone talk me out of spending another $2k on my server.

No. You need it. Must. Have.

[looney]

No. You need it. Must. Have.

I agree, he should't spend that money, he should spend 8k on servers instead

[handruin]

I'm contemplating going all YOLO-fukit and adding full array redundancy in the form of 8 x 8 TB Seagate archive drives in JBOD on a "cheap" Adaptec 2xSAS raid card adapted down to PCI-e x1 since that's all I have left in the server (GPU + 10Gbit card + 9280i only leaves the x1 slot open).  Someone talk me out of spending another $2k on my server.

 

 

I'm all about advocating for buying more storage, redundancy, and data hoarding, but I'd have concerns on the performance that those 8 TB archive drives have due to the SMR nature of things.  I get the idea you're doing this for redundancy but you may have issues with the amount of time it'll take to syn/backup/restore.  Have you looked into the performance characteristics of those drives?

[handruin]

Yep, that's very true. I think what I still can't get over is how low the vCore is on the Xeon. At idle it uses 0.764v, under Intel Burn Test, it uses 0.934v, and under normal full load (typical render), it uses 0.890v.

 

On Cinebench R15, single thread performance is 104cb, and multicore is 1608cb.

 

Compared to my i5-2500K in my main PC, it uses 1.114v at idle and 1.398v (OC'd to 4.6GHz). The Xeon absolutely crushes the 2500K though.

 

However, the 2500K wins in single thread performance due to the higher clock.

 

And yes, this CPU is 100% overkill for a NAS. I have yet to ever see it go above 5% use in normal use even with my content creation programs. I still remember when my original plan was to use a Haswell i3 with a server board...

 

 

I over-spec'ed the CPU on my NAS also and I'm glad I did.  I now run a virtual machine on my NAS along with a Plex server.  On top of that it uses spare CPU cycles to run handbrake to convert movies for me.

[Helly]

Here i am again. Quick question which i don't really think needs its own topic... let me know if its better if i do make a new topic for it.

 

My 40TB array degraded 2 days ago. I've tested the drive that failed and it seems fine. The log showed it gave a timeout error. Is this drive actually dying? I've never had much of an idea of what the SMART stuff means but it seems fine

 

 

I've got everything backed up so im in no way concerned, it's also a RAID-6 so another one can fail and all the data would still be fine. Should i prepare for this drive to fail more often or is something else wrong? Is it just some glitch or something that made the drive fail? When i looked before all the SMART stuff was just "-" and no numbers. I don't want to send in a perfectly fine drive for RMA

 

edit: i've got the drive back in and the array is rebuilding...

[scottyseng]

Here i am again. Quick question which i don't really think needs its own topic... let me know if its better if i do make a new topic for it.

 

My 40TB array degraded 2 days ago. I've tested the drive that failed and it seems fine. The log showed it gave a timeout error. Is this drive actually dying? I've never had much of an idea of what the SMART stuff means but it seems fine

 

RAID-6-Degraded-Drive.png

 

I've got everything backed up so im in no way concerned, it's also a RAID-6 so another one can fail and all the data would still be fine. Should i prepare for this drive to fail more often or is something else wrong? Is it just some glitch or something that made the drive fail? When i looked before all the SMART stuff was just "-" and no numbers. I don't want to send in a perfectly fine drive for RMA

 

edit: i've got the drive back in and the array is rebuilding...

 

What are the specs of the server and models of the drives? You might be using consumer drives that aren't made for RAID.

[Helly]

Im sorry but at least a tiny bit of searching would have had you learn they are WD reds connected to an areca 1880i... They are made for raid....

[scottyseng]

Im sorry but at least a tiny bit of searching would have had you learn they are WD reds connected to an areca 1880i... They are made for raid....

 

Sorry, I had missed your name when looking through the server lists on page 1. Your drive there appears to be in perfect condition and I don't honestly see why the drive would be dropping from the array unless it's a glitch of some kind.

[AnonymousGuy]

Here i am again. Quick question which i don't really think needs its own topic... let me know if its better if i do make a new topic for it.

 

My 40TB array degraded 2 days ago. I've tested the drive that failed and it seems fine. The log showed it gave a timeout error. Is this drive actually dying? I've never had much of an idea of what the SMART stuff means but it seems fine

 

RAID-6-Degraded-Drive.png

 

I've got everything backed up so im in no way concerned, it's also a RAID-6 so another one can fail and all the data would still be fine. Should i prepare for this drive to fail more often or is something else wrong? Is it just some glitch or something that made the drive fail? When i looked before all the SMART stuff was just "-" and no numbers. I don't want to send in a perfectly fine drive for RMA

 

edit: i've got the drive back in and the array is rebuilding...

 

 

I've had shitty SATA cables cause drive drop-reappear on me before.

[b348xyx]

I've had shitty SATA cables cause drive drop-reappear on me before.

 

Just wanted to add i had this issue with couple drives which were coming up disconnected randomly (which was very annoying!)... Lots trouble shooting later...  i replaced all my sata/sas cables with new high quality cables.. and never had issue since. also have cables which have clip so they clip to motherboard and hdd..

 

Ever since i always use high quality cables..

 

may not be this but is amazing that lot people over look a good quality cable in expensive setups sometimes. 

[alpenwasser]

Update: (((((((((((((((Update #6 Added an additional 6TB/Replaced MB/Scrapped outside Fan controller/Added 2U for future Expansion))))))))))))))

 

Updated

 

- snip -

Very nice, welcome to the list!

[Downloadski]

Nice thread.

I run 4 nas servers with freebsd/zfsguru.

1 with 10x4 and 10x3 TB (2 raid-z2)

2 with 10x4 and 10x3 TB (2 raid-z2)

3 with 20x4 TB (2 raid z2)

4 with 5x4 TB (raid z1)

The 4 TB drives are hgst 7k4000

The 3 TB drives are toshiba DT01ACA300 (7k3000 of hgst)

Split it over multiple servers and pools as i not want everything online all the time.

10GE ethernet via x520-da2 cards with DA cupper cables.

[b348xyx]

Nice thread.

I run 4 nas servers with freebsd/zfsguru.

1 with 10x4 and 10x3 TB (2 raid-z2)

2 with 10x4 and 10x3 TB (2 raid-z2)

3 with 20x4 TB (2 raid z2)

4 with 5x4 TB (raid z1)

The 4 TB drives are hgst 7k4000

The 3 TB drives are toshiba DT01ACA300 (7k3000 of hgst)

Split it over multiple servers and pools as i not want everything online all the time.

10GE ethernet via x520-da2 cards with DA cupper cables.

 

We need the photos and print screens

[Downloadski]

Took the photo's offline. The anti piracy heat gets worse here.

[alpenwasser]

Nice thread.

I run 4 nas servers with freebsd/zfsguru.

1 with 10x4 and 10x3 TB (2 raid-z2)

2 with 10x4 and 10x3 TB (2 raid-z2)

3 with 20x4 TB (2 raid z2)

4 with 5x4 TB (raid z1)

The 4 TB drives are hgst 7k4000

The 3 TB drives are toshiba DT01ACA300 (7k3000 of hgst)

Split it over multiple servers and pools as i not want everything online all the time.

10GE ethernet via x520-da2 cards with DA cupper cables.

 

Nice! Since we're trying to keep the thread consistent, feel free to use looney's template

from this post and I'll add you to the list if you want.

[Downloadski]

No i want not to be in the list..

I like to read this topic for nice servers.

My norco case had Original 5x 80mm fans very loud in the case, changed it to 3x120 mm fans (noctua)

In server 3 i took out the fans plate and put in 3x140mm fans (noctua) without a plate. Noise is now better with server 3.

So updates will follow for server 1 and 2, perhaps even the norco cases will be changed to rivier case.

Had doa issues with 2 backplanes for the hdd's in the borco cages. They were swapped under warrantee, but still have 4 hdd's with a lot of cable errors

[AnonymousGuy]

Update #63 part 2/5

 

I've had 3 drives die in slot 9 of my raid array.  One was original, so the failure was probably legitimate.  I swapped in a recert one I had on my desk...dies within 12 hours of rebuild finishing...swapped in another recert one where now slot 0 also died putting my array in a critical state (1 more failure = data loss).  slot 0 rebuilds fine then slot 9 tries to rebuild and inst-fails.

 

Either Seagate is churning out shit for their recert drives or my SFF-8087 SATA cable is bad.  I have 2 HGST Deskstars being delivered today and another cable. We'll get to the bottom of it.  I figure I'm going to mix HGST Ultrastars with HGST Deskstars and see if I notice any difference in reliability.  1M MTBF vs. 2M MTBF but bother have rotational vibration sensors so it might not be worth the $50 premium to upgrade to full enterprise grade.

 

And to reiterate..holy fucking balls Seagate 3TB are bad.  20 drives and with 5 cold spares on my desk I can't even keep up RMA-ing them (and that's *with* the $12 advanced cross ship RMA) and now have 0 spares.

 

EDIT: when I say a drive "dies", I mean the raid controller has marked it as failed.  I have no idea really what criteria the raid controller uses to determine a drive has failed (maybe SMART, maybe communication issues, maybe drive data is corrupted, etc), but even manually trying to put the drive back in service it will immediately get kicked back out as failed. 

[Gronnie]

I think you should be safe to save the $50