Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
looney

LTT Storage Rankings

Recommended Posts

This might sound stupid but what the hell do you guys even store here? 

 

Like all the movies in the world or what?!?!?

 

For me, I built my NAS (see signature) as a learning device to get better with ZFS management (personal and job reasons).  In addition to that I use mine to backup all my home PCs to it using the local (free) features of CrashPlan.  My fiancee's PC is a photo-editing workstation for her personal business that has Terabytes of Canon RAW files from the work she does.  Those get backed up to my (NAS 5+TB and growing) and also to CrashPlan cloud as an off-site protection.  The cloud backup takes considerably longer so having a local option helps to stage the backups as we wait for the cloud to sync.

 

My personal workstation also has a couple TB of Canon RAW files for my own personal photo hobby along with other content.  I also have a large collection of FLAC music that I've ripped from my personal CD collection over the years.  I'm also working on ripping all my DVD's and Bluray into mp4 files to serve up to my TV via Plex.  My workstation probably qualifies for this thread in storage space but since it's not a dedicated storage server I can't list it.  :-)  I also run multiple Linux VMs and I plan to move those over to my NAS to reduce volatility from my workstation.  I host some basic gaming servers for friends and mumble chat server on my Linux VM via a domain name I registered.  Those will eventually get hosted on the same machine I use as a NAS since it has decent cpu power.

 

I'm working on revamping my old NAS (8 x 1.5TB | Dell Perc 6I | Server 2012 | AMD 4600+) to ship off to my dad's house so I can replicate my data there and also give him a place to backup some data locally.  I haven't started that project yet and not sure how I want to manage that.  One of the 1.5TB drives failed recently and I can't find a replacement to match.  I plan to convert the system into Linux with ZFS and make the remaining 7 into its own pool for slower storage.  I'll possibly buy a few 4-6TB drives to add on to that system in a different ZFS pool.


Workstation 1: Intel i7 4790K | Thermalright MUX-120 | Asus Maximus VII Hero | 32GB RAM Crucial Ballistix Elite 1866 9-9-9-27 ( 4 x 8GB) | 2 x EVGA GTX 980 SC | Samsung 850 Pro 512GB | Samsung 840 EVO 500GB | HGST 4TB NAS 7.2KRPM | 2 x HGST 6TB NAS 7.2KRPM | 1 x Samsung 1TB 7.2KRPM | Seasonic 1050W 80+ Gold | Fractal Design Define R4 | Win 8.1 64-bit
NAS 1: Intel Intel Xeon E3-1270V3 | SUPERMICRO MBD-X10SL7-F-O | 32GB RAM DDR3L ECC (8GBx4) | 12 x HGST 4TB Deskstar NAS | SAMSUNG 850 Pro 256GB (boot/OS) | SAMSUNG 850 Pro 128GB (ZIL + L2ARC) | Seasonic 650W 80+ Gold | Rosewill RSV-L4411 | Xubuntu 14.10

Notebook: Lenovo T500 | Intel T9600 | 8GB RAM | Crucial M4 256GB

Link to post
Share on other sites

This might sound stupid but what the hell do you guys even store here? 

 

Like all the movies in the world or what?!?!?

I have a very large media collection that I've built up over the past 10 years. I have around 1300 Movies, 150 of which are DVD quality, with the rest primarily being 1080p. The same goes for TV shows, except I rip those in 720p for size constraints. I tend to wait till shows are over, and I'll buy the bluray off craigslist or ebay for cheap, and rip it to my server. I love binge watching.

 

like @handruin, I also run quite a few VM's along with a database server.

 

I also have a VPN network established between me and the rest of my family. If anyone wants access to my library, I simply make them purchase a chromecast and a mikrotik router. I'll setup the router for direct VPN access, and then I show them how to use Plex Sync to access my library. I don't have a big enough Upload Connection to stream 1080p, so its really the only option unless they want to deal with crap tons of buffering lol

 

I don't mind paying for movies and tv shows, I just love being able to access it whenever/wherever I want.

 

My server has also provided me with valuable experience that I'm able to put on my resume. Over the past 8 years I have forced myself to learn linux and now I'm VERY proficient with it. A skill that is very applicable in the tech industry. This combined with my familiarity with hardware RAID and high capacity storage arrays; I believe it gives me a leg up over other interview candidates. Many times when I am interviewing for jobs I will be asked about my server. It may not be directly applicable to the job at hand, but I feel that it shows off some ingenuity and desire to learn. Another skill which is very applicable.

 

It all started as a pet project, but I'm very proud of what it's grown into.

Link to post
Share on other sites

@MrBucket101 & @handruin

 

Thanks for explaining to me what you use your servers for :D

 

Now that I've read this, I might make my own NAS server now, as I have wanted to for a while. Might be able to clear my DVD / CD collection up a little as there's alot of space needed for all of this content.

Seeing that it looks good on your resume, I might learn how to use Linux but I own a VPS right now, that has Linux on it, accessed with PuTTY etc.

 

Anyways, I'll see what happens in the next year or so :D


Gamer & PC Enthusiast


Gaming Community's TeamSpeak: ts.the-eux.uk


If I've helped you out, Click that "like" button to show me some love :)


Link to post
Share on other sites

Hardware

CASE: Silverstone DS380

PSU: Silverstone ST45SF-G

MB: Supermicro A1SRi-2758F

CPU: Intel Atom C2758

HS: Stock Supermicro heatsink

RAM: 16GB DDR3 SODIMM 1333

RAID CARD 1: HP Smart Array P410 512MB With BBU

SSD: Kingmax SMP35 240GB

HDD 1: 5x 2TB Western Digital Green

Software and Configuration:

The server is running ESXi 5.5 and currently hosts 3 virtual machines with more to be added once the RAM is upgraded.

Usage:

I use the server for storage of audio recordings from gigs (24 individual channels and a stereo channel, all in raw WAV audio) and also to store TV Shows and Movies for watching on the media center pcs, the storage server is running on Windows Server 2012 R2. Two other virtual machines run linux, One runs Turnkey MineOS, the other Runs Ubuntu server to run the Mumble server.

Backup:

The server is backed up via an LTO 2 Tape drive on a workstation set up for backup running VEEAM

Additional info:

This server was built specifically for low power usage while still being powerful enough

Photo's:

To be Added once I return home and can take pictures (currently away for work).

Im finally back home now so I took a few pics of my server to add to the album above.

Link to post
Share on other sites

Update:

 

I replaced all my 2.5" 500G WD blacks with 2.5" 1TB Reds.

 

DSCN0367.jpg

 

DSCN0369.jpg

 

This is just really interesting to me for some reason. :)

 

Any chance you have a picture of it in whatever rig it belongs too? :)


Desktop - CPU Core i7 4770k - MOBO ASUS Maximus VI Hero - GPU Nvidia GTX770 - RAM 16Gb Corsair Dominator GT - PSU Corsair TX750V2 - Storage OCZ Vertex III 120Gb - Case Fractal Design Arc Midi R2


unRAID - CPU Phenom X6 1090T - MOBO ASUS Sabertooth 990FX R1 - RAM 16Gb Kingston ECC - PSU EVGA 650G1 - Storage 5x3TB WD Red - Case NZXT Source 210

Link to post
Share on other sites

Just a general FYI, I haven't forgotten about this, but I'm in the middle of exams at the moment and will then be away for a while, so I'll update the list when I get back.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites
Posted · Original PosterOP

New 152TB server:

 

Hardware I
This is the hardware of the Storage node:
Case 1: Supermicro SuperServer 6027TR-DTRF
Case 2: Supermicro SC847 E16-RJBOD1
PSU Case 1: 1280W Redundant High-efficiency Digital Power Supplies w/ PMBus 1.2
PSU Case 2: 1400W high-efficiency (1+1) redundant power supply with PMBus
MB: Supermicro X9DRT-HF
CPU: 2x Intel Xeon E5-2609V2
HS: Stock Supermicro heatsink
RAM: 4x Kingston 8GB 1866MHz DDR3 ECC Reg CL13 DIMM
RAID CARD: LSI MegaRAID SAS 9286 8e SGL 8 + LSIiBBU09
SSD 1: 2x Samsung 850 Pro 512GB (RAID 1)

SSD 2: 4x Samsung 850 Pro 1TB (RAID 10)
HDD: 37x 4TB Seagate Barracuda Desktop (ST4000DM000)

NIC: Supermicro AOC-STGN-I2S 10Gbit NIC 
 
Hardware II
This is the hardware of the EXS node which is in the same chassis as the storage node (storage not counted for 10TB+)
Case: Supermicro SuperServer 6027TR-DTRF
MB: Supermicro X9DRT-HF
CPU: 2x Intel Xeon E5-2650V2
HS: Stock Supermicro heatsink
RAM: 8x Kingston 16GB 1866MHz DDR3 ECC Reg CL13 DIMM
NIC: Supermicro AOC-STGN-I2S 10Gbit NIC & 2x Quad Gbit NIC

 

Software and Configuration:
My server is running windows server 2012R2 on which I have 2 arrays.
Array one consist of 4 4TB drives in RAID 5 and array two consists of 32 4TB drives in RAID 60.
The remaining 4TB drive is being used as a global hot spare.
 
ss+(2015-02-02+at+09.05.27).png

Usage:
I use the storage for movies and series, i have media players around the house that access it.
Its also for backing up my computers.

Backup:
The is arrays backed up to LiveDrive, this is a online cloud backup service. (I have 100/100 Mbps connection)

Photo's:
IMG_0230%20(Custom).JPG
 
 
IMG_0246%20(Custom).JPG
 
 
IMG_0267%20(Custom).JPG
 
 
IMG_0529%20(Custom).JPG
 
 
IMG_0453%20(Custom).JPG
 
ss+(2015-02-02+at+09.11.53).png
 
1a1e9f3136.png
ss+(2015-02-02+at+09.14.01).png


Respect the Code of Conduct!

>> Feel free to join the unofficial LTT teamspeak 3 server TS3.schnitzel.team <<

>>LTT 10TB+ Topic<< | >>FlexRAID Tutorial<<>>LTT Speed wave<< | >>LTT Communies and Servers<<

Link to post
Share on other sites

Very nice setup.  What made your decision to go with RAID 60 for this environment?  Are 16 drives the largest contiguous span in RAID-6 you can get from the LSI controller?

 

That's also a lot of cores/computer for a storage server.  Are you really "only" using it for streaming media and backups?  You could easily run several VMs on that setup for various tasks.  None the less I love lots of superfluous hardware so don't take my questions as insult/negative.  :-)


Workstation 1: Intel i7 4790K | Thermalright MUX-120 | Asus Maximus VII Hero | 32GB RAM Crucial Ballistix Elite 1866 9-9-9-27 ( 4 x 8GB) | 2 x EVGA GTX 980 SC | Samsung 850 Pro 512GB | Samsung 840 EVO 500GB | HGST 4TB NAS 7.2KRPM | 2 x HGST 6TB NAS 7.2KRPM | 1 x Samsung 1TB 7.2KRPM | Seasonic 1050W 80+ Gold | Fractal Design Define R4 | Win 8.1 64-bit
NAS 1: Intel Intel Xeon E3-1270V3 | SUPERMICRO MBD-X10SL7-F-O | 32GB RAM DDR3L ECC (8GBx4) | 12 x HGST 4TB Deskstar NAS | SAMSUNG 850 Pro 256GB (boot/OS) | SAMSUNG 850 Pro 128GB (ZIL + L2ARC) | Seasonic 650W 80+ Gold | Rosewill RSV-L4411 | Xubuntu 14.10

Notebook: Lenovo T500 | Intel T9600 | 8GB RAM | Crucial M4 256GB

Link to post
Share on other sites
Posted · Original PosterOP

Very nice setup.  What made your decision to go with RAID 60 for this environment?  Are 16 drives the largest contiguous span in RAID-6 you can get from the LSI controller?

 

That's also a lot of cores/computer for a storage server.  Are you really "only" using it for streaming media and backups?  You could easily run several VMs on that setup for various tasks.  None the less I love lots of superfluous hardware so don't take my questions as insult/negative.  :-)

I prefer the extra redundancy with RAID 60, the card has no problems with running them all in RAID 6 though.

I also tested RAID 6 and RAID 60 did give me improved read speeds.

 

As far as the cores goes, the storage node needed 2 CPU's to make all the needed components on the motherboard work so I just got some cheap quad core non hyper thread Xeons for the storage node.

 

Right now I'm not virtualizing on the storage node as I have the ESX node for that (Hardware II)


Respect the Code of Conduct!

>> Feel free to join the unofficial LTT teamspeak 3 server TS3.schnitzel.team <<

>>LTT 10TB+ Topic<< | >>FlexRAID Tutorial<<>>LTT Speed wave<< | >>LTT Communies and Servers<<

Link to post
Share on other sites
 

Not to make @looney feel inadequate, but I'm about to upgrade my server from 2TB usable storage to 7TB usable storage, so you have some stiff competition! 

 

*Weeps in corner*


CPU i5 4430 3Ghz | Ram: 16GB DDR3 1600 | GPU: GTX 650 Ti 1GB | Mobo: H87N-Wifi | Case: White Bitfenix Prodigy | Boot Drive: 120GB 840 Evo (Mac OS X) 120gb OCZ Vertex 3 (Windows) | Games Drive: 640GB WD Green | OS: Windows 8 & OS X 10.9.1

I love all technology. The perfection of macs for my designer side, and the hardware and fun of tinkering on the of the pc side. We can have it all, just not at the same time.

Link to post
Share on other sites

Not to make @looney feel inadequate, but I'm about to upgrade my server from 2TB usable storage to 7TB usable storage, so you have some stiff competition! 

 

*Weeps in corner*

@looney, you better add another 30 HDDs, just to be sure. :P


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites
Posted · Original PosterOP

@looney, you better add another 30 HDDs, just to be sure. :P

I'll have to get a second jbod :P


Respect the Code of Conduct!

>> Feel free to join the unofficial LTT teamspeak 3 server TS3.schnitzel.team <<

>>LTT 10TB+ Topic<< | >>FlexRAID Tutorial<<>>LTT Speed wave<< | >>LTT Communies and Servers<<

Link to post
Share on other sites

I'll have to get a second jbod :P

I'm putting your new server into the system now, question: Is the old one still active and should be left in, or is it dismantled, or its config changed?

EDIT: Are the SSDs for caching (and should be counted), or are they for the OS only?


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites
Posted · Original PosterOP

Old system is gone for now.

I'll revive it later as a test system for writing tutorials.

Ssd's are os only so 148tb


Respect the Code of Conduct!

>> Feel free to join the unofficial LTT teamspeak 3 server TS3.schnitzel.team <<

>>LTT 10TB+ Topic<< | >>FlexRAID Tutorial<<>>LTT Speed wave<< | >>LTT Communies and Servers<<

Link to post
Share on other sites

Update:

 

I replaced all my 2.5" 500G WD blacks with 2.5" 1TB Reds.

 

Updated. :)

 

Old system is gone for now.

I'll revive it later as a test system for writing tutorials.

Ssd's are os only so 148tb

Added, master looney. :P

@MrBucket101: I think caching SSDs should be counted, but I'll need to figure something out first, will update you when I've done so.


BUILD LOGS: HELIOS - Latest Update: 2015-SEP-06 ::: ZEUS - BOTW 2013-JUN-28 ::: APOLLO - Complete: 2014-MAY-10
OTHER STUFF: Cable Lacing Tutorial ::: What Is ZFS? ::: mincss Primer ::: LSI RAID Card Flashing Tutorial
FORUM INFO: Community Standards ::: The Moderating Team ::: 10TB+ Storage Showoff Topic

Link to post
Share on other sites

 

 

If you don't care to read my horror story skip down to the second bolded sentence.

 

Things weren't looking good for me. I nearly lost ALL my data, and the root cause of it all was my decision to use cheap consumer drives in an always on RAID environment. Heed my warning, STAY AWAY FROM SEAGATE ST3000DM001

 

A little background story. I originally bought 8 - 3TB ST3000DM0001 and was running them in RAID 6. Late november early december I ran into a massive string of failures. I lost 3 drives in 3 weeks. I had a spare availiable, and thankfully all of them were covered under warranty, so I got them replaced for free.

 

2 weeks ago I started to run out of space on my array. I had 1TB left out of the 16.3TB. So I got rid of my dedicated hot-spare and added it to the array. This is where shit hit the fan. The expansion should have only taken 3 days as it has done so in the past, but the expansion took a grand total of 9 days. Which definitely had me worried. But I wasn't going to disturb anything. According to the logs, 10 minutes before the expansion finished, another drive died on me. (that makes 4 of the original 8 within 2 years) I didn't really think much of it at the time. The expansion finished and megaraid reported the new size at 19TB, it was just degraded.  So this time I bought a nice HGST 3TB NAS drive, and put that in to replace the failed drive. But when the rebuild finished....things took a turn for the worse. The OS was only able to see the array as 16.3TB, it was as if the expansion hadn't actually worked. I know when you expand you also have to move the GPT data to the end of the disk and then expand the partition. However the OS reported the drive was only 16.3TB.

 

On top of this, the alarm on the card would not go off. I scoured the logs and saw no problems that had not been taken care of. Yet something was clearly wrong. The alarm would not turn off, unless I did so manually, and the OS could not see the expansion.

 

At this point, I had no effing idea what was going on with my array. But the OS could still run, and I could still access my data. So I left the server alone for a couple days while I did some research. Unfortunately at this point though, I was now getting non-stop kernel panics. The OS was reporting that the device had taken longer than 120s to respond, and so the kernel halted waiting for the array to respond back. The entire OS locked up and I was forced to hard reboot. 

 

This kept up for the next 2 days and the random kernel panics and hard resetting corrupted my filesystem...then the panic set in.

 

I was able to repair the damage done, and verify that my data was in tact. But I just couldn't take this any longer. My stomach was in knots from nearly loosing all of my data from the past 8 years.

 

I snapped and bought new drives. I don't like to do this, but I had to throw in the towel. The system had beaten me and I could not stand to lose my data.

 

SOOOOO on to the juicy stuff.

 

I bought 8 - Hitachi HGST 4TB NAS drives. I picked these over the WD RED for a number of reasons. These drives are comparable to the WD RED PRO line, but they do not have the price tag to match. The drives are 7200rpm, and come with a rotational vibration sensor. Something the WD RED's don't. On top of this the drive supports TLER, is rated for 1million power on hours, and comes with a 3 year warranty. All identical to the WD RED. All this doesn't even come at a much larger price premium. On newegg, the WD RED 4TB drive costs $166, where these drives are around $175. I managed to get 5 of my 4TB drives on sale for $160, and then I had to pay $175 for the last 3 since the coupon had a limit of 5 drives.

 

Since I was buying new drives, I decided to buy another SSD so that I could run my SSD Cache in RAID 1 and safely enable write caching. I just got a crucial MX100. My other SSD is an OCZ Vertex 4. So I wasn't too worried about matching performance. Basically, the cache drives are used to speed up read/write speeds from my array. Data read from the array once is moved onto the SSD cache for faster access times. When I am writing data, the controller will write the data to the SSD and report back to the OS the transfer is done, and then it will move the data from the SSD to the disks at a more convenient time. This feature REALLY speeds things up. I did some rough testing, and I was getting around 1.2GB/s write speed, and 1.7GB/s read speed. I didn't do much to validate the results, but the numbers were enough to convince me it was working. Previously, I was getting around 600MB/s write speed, and 800MB/s read speed. Fun fact, according to the manual my raid card tops out at 1.8GB/s :P

 

I put the new drives in my system and set them up in RAID 6 and used rsync to transfer the content from the bad array, to the new one. rsync is nice in that it will hash check the new file against the original to make sure there was no issue with the transfer.

 

Once I had migrated my data off of the bad array, I booted back into the cards bios and deleted the array. Soon as I did that the alarm finally stopped going off. Whatever it was that had went wrong, had now been fixed...with money lol

 

My OS drive got corrupted with all the kernel panics as well, and i didn't care to go rescue my data from it. I had an image from mid january 2015 that I just copied back onto the system.

 

I plan on taking the crappy 3TB drives and individually filling them up with my data; and then I'm going to put them back in the box my new hard drives came in, and store them in my basement. This will give me a dated offline backup incase of any future catastrophe.

 

I also took the 3TB HGST NAS drive, and I replaced the previous 2TB drive I was using for downloads.

 

To help make things easier, for those that haven't been keeping up with everything. Here is my current configuration.

 

Hardware:

SSD 1: 2 x 120GB SAMSUNG 840 EVO (RAID 1 for the OS)

SSD 2: 1 x 128GB OCZ Vertex 4

SSD 3: 1 x 128GB Crucial MX100 (RAID 1 w/ SSD 2 for SSD cache)

HDD 1: 1 x 320GB WD Black 2.5" (for storing VM disks)

HDD 2: 1 x 250GB Hitachi Travelstar 5k500 2.5" (for backing up my /home/ folder and some application config files)

HDD 3: 1 x 3TB HGST Deskstar NAS

HDD 4: 8 x 4TB HGST Deskstar NAS (RAID 6)

 

(36.066TB ~36TB :D)

 

And now for the pr0n!!!

 

2015-02-10145213_zps9b297725.jpg

 

2015-02-10145224_zps52610a57.jpg

 

2015-02-10153737_zpse980587a.jpg

 

2015-02-10153830_zps09cb7c8e.jpg

 

The graveyard :P

 

2015-02-10154242_zpsd279c241.jpg

 

 

 

Spoiler

SERVER---


Server IP: 10.254.6.250
Server Name: SERVER
OS name: Linux
OS Version: 3.13.0-44-generic
OS Architecture: x86_64
Driver Name: megaraid_sas
Driver Version: 06.700.06.00-rc1
Application Version: MegaRAID Storage Manager - 14.08.01.02

HARDWARE---
Controller: LSI MegaRAID SAS 9260-8i(Bus 3,Dev 0)
Status: Optimal
Firmware Package Version:12.15.0-0205
Firmware Version: 2.130.403-3835
BBU: YES
Enclosure(s): 1
Drive(s): 12
Virtual Drive(s): 3

BBU---
BBU Type: IBBU
BBU Status: Optimal

Enclosures---
PRODUCT NAME TYPE STATUS
HPSASEXPCard Ses OK

Drives---
CONNECTOR PRODUCT ID VENDOR ID STATE DISK TYPE CAPACITY POWER STATE
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 CrucialCT128MX1 ATA Online SATA 118.719 GB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 SamsungSSD840 ATA Online SATA 111.281 GB On
Port 0 - 3 SamsungSSD840 ATA Online SATA 111.281 GB On
Port 0 - 3 OCZVERTEX4 ATA Online SATA 118.719 GB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On
Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Virtual Drive(s):---
TARGET ID NAME CAPACITY STATE RAID LEVEL MegaRAID RECOVERY
3 - 21.829 TB Optimal RAID 6 NO
1 - 118.719 GB Optimal RAID 1 NO
2 - 111.281 GB Optimal RAID 1 NO

Link to post
Share on other sites

Great story MrBucket01, I have had a similar experience from the same drives, I have had one just die out of no where, the seconds started getting bad sectors in September and now the warranty replacement has the same behaviour!

 

So I have just replaced the final one with another 4TB seagate (we will see how this pans out) but I am confident it should be better quality.

 

I have updated my original post as I now have a 14TB array


| Intel Core i7 7600K | ASRock H170M ITX | 16GB DDR4 | OCZ GTX1060 6GB  |
| Crucial M4 256GB | Samsung 850 EVO 250GB  | Antec True Power 650 | Fractal Design Node 304 | Coolermaster 212+ |

Link to post
Share on other sites

Great story MrBucket01, I have had a similar experience from the same drives, I have had one just die out of no where, the seconds started getting bad sectors in September and now the warranty replacement has the same behaviour!

 

So I have just replaced the final one with another 4TB seagate (we will see how this pans out) but I am confident it should be better quality.

 

I have updated my original post as I now have a 14TB array

 

The biggest mistake I made was not reading the fine print.

 

The ST3000DM001 is only rated for usage 8 hours a day, 5 days a week; or 2080 hours a year. I don't have the exact numbers, but most of the drives died at around the 9000 hour mark. ~1 year of 24/7 usage. Right around 4x the recommended usage per year; so they were definitely being abused.

 

I had 5 of the early 7200.11 1TB models from seagate, and they all had the defective firmware that would cause the drives to drop out and cause the OS to hang intermittently. The firmware updates helped, but it was still pretty bad. I gave seagate another chance because the price on these drives was competitive, and at the time of purchase they had 4.5 stars on newegg.

 

Needless to say, I'll pretty much never touch a seagate drive again. 

Link to post
Share on other sites

Does anybody have any experience with the new seagate archive 8TB drives? curious if I would notice any performance difference especially in my array (storage spaces) with my WD greens.

Link to post
Share on other sites

Does anybody have any experience with the new seagate archive 8TB drives? curious if I would notice any performance difference especially in my array (storage spaces) with my WD greens.

 

You can read above for my horrible track record with seagate. But i'm definitely biased.

 

With that said, looking at the specs sheet. The specs don't seem terrible, 800k MTBF is 200k hours short of the MTBF on WD REDS and HGST NAS, but it is rated for a 24/7 workload and comes with a 3yr warranty.  I also couldn't find any mention of TLER, or similarly named feature, this doesn't confirm the drives do/don't have it...but it is worth mentioning. Had I not been burned 2x already I might be willing to give one of those drives a chance. 

 

As for your question on performance, the spindle RPM is not listed anywhere on their website or spec sheet. Which most likely means it's not 7200rpm, probably 5400rpm.  But given you are on WD Greens I doubt you will see any less performance...usually RAID on WD Greens is frowned upon. 

 

Sometimes being on the bleeding edge isn't always a good thing. SMR is extrememly new, and those drives haven't made their way into the masses yet. I would wait until all the tech sites get their hands on them and do reviews and give us some actual numbers and information.

Link to post
Share on other sites

This will officially be my 2nd update.

If you don't care to read my horror story skip down to the second bolded sentence.

Things weren't looking good for me. I nearly lost ALL my data, and the root cause of it all was my decision to use cheap consumer drives in an always on RAID environment. Heed my warning, STAY AWAY FROM SEAGATE ST3000DM001

A little background story. I originally bought 8 - 3TB ST3000DM0001 and was running them in RAID 6. Late november early december I ran into a massive string of failures. I lost 3 drives in 3 weeks. I had a spare availiable, and thankfully all of them were covered under warranty, so I got them replaced for free.

2 weeks ago I started to run out of space on my array. I had 1TB left out of the 16.3TB. So I got rid of my dedicated hot-spare and added it to the array. This is where shit hit the fan. The expansion should have only taken 3 days as it has done so in the past, but the expansion took a grand total of 9 days. Which definitely had me worried. But I wasn't going to disturb anything. According to the logs, 10 minutes before the expansion finished, another drive died on me. (that makes 4 of the original 8 within 2 years) I didn't really think much of it at the time. The expansion finished and megaraid reported the new size at 19TB, it was just degraded. So this time I bought a nice HGST 3TB NAS drive, and put that in to replace the failed drive. But when the rebuild finished....things took a turn for the worse. The OS was only able to see the array as 16.3TB, it was as if the expansion hadn't actually worked. I know when you expand you also have to move the GPT data to the end of the disk and then expand the partition. However the OS reported the drive was only 16.3TB.

On top of this, the alarm on the card would not go off. I scoured the logs and saw no problems that had not been taken care of. Yet something was clearly wrong. The alarm would not turn off, unless I did so manually, and the OS could not see the expansion.

At this point, I had no effing idea what was going on with my array. But the OS could still run, and I could still access my data. So I left the server alone for a couple days while I did some research. Unfortunately at this point though, I was now getting non-stop kernel panics. The OS was reporting that the device had taken longer than 120s to respond, and so the kernel halted waiting for the array to respond back. The entire OS locked up and I was forced to hard reboot.

This kept up for the next 2 days and the random kernel panics and hard resetting corrupted my filesystem...then the panic set in.

I was able to repair the damage done, and verify that my data was in tact. But I just couldn't take this any longer. My stomach was in knots from nearly loosing all of my data from the past 8 years.

I snapped and bought new drives. I don't like to do this, but I had to throw in the towel. The system had beaten me and I could not stand to lose my data.

SOOOOO on to the juicy stuff.

I bought 8 - Hitachi HGST 4TB NAS drives. I picked these over the WD RED for a number of reasons. These drives are comparable to the WD RED PRO line, but they do not have the price tag to match. The drives are 7200rpm, and come with a rotational vibration sensor. Something the WD RED's don't. On top of this the drive supports TLER, is rated for 1million power on hours, and comes with a 3 year warranty. All identical to the WD RED. All this doesn't even come at a much larger price premium. On newegg, the WD RED 4TB drive costs $166, where these drives are around $175. I managed to get 5 of my 4TB drives on sale for $160, and then I had to pay $175 for the last 3 since the coupon had a limit of 5 drives.

Since I was buying new drives, I decided to buy another SSD so that I could run my SSD Cache in RAID 1 and safely enable write caching. I just got a crucial MX100. My other SSD is an OCZ Vertex 4. So I wasn't too worried about matching performance. Basically, the cache drives are used to speed up read/write speeds from my array. Data read from the array once is moved onto the SSD cache for faster access times. When I am writing data, the controller will write the data to the SSD and report back to the OS the transfer is done, and then it will move the data from the SSD to the disks at a more convenient time. This feature REALLY speeds things up. I did some rough testing, and I was getting around 1.2GB/s write speed, and 1.7GB/s read speed. I didn't do much to validate the results, but the numbers were enough to convince me it was working. Previously, I was getting around 600MB/s write speed, and 800MB/s read speed. Fun fact, according to the manual my raid card tops out at 1.8GB/s :P

I put the new drives in my system and set them up in RAID 6 and used rsync to transfer the content from the bad array, to the new one. rsync is nice in that it will hash check the new file against the original to make sure there was no issue with the transfer.

Once I had migrated my data off of the bad array, I booted back into the cards bios and deleted the array. Soon as I did that the alarm finally stopped going off. Whatever it was that had went wrong, had now been fixed...with money lol

My OS drive got corrupted with all the kernel panics as well, and i didn't care to go rescue my data from it. I had an image from mid january 2015 that I just copied back onto the system.

I plan on taking the crappy 3TB drives and individually filling them up with my data; and then I'm going to put them back in the box my new hard drives came in, and store them in my basement. This will give me a dated offline backup incase of any future catastrophe.

I also took the 3TB HGST NAS drive, and I replaced the previous 2TB drive I was using for downloads.

To help make things easier, for those that haven't been keeping up with everything. Here is my current configuration.

Hardware:

SSD 1: 2 x 120GB SAMSUNG 840 EVO (RAID 1 for the OS)

SSD 2: 1 x 128GB OCZ Vertex 4

SSD 3: 1 x 128GB Crucial MX100 (RAID 1 w/ SSD 2 for SSD cache)

HDD 1: 1 x 320GB WD Black 2.5" (for storing VM disks)

HDD 2: 1 x 250GB Hitachi Travelstar 5k500 2.5" (for backing up my /home/ folder and some application config files)

HDD 3: 1 x 3TB HGST Deskstar NAS

HDD 4: 8 x 4TB HGST Deskstar NAS (RAID 6)

(36.066TB ~36TB :D)

And now for the pr0n!!!

The graveyard :P

SERVER---

Server IP: 10.254.6.250

Server Name: SERVER

OS name: Linux

OS Version: 3.13.0-44-generic

OS Architecture: x86_64

Driver Name: megaraid_sas

Driver Version: 06.700.06.00-rc1

Application Version: MegaRAID Storage Manager - 14.08.01.02

HARDWARE---

Controller: LSI MegaRAID SAS 9260-8i(Bus 3,Dev 0)

Status: Optimal

Firmware Package Version:12.15.0-0205

Firmware Version: 2.130.403-3835

BBU: YES

Enclosure(s): 1

Drive(s): 12

Virtual Drive(s): 3

BBU---

BBU Type: IBBU

BBU Status: Optimal

Enclosures---

PRODUCT NAME TYPE STATUS

HPSASEXPCard Ses OK

Drives---

CONNECTOR PRODUCT ID VENDOR ID STATE DISK TYPE CAPACITY POWER STATE

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 CrucialCT128MX1 ATA Online SATA 118.719 GB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 SamsungSSD840 ATA Online SATA 111.281 GB On

Port 0 - 3 SamsungSSD840 ATA Online SATA 111.281 GB On

Port 0 - 3 OCZVERTEX4 ATA Online SATA 118.719 GB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Port 0 - 3 HGSTHDN724040AL ATA Online SATA 3.638 TB On

Virtual Drive(s):---

TARGET ID NAME CAPACITY STATE RAID LEVEL MegaRAID RECOVERY

3 - 21.829 TB Optimal RAID 6 NO

1 - 118.719 GB Optimal RAID 1 NO

2 - 111.281 GB Optimal RAID 1 NO

Sorry to hear about your troubles and the new drives solve your problems. Hopefully we both made a good choice in drives. I have the same pr0n as you and I'm running them in the RAID 6 equivalent under ZFS. :-) I'm waiting for the price to drop back to or below $160 so I can buy four more of them to complete my NAS allocation. When I bought my first allotment, I got into the same issue with the 5-drive limitation. I think when I purchased them, they were $165 each.

nOyE3fN.jpg

Edited by Blade of Grass, added spoiler.


Workstation 1: Intel i7 4790K | Thermalright MUX-120 | Asus Maximus VII Hero | 32GB RAM Crucial Ballistix Elite 1866 9-9-9-27 ( 4 x 8GB) | 2 x EVGA GTX 980 SC | Samsung 850 Pro 512GB | Samsung 840 EVO 500GB | HGST 4TB NAS 7.2KRPM | 2 x HGST 6TB NAS 7.2KRPM | 1 x Samsung 1TB 7.2KRPM | Seasonic 1050W 80+ Gold | Fractal Design Define R4 | Win 8.1 64-bit
NAS 1: Intel Intel Xeon E3-1270V3 | SUPERMICRO MBD-X10SL7-F-O | 32GB RAM DDR3L ECC (8GBx4) | 12 x HGST 4TB Deskstar NAS | SAMSUNG 850 Pro 256GB (boot/OS) | SAMSUNG 850 Pro 128GB (ZIL + L2ARC) | Seasonic 650W 80+ Gold | Rosewill RSV-L4411 | Xubuntu 14.10

Notebook: Lenovo T500 | Intel T9600 | 8GB RAM | Crucial M4 256GB

Link to post
Share on other sites

Sorry to hear about your troubles and the new drives solve your problems.  Hopefully we both made a good choice in drives.  I have the same pr0n as you and I'm running them in the RAID 6 equivalent under ZFS.  :-)  I'm waiting for the price to drop back to or below $160 so I can buy four more of them to complete my NAS allocation.  When I bought my first allotment, I got into the same issue with the 5-drive limitation.  I think when I purchased them, they were $165 each.

 

 

 

 

I went back and looked at your build log. When you do expand, you should look into an SAS expander. I got mine off ebay decently cheap. You can read about it in my UPDATE1, click the link in my SIG. You would use a reverse breakout cable and go from the 8 LSI sata ports on your mobo to the 2 inputs on the SAS expander. Then you would have 6 SAS ports to do what you want with. SAS -> SAS or SAS breakout cable. Now that I think about it, if you check on ebay for supermicro 4U cases, alot of those cases come with built in SAS expanders.

 

How is ZFS on ubuntu? I considered it when I started this whole server thing. But that was 5 years ago, and at that time ZFS was still pretty new. I just didn't like the fact that I was going to be running a filesystem that recommended I always run the latest/greatest version or risk losing my data. I'm more of a fan of the "if it isn't broke, don't fix it" mentality. I went with an XFS filesystem instead. If ZFS ever actually gets to a point of being done I'd like to give it a try.  Have you ever seen it "heal" a file?

Link to post
Share on other sites

I'm definitely familiar with SAS expanders but haven't implemented on in my own build yet.  I'll certainly consider them down the road.  I have four more SATA ports that I can connect drives to which fills out the chassis for this NAS.  I'll need to move into a bigger case if I want to grown beyond that much like the supermicro case or one of those Norco cases.  I'd of preferred the breakout cables for much less clutter in my chassis but I was able to manage it a bit better than those initial pictures show in my build thread.  The issue I might run into with my setup is the built-in LSI adapter already breaks out the 8 ports as SATA ports.  I'm not sure I can connect those to a SAS expander.  My fall-back was to add another LSI adapter and do as you've suggested.

 

So far ZFS has run perfectly for me in this setup.  I built this NAS specifically with ZFS in mind because my new job focuses on it for the product me build.  I wanted a reason to learn more at home on my own time.  I like the options it offers for expansion, snapshots, and data integrity via checksums (bit rot protection) among other things.  I've done periodic zfs scrubs and I've yet to return an error which needed to be fixed ("healed") as you've asked about.  The way ZFS stores the data it writes a block with a SHA-256 checksums to calculate for parity problems and stores a pointer to the checksum value.  Then the pointer is also checksummed for increased data integrity.  There is also self-healing which can occur because I'm using RAID Z in my setup.  No other mainstream filesystem protects against this issue that I'm aware of.  Performance has been very good so far.  In all cases I'm limited by my 1GbE adapter.  I'm also using an SSD for L2ARC for cached reads.

 

I'd say it's worth you giving another look at ZFS even if to experiment.  It offers a nice way to detach you from a physical RAID card's propriety environment and would allow you to export a pool and import it on another system if needed down the road once you migrate to newer hardware.


Workstation 1: Intel i7 4790K | Thermalright MUX-120 | Asus Maximus VII Hero | 32GB RAM Crucial Ballistix Elite 1866 9-9-9-27 ( 4 x 8GB) | 2 x EVGA GTX 980 SC | Samsung 850 Pro 512GB | Samsung 840 EVO 500GB | HGST 4TB NAS 7.2KRPM | 2 x HGST 6TB NAS 7.2KRPM | 1 x Samsung 1TB 7.2KRPM | Seasonic 1050W 80+ Gold | Fractal Design Define R4 | Win 8.1 64-bit
NAS 1: Intel Intel Xeon E3-1270V3 | SUPERMICRO MBD-X10SL7-F-O | 32GB RAM DDR3L ECC (8GBx4) | 12 x HGST 4TB Deskstar NAS | SAMSUNG 850 Pro 256GB (boot/OS) | SAMSUNG 850 Pro 128GB (ZIL + L2ARC) | Seasonic 650W 80+ Gold | Rosewill RSV-L4411 | Xubuntu 14.10

Notebook: Lenovo T500 | Intel T9600 | 8GB RAM | Crucial M4 256GB

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×