LTT Storage Rankings

weeandykidd · August 10, 2015

All my drives are old

My oldest is just shy of 25k hours, you have me beat by quite a bit!

alpenwasser · August 10, 2015

-RAIDZ2-

RAIDZ2 is a funny thing. You're getting almost 22TB out of a 10-drive 30TB pool, I'm barely getting 21TB out of an 8-drive 32TB one.

That's 1TB less usable space with 2TB more advertised space. Go figure.

I was recently digging around a bit and found this: http://blog.delphix.com/matt/2014/06/06/zfs-stripe-width/

The author is one of the ZFS developers. It seems RAIDZ space usage is indeed almost magic

and how much space you lose can vary quite a bit.

-update-

Updated!

-snip-

Oi, pretty cool setup, me likey!

-update-

Updated.

-snip-

I like it, welcome to the list!

Just throwing mine out there even though it sadly doesn't qualify due to being my main / gaming PC.

16TB Raw Storage excluding the SSD's.

Tststs, why did you have to be honest and tell me that it's not also used for storage?

Nice box though.

I've also updated my own systems. Had two of my RE4s start producing errors, so bought

three new Reds, made a new ZFS pool with 10 WD Red striped across two 5-drive vdevs in

RAIDZ2. I've also added two WD Greens to the machine as solo-drive backups. I am however

also running those as single-drive ZFS pools for now, see how things go with that. Reason

being, I can do the magical

zfs send crypt_tank_0/dataset@snapshot | pv | zfs receive green_0/dataset

Which is significantly faster than rsync, especially for lots of small files. It's pretty

awesome so far.

Also, my ZFS pools now run on top of encrypted LUKS devices. The cryptfs containers are

unlocked via keyfile after logging in, then the ZFS pools gets imported from the unlocked

device mapper drives. The keyfile itself (yes, I'm using the same keyfile for all drives

belonging to a single pool) is stored on an encrypted USB drive which I unplug after unlocking

the cryptdevs for the ZFS pools.

Performance is much better than before (although the previous pool was getting very full,

and filled ZFS pools can degrade significantly in performance). I can get 400~500 MB/s

sequential read and write, and random writes (without compression, with lz4 compression

it can go higher, depending on how compressible the data is). Random reads are slower due

to the mechanical disks (I presume the random writes are only so fast due to some ZFS

caching magic, but I have not yet been able to find the limits of that. But yeah, I

can now easily saturate my gigabit network most of the time.

Pool scrubbing is ~440 MB/s on average.

After reading @weeandykidd post, I had a look at my own drives power on hours, and most of my drive has about 18-22k hours on and then there is this one o_O

37824 hours

1576 days

225 weeks

4.3 years.

All my drives are old

My oldest is just shy of 25k hours, you have me beat by quite a bit!

This is the oldest drive I still use. From 2008 if I remember right:

Still works like a champ, knock on wood.

What has me rather concerned at the moment though is that I have three WD Red drives

whose SMART data looks like this (this is the worst one):

The drive is about one year old. Note that they're rated for 300k load/unload cycles,

so this drive is already 10% beyond its rated cycle count. Not quite happy with this.

At first I thought it might be a firmware bug, but it isn't (at least one of the drives

which do not exhibit this behavior has same firmware as the one in above pic).

As a comparison, this is one of my remaining RE4s:

So yeah... Not happy at all. I'm already preparing myself for pre-emptively buying

more drives to replace these, then run these into the ground until they fail so

that I can RMA them.

On a more positive note, this was the continuous uptime on my server before I shut

it down for maintenance. Ran permanently without reboot from last September until

end of July.

EDIT:

@Captain_WD: You wouldn't happen to have any idea about what's up (or what could be up)

with that load_cycle count by any chance? The Reds which are exhibiting

this behavior have had the exact same workload as the ones which are not

(they were all part of a six-drive RAIDZ2 pool for the past year or so),

so I'm really rather stumped TBH.

Edited August 10, 2015 by alpenwasser

Eric The Tech Guru · August 10, 2015

Oi, pretty cool setup, me likey!

Thanks, It took a bit of convincing that it was worth it.

For a bit of more detail:

One of the 4TB drives is being used as torrent space, so active torrents go there.

I copy the completed torrent files over to one of the other 4TB drives which is mirrored to another 4TB drive, that's where my organized plex media files are stored.

The last 4TB is just for local backups. I keep a few days/weeks of backups locally, and they push to cloud backup from the server so you can backup locally, don't have to leave ur laptop/desktop running for like 12 hours, then the server does all the uploading after it gets a local copy.

TJStamp · August 10, 2015

Given my job involves playing with mahoosive storage arrays I really should put some effort in and get my personal rig higher up the list

I've got 4x Seagate 8TB Archive drives on the way for ZFS cold storage, will hopefully be able to provide some feedback on them next week

Str_Mape · August 11, 2015

Tststs, why did you have to be honest and tell me that it's not also used for storage?

Nice box though.

Thanks, Can it count if it's used as a media server for the other 4 people in the house and for streaming to my HTPC?

Will have another 8TB drive by the end of the year.

TJStamp · August 11, 2015

My plans have been slightly scuppered... UK stock of the seagate archive 8TB drives is seriously limited at the moment so only 1 of my 4 have arrived with no ETA for the other 3 I'll run attoBench on the single drive later on and post it here anyway.

TJStamp · August 12, 2015

here's how a single seagate 8TB archive drive performed in a couple of very basic tests... it's pretty clear this drive will be terrible for IO heavy usage and will not work as a boot drive or virtualization storage target but for NAS usage (WORM workloads) it's almost perfect

Riley Smuff · August 16, 2015

This is all the sort of stuff I'm excited for for when I'm able to have an income. Right now I'm still in school and I'm stuck drooling over majority of what goes on here.

Very cool!

Would love to be able to have 42tb of hdd space(i only have 5.5tb)

Doowstados · August 21, 2015

Hardware
CASE: Antec 300
PSU: EVGA 430W (80 Plus Cert)
MB: ASUS M5A78L/USB3
CPU: AMD FX-6300 (Six Core)
HS: Stock AMD Fan/Heatsink
RAM: 4GB DDR3 1600
RAID CARD: None
SSD: Kingston NOW V300 120GB
HDD 1: 1 x 3TB WD Green
HDD 2: 1 x 2TB Hitachi 7K3000 Deskstar
HDD 3: 3 x 2TB Hitachi 7K3000 Ultrastar

Software and Configuration:
Server runs Windows Server Essentials 2012 R2. The hard drives are set up using Windows Server's storage spaces. I have two spaces, one non-mirrored and one mirrored, configured in an 11TB advertised (actual capacity 10TB) pool.

Usage:
It is used for home backup of my desktop/laptop, my fiancee's laptop and my room mates desktop/laptop. I also use it to host a personal cloud and various other misc. tasks.

Backup:
Currently it is backed up to a 5TB WD MyBook weekly and the necessary files for bare metal and client recovery are backed up to a 1TB WD Passport three times a day. I will eventually need to get a second 5TB external hard drive and stripe them as the server continues to fill up (not everything has been moved to it yet).

Additional info:
Got the AMD six core processor and motherboard together for under $80 during a microcenter sale. I will soon upgrade to 8GB of RAM and probably will get a more efficient/reliable power supply in the coming weeks. I also still need a UPS.

Photo's:

Jarsky · August 21, 2015

All my drives are old

Feel better now?

Helly · August 26, 2015

I finally got my drives today... SOOOOOO happy, finally getting more storage, was running out and even had to move stuff to other PC's to keep it all.

Im still in the process of backing up and preparing to replace the drives so no specs yet. Turned out the 6TB drives i wanted (nobody remembers i mentioned this right?) kept being to expensive, so i went with 5TB drives. Will post a real update soon . Attached a pic of the drives so i at least post something useful.

arsonik007 · August 27, 2015

Hardware
CASE: Norco RPC-4220
PSU: Rosewill RBR1000-M 1000W
MB: SUPERMICRO MBD-X9SCL+-F
CPU: Intel Celeron G540
HS: Supermicro SNK-P0046P
RAM: 4x Kingston ValueRAM 4GB

RAID CARD 1: M1015 flashed to 9211-8i in IT mode
RAID CARD 2: M5015

SAS Expander: HP SAS Expander
Boot: WD Scorpio Black 750GB
HDD 1: 8x 3TB WD Red (WD30EFRX)
HDD 2: 7x 2TB Seagate Barracuda

HDD 3: 1x 1TB Seagate Barracuda

HDD 4: 4x 300GB Seagate Cheetah 15K.7 (ST3300657SS)

HDD 5: 2x 240GB Kingston SSDNow V300 (SV300S37A)

Software and Configuration:
This server is running W2012 R2. It have the following arrays:
Raid5 8*3TB Red for media and low speed iSCSI target. Will be migrated to RAID 6 once I receive the M5015 Advanced Features Key.
Raid5 7*2TB important files backup and other stuff. It will be removed when I increase the storage on the 8*3TB array. (Plugged on the M1015 in Storage Spaces)
Raid5 4*300GB for high speed iSCSI target with 6GB RAM caching (PrimoCache)
Raid0 2*240GB SSD for high speed iSCSI test target. (Plugged on the M1015)
The last 1TB used for VM backups (Plugged on the M1015)

Usage:
I use the storage for movies and series, photos, personal files, emulators and roms.

For the Cheetah and SSD arrays, it's used for iSCSI target for VM. I have around 5 VM per target currently.

Backup:
The important stuff is backed to the 1TB drive then backed to tape one a week and an archive tape is send to my stepfather house one a month. The tape is located in my ESXi server and managed by Veeam.

Additional info:

It use around 400watts when idle. The 1000W PSU may seems overkill but I had many problem when using 600W one.

SERVER---

Server IP: -

Server Name: -

OS name: Windows Server 2012

OS Version: 6.2

OS Architecture: x86_64

Driver Name: LSI_SAS2

Driver Version: 2.00.76.00

Application Version: MegaRAID Storage Manager - 15.03.01.00

HARDWARE---

Controller: Controller1: SAS9211-8i(Bus 1,Dev 0)

Status: Optimal

Firmware Package Version:00.00.00.00

Firmware Version: 20.00.02.00

BBU: NO

Enclosure(s): 0

Drive(s): 8

Drives---

PRODUCT ID VENDOR ID STATE DISK TYPE CAPACITY POWER STATE

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

WDCWD20EARX00P ATA Unconfigured Good SATA 1.819 TB On

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

ST31000528AS ATA Unconfigured Good SATA 931.513 GB On

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

ST2000DM0011CH1 ATA Unconfigured Good SATA 1.819 TB On

SERVER---

Server IP: -

Server Name: -

OS name: Windows Server 2012

OS Version: 6.2

OS Architecture: x86_64

Driver Name: megasas.sys

Driver Version: 6.600.21.08

Application Version: MegaRAID Storage Manager - 15.03.01.00

HARDWARE---

Controller: Controller0: ServeRAID M5015 SAS/SATA Controller(Bus 2,Dev 0,Domain 0)

Status: Optimal

Firmware Package Version:12.15.0-0205

Firmware Version: 2.130.403-3835

BBU: NO

Enclosure(s): 1

Drive(s): 12

Virtual Drive(s): 2

Enclosures---

PRODUCT NAME TYPE STATUS

HPSASEXPCard Ses OK

Drives---

CONNECTOR PRODUCT ID VENDOR ID STATE DISK TYPE CAPACITY POWER STATE

Port 4 - 7 ST3300657SS SEAGATE Online SAS 278.465 GB On

Port 4 - 7 ST3300657SS IBM-ESXS Online SAS 278.465 GB On

Port 4 - 7 ST3300657SS SEAGATE Online SAS 278.465 GB On

Port 4 - 7 ST3300657SS IBM-ESXS Online SAS 278.465 GB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Port 4 - 7 WDCWD30EFRX68E ATA Online SATA 2.728 TB On

Virtual Drive(s):---

TARGET ID NAME CAPACITY STATE RAID LEVEL MegaRAID RECOVERY

0 Data 19.099 TB Optimal RAID 5 NO

1 HighPerf 835.395 GB Optimal RAID 5 NO

Photos:

The last one is while transferring old drive to the new array.

Sousuke · August 27, 2015

I finally got my drives today... SOOOOOO happy, finally getting more storage, was running out and even had to move stuff to other PC's to keep it all.

Im still in the process of backing up and preparing to replace the drives so no specs yet. Turned out the 6TB drives i wanted (nobody remembers i mentioned this right?) kept being to expensive, so i went with 5TB drives. Will post a real update soon . Attached a pic of the drives so i at least post something useful.

I am getting storage envy :lol: Only got 4 3TB WD Reds for my RAID Storage.

Doowstados · August 30, 2015

The ranking script did not pick up my build :mellow:

Sousuke · August 30, 2015

The ranking script did not pick up my build :mellow:

Its not a script, one of the Moderators/Administrators updates it manually every once in a while. Don't worry yours will be added soon

jscho · August 30, 2015

The ranking script did not pick up my build :mellow:

The script just updates and generates the stats when a new build is put into the list :lol:

Someone will add you eventually.

Helly · August 30, 2015

Ok so here is my update

Hardware

CASE: Antec P183 Performance One
PSU: Cooler Master Realpower Pro 520W Modular
MB: Asus Sabertooth 990FX
CPU: AMD Phenom x6 1090T
HS: Scythe Ninja 2 plus Rev. b
RAM: 8GB OCZ DDR3 PC16000 CL10 Gold
RAID CARD 1: Areca 1880i
RAID CARD 2: Areca ARC-1200

HDD 1: 2x 150GB Raptor
HDD 2: 8x 5TB WD Red
HDD 3: 1x 2TB Toshiba

Software and Configuration:
Still running Windows 7, haven't had the energy or the will to update yet. I also don't really know what to go for, server 2012 or some form of freenas. Is it easy to install stuff on freenas? like FTP server and webserver? Cuz i kinda need those.

My main array is the 8x WD's running in RAID-6 (on the 1880i). giving me a total of just over 27TB of usable space.

My 2nd array are the raptors on the ARC-1200 running in raid-0 as boot drives. The toshiba has become obsolete and i really don't have an idea of what to use it for. Might just remove it some day.

Usage:

Storing my media files (movies and tv shows). currently just over 9TB.

Backup:
All of the media files i have atm are backed up on my old 2TB drives. Those will be stored offline, or is that just called cold storage/cold backup? No idea really. I will keep filling them until they are full. At which point i will make a selection of the most important stuff to be backed up and only that will remain on the drives. Doing this because i want to make sure the 5TB drives are all good. It also makes it easier to move to freenas with ZFS, i can just reformat the drives and change the filesystem to ZFS without having to make a very lengthy backup again.

Pic of the server in the attachment, probably a very crappy picture... it's from my phone . Ill make a better one if it really is that crappy.

EDIT: tried to mark where the drives are... but that didn't turn out all that well.... might redo that later

looney · August 30, 2015

Its not a script, one of the Moderators/Administrators updates it manually every once in a while. Don't worry yours will be added soon

The first few posts are generated with a script but the script does still require manual input.

MG2R · August 31, 2015

The ranking script did not pick up my build :mellow:

As said before: you just need to poke @alpenwasser a bit. He sure does like those notifications! Don't you, @alpenwasser?

alpenwasser · August 31, 2015

Sorry for the delay folks, offline life is keeping me a bit busy these days.

- snip -

Excellent, welcome to the forums, and the list, nice to have you here!

-snip-

Damn, those are quite a few disks you have there!

Ok so here is my update

- snip-

Yay, moar storage!!!

As said before: you just need to poke @alpenwasser a bit. He sure does like those notifications! Don't you, @alpenwasser?

Oh, you know me so well! :wub:

I should have something for you this week btw., the reason I keep not being online so much.

MG2R · August 31, 2015

I should have something for you this week btw., the reason I keep not being online so much.

The hype train is leaving! All aboard!

Brandonberg · September 1, 2015

ZedNAS

Hardware

CASE: Fractal Design Node 304 (Black)

PSU: Corsair HX750

MB: ASRock Rack E3C224D2I

CPU: Intel Xeon E3-1231V3

COOLER: Noctua NH-L9i

RAM: 16GB Kingston KVR16E11/8I 1600Mhz DDR3 Unbuffered ECC (2x8GB)

BOOT: Fully sick 8GB Sandisk thumb drive

HDD: 6 x 4TB HGST Deskstar NAS Drive (Total Drive Capacity: 24TB)

Software & Configuration

FreeNAS 9.3 running ZFS. Raid-Z2 resulting in total pool size of 14TiB or ~15.4TB. Using Plex media server plugin and still experimenting with other plugins at the moment.

Usage

Home storage server for personal files, movies, other media, backup for parents business PC, photos and girlfriends iPhone photo backup (which almost needs its own raid.. seriously she is obsessed..)

Backup

I have an external 2TB Drive and two 1TB Drives which important stuff is backed up to. Also have a 1TB cloud storage account for important files that need to be accessed on the go.

Additional Info

The NAS uses about 85-90W when idle, 100-110W when under full network transfer to RAID and 120-130W when transcoding media via Plex to 3 different devices at once. The Xeon is probably overkill but opted for a Xeon over my previously planned Avoton C2750D4i due to similar pricing and sourcing of components in Australia. I also opted for the 1231 Xeon as it offers Hyperthreading for only a few dollars more (Again in AUS). The HX750 PSU is way more power than i need but i had this lying around so decided to put it to use. I may re purpose this PSU for another build in the future in which case i would probably run the NAS on a Seasonic or Corsair 400-500W PSU.

Photos (More photos below in spoiler tag)

Duff Beer added for extra Awesome (and size reference)

79wjd · September 1, 2015

Software & Configuration

FreeNAS 9.3 running ZFS. Raid-Z2 resulting in total pool size of 14TiB or ~15.4TB. Using Plex media server plugin and still experimenting with other plugins at the moment.

Windows7.png

FreeNAS.png

2015-09-01 10.54.03.jpg

2015-09-01 09.07.07.jpg

2015-09-01 09.06.54.jpg

2015-09-01 09.07.29.jpg

2015-09-01 09.07.35.jpg

It's pictures like that that kind of make me wish I didn't cheap out with a Source 210....although it is sitting out of site anyway....so meh

P.s. have you had any weird issues with the plex plugin? For whatever reason mine shuts off whenever anything changes/when the system is restarted and I'm forced to reinstall the plugin to turn it back on.

Brandonberg · September 1, 2015

It's pictures like that that kind of make me wish I didn't cheap out with a Source 210....although it is sitting out of site anyway....so meh

P.s. have you had any weird issues with the plex plugin? For whatever reason mine shuts off whenever anything changes/when the system is restarted and I'm forced to reinstall the plugin to turn it back on.

I have played around with FreeNAS an awful lot on some crappy hardware to try and get it running sweet prior to forking out the cash for the NAS hardware. From my experience, you need to install Plex last and not change any permissions or other settings after Plex is up and running otherwise it will break. The exact order i configured my setup was

FreeNAS Install => Configure FreeNAS (Interfaces etc..) => Storage Volume Setup => Configure User Groups => Create DataSet and configure permissions => Setup Shares (I only used CIFS) => Then only once everything else works... Install Plex Plugin.

If i install Plex then play around with permissions, plex always has issues. At this stage **fingers crossed** i havnt had any Plex issues after server restarts.

79wjd · September 1, 2015

I have played around with FreeNAS an awful lot on some crappy hardware to try and get it running sweet prior to forking out the cash for the NAS hardware. From my experience, you need to install Plex last and not change any permissions or other settings after Plex is up and running otherwise it will break. The exact order i configured my setup was

FreeNAS Install => Configure FreeNAS (Interfaces etc..) => Storage Volume Setup => Configure User Groups => Create DataSet and configure permissions => Setup Shares (I only used CIFS) => Then only once everything else works... Install Plex Plugin.

If i install Plex then play around with Permissions, plex always has issues. At this stage **fingers crossed** i havnt had any Plex issues with server restarts.

But even after I uninstall and re-install plex I have issues.

Not that it really matters too much....the only reason I'd use Plex is to stream over a VPN since I'm having serious issues trying to stream normally over a VPN and for some reason I keep running into firewall issues when trying to access plex over the VPN even though there are no firewalls enabled.

Sign In

LTT Storage Rankings

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites

Link to post

Share on other sites

Link to comment

Share on other sites