Jump to content
looney

LTT Storage Rankings

Recommended Posts

I added four 256GB 850PROs to my NAS (link to earlier in this thread).  Now at 33.5TB raw space

 

8x 4TB WD Red in RAIDZ2

2x 250GB Crucial MX200 mirrored

4x 256GB Samsung 850PRO in RAIDZ1

FreeNAS is running from a 16GB PNY stick, so all the SSDs are storage. 

 

Still need to configure the Samsungs, but I'm completely wiping the NAS and doing a fresh install of Freenas 9.10 first (I'm still on 9.3 currently). 

I know that FreeNAS11 will be released real soon, but I'd rather not update every 2 days because of bug fixes.  So I'll wait a little while before switching to that.

 

I had to get pretty creative though.  Because my Define R5 already had all 8 drive bays and the two 2.5" trays filled, I ended up mounting the 850s in the top grille. 

 

5933391de9427_850mounting.jpg.3226c08e02c93e2482989a7fa6c28540.jpg

(After taking the photo I put the modu-vents back on, of course)

 

At this point I'm out of SATA ports, so any more upgrades will involve a new HBA and a larger case or a rack. 

 

Methinks I'll stick with the Tentagon name, even though it should really be Fourteentagon now. 

 

59333c777c068_NASdiskoverview.jpg.c4450ac2a7773daf786a0f6661bad4ca.jpg

 

Link to post
Share on other sites
13 minutes ago, Captain Chaos said:

At this point I'm out of SATA ports, so any more upgrades will involve a new HBA and a larger case or a rack

Your card is a a SAS card which means it can support up to 255 devices connected to it.
You can use an expander such as this if you need more ports: http://www.ebay.com/itm/New-HP-24-Bay-PCI-e-SAS-Expander-Card-468405-001-487738-001-/262633260835

That of course wouldn't resolve your case size issue if you do need to expand further :P


Spoiler

Intel i7 3770K @ 4.6ghz | EVGA Z77 FTW | 2 x EVGA GTX1070 FTW | 32GB (4x8GB) Corsair Vengeance DDR3-1600 | Corsair H105 AIO, NZXT Sentry 3, Corsair SP120's | 2 x 256GB Samsung 850EVO, 4TB WD Black | Corsair 600T (White) | OCZ ZX 1250w | Samsung 28" 4K Display | Ducky Shine 3 Keyboard, Corsair M50 Mouse, MicroLab Solo 7C Speakers, Razer Goliathus Extended, X360 Controller | Windows 10 Pro | SteelSeries Siberia 350 Headphones

 

Spoiler

Corsair 400R, IcyDock MB998SP & MB455SPF, Seasonic X-Series 650w PSU, 2 x Xeon E5540's, 24GB DDR3-ECC, Asus Z8NA-D6C Motherboard, AOC-SAS2LP-MV8, LSI MegaRAID 9271-8i, RES2SV240 SAS Expander, Samsung 840Evo 120GB, 2 x 8TB Seagate Archives, 12 x 3TB WD Red

 

Link to post
Share on other sites
Just now, Jarsky said:

Your card is a a SAS card which means it can support up to 255 devices connected to it

True, with an expander I could indeed get more drives without needing to put an extra HBA in.  I should have thought of that, but I completely forgot about expanders.  

As you said I would still have the case problem to sort out. 

 

Right now I have around 13TB of data and 21TB of usable space (should be 25TB but I seem to have a lot of ZFS overhead).  I don't expect to run out of disc space in the next 5 years. 

The main reason for adding the Samsungs is because my music collection has gotten out of hand ever since I switched that over to FLAC.  It just doesn't fit on the Crucials anymore and I don't want to store it on the HDDs. 

Link to post
Share on other sites
2 minutes ago, Captain Chaos said:

Right now I have around 13TB of data and 21TB of usable space (should be 25TB but I seem to have a lot of ZFS overhead).  I don't expect to run out of disc space in the next 5 years. 

 

If you did multiple vdev's then you would have more zfs overhead as you essentially have multiple raid5's.

I would assume the way you've done it is 2x 4x4TB RAIDZ1 vdev's together in a zpool then with the parity overhead of having multiple vdev's 21TB would sound about right. It would be the recommended way of doing it with that number of disks typically for expansion in the future as well, compared to a raidz2 with all those disks. Rebuild times would be significantly lessened as well.


Spoiler

Intel i7 3770K @ 4.6ghz | EVGA Z77 FTW | 2 x EVGA GTX1070 FTW | 32GB (4x8GB) Corsair Vengeance DDR3-1600 | Corsair H105 AIO, NZXT Sentry 3, Corsair SP120's | 2 x 256GB Samsung 850EVO, 4TB WD Black | Corsair 600T (White) | OCZ ZX 1250w | Samsung 28" 4K Display | Ducky Shine 3 Keyboard, Corsair M50 Mouse, MicroLab Solo 7C Speakers, Razer Goliathus Extended, X360 Controller | Windows 10 Pro | SteelSeries Siberia 350 Headphones

 

Spoiler

Corsair 400R, IcyDock MB998SP & MB455SPF, Seasonic X-Series 650w PSU, 2 x Xeon E5540's, 24GB DDR3-ECC, Asus Z8NA-D6C Motherboard, AOC-SAS2LP-MV8, LSI MegaRAID 9271-8i, RES2SV240 SAS Expander, Samsung 840Evo 120GB, 2 x 8TB Seagate Archives, 12 x 3TB WD Red

 

Link to post
Share on other sites
2 minutes ago, Jarsky said:

I would assume the way you've done it is 2x 4x4TB RAIDZ1 vdev's together in a zpool

Whoops, forgot to write that the HDDs are in RAIDZ2.  I'm getting 19.9TB, whereas a perfect 2 drive parity would give me 24TB. 

Link to post
Share on other sites
3 hours ago, Captain Chaos said:

 

I had to get pretty creative though.  Because my Define R5 already had all 8 drive bays and the two 2.5" trays filled, I ended up mounting the 850s in the top grille.

I like your mounting method. My solution to mounting SSDs without proper bays usually involves adhesive-backed velcro.

Link to post
Share on other sites

@Jarsky sort of convinces me to enter, I got a pretty simple NAS sitting below my desk (and sometimes rest my feet) as well as a really ghetto server that is the backup for said NAS, I will only enter with my main NAS though, since I can't make pictures of my (offsite) backup atm.

 

Hardware:

Spoiler

 

Kabylake G4560

ASRock H110M ITX

2x4GB GSkill value DDR4

240GB Crucial BX200

3x2TB Seagate Ironwolf 5900 rpm

 

Meaning I got 4TB of accessible storage and 240GB of cache.

 

Software:

I am running unraid, with one parity drive, to access my offsite backup I use OpenVPN and this server also hosts my Plex.

 

Storage setup:

My servers both are organized in in 2 public shares each, one of which is read only, and holds a copy of the other server as well as timed versions of important folders of that server, the other is holding my movies, images and important data, which is then (in a weekly fashion) copied onto the other server in a similar way, giving me weekly rollbackpoints for everything but pictures and movies.

Link to post
Share on other sites

FreeNAS Box (Black 4224): 12x4TB (2 6 drive vdevs each in RaidZ2)

Cold Storage Box (Green Google 4u):  Currently 1x8TB, 2x2TB, 1x1TB

ESXi Box (Google Yellow 2u): 2x512GB

 

Definitely wanting to get more drives to fill up the FreeNAS box

Homelab.jpg


3930k | RIVBE | 32GB Dominator Plat. | Titan XP | Intel 750 1.6TB | Mellanox 10Gb NIC | AX1200i 

Custom TJ07 | 2xD5 pumps | EK Supremacy EVO | EK Titan XP | EK RIVBE Block | Cu Tubing | 25x120mm Rad

Samsung U28D590D 4K | Leopold Otaku w/Browns | Corsair M65 | Corsair SP2500 | Sennheiser HD595

Link to post
Share on other sites

I have totally reconfigured my Server Setup, which is as follows now:

 

Running on a Dell T410 Tower Server

Base OS: ESXi

OS drive: Samsung 850 Evo 120 GB SSD

 

ESXi Datastore drives:

Dell PERC H200 RAID Card in IR Mode

2x 146GB 15K SAS - RAID1

2x 300GB 15K SAS - RAID1

 

Main Storage Drives:

LSI 9207-8e RAID Card in IT Mode

4x Toshiba 3TB SATA; 2x Seagate 3TB SATA - RAIDZ1

 

Guest Operating Systems:

1. FreeNAS - using PCI Passthrough to have direct access to the above LSI Controller

- Runs the main pool RAIDZ1

- Shares the pool in variouns SMB Shares across the network

- Runs a Transmission Jail

2. Windows Server 2016 Standard

- Hosts a Plex Server


* Intel i7-4770K * ASRock Z97 Anniversary * 16GB RAM * 750w Seasonic Modular PSU *

* Crucial M4 128GB SSD (Primary) * Hitachi 500GB HDD (Secondary) *

* Gigabyte HD 7950 WF3 * SATA Blu-Ray Writer * Logitech g710+ * Windows 10 Pro x64 *

 

Link to post
Share on other sites

It has been a while since I last updated my post, my 412+ has been on backup duties for a while, not sure if I should update the old post with its current setup?

 

Hardware

Synology DS1815+

PSU:  Power Brick 250w

MOBO: Custom Synology Board

CPU:  Intel Atom C2538 2.4 GHz

RAM: 6GB Kingston DDR3
HDD 1: Seagate Barracuda 4TB (ST4000DM000)
HDD 2: Seagate Barracuda 4TB (ST4000DM000)
HDD 3: Seagate Barracuda XT 4TB (ST4000DX00)
HDD 4: Seagate Barracuda XT 4TB (ST4000DX00)

HDD 5: Toshiba 3TB (DT01ABA300)

HDD 6: WD Blue 4TB (WD40EZRZ)

HDD 7: WD Red 4TB (WD40EFRX)

HDD 8: Drevo 256GB SSD (X1 Pro)

 

Software, Configuration & Usage

 

My thirst for data outgrew my now older 412+ Synology, I love their products so stuck with them and upgraded to a 1815+ around 18 months ago, I currently have a hodge podge of drives, this is due to looking out for drives on eBay as you can pick up some real bargains. I quite often shuck externals, the two 4TB Seagate's are from Expansions the WD red I picked up by shucking a WD Studio external. So all in I have just over 27TB of raw storage in this box.

 

I currently have the 256gb SSD dedicated to download duties and stores all the Docker containers to keep things snappy, the remaining drives are setup in SHR2 aka RAID 6.

 

I store mainly linux ISO's, family photos etc on the box and it happily runs a Minecraft server for my son and his friends, I have everything setup in Docker containers for usenet automation and alike, I write guides over on my website. I never seem to be able to stratch my storage itch so across my other drives in the 412 and a Microserver I have another 15TB!

 

Almost forgot I have a Cyberpower UPS for backup power in case anyone accidently unplugs the box or trips off the mains power as the last thing I wont is drives lost to power issues.

 

 

IMG_20160702_134804.jpg

IMG_20170708_111834.jpg


| Intel Core i5 6600 | ASRock H170M ITX | 16GB DDR4 | GTX970 |
| Crucial M4 256GB | Samsung 850 EVO 250GB  | Antec True Power 650 | Fractal Design Node 304 | Coolermaster 212+ | Samsung 2433BW |

Link to post
Share on other sites

Here's my storage setup for Home:

 

1 Wd Green 3Tb in my main rig which is a backup of the files stored on the NAS.

 

NAS server:

4U Inter Tech 4098

350W Corsair PSU

Asrock Rack e3c224-4l

Pentium G3420 with stock aircooler

2x4Gb DDR3 1600 ECC

Intel 10Gb AF-DA sfp+ NIC

Icy Box SATA Backplane 4x3.5" in 3x 5.25"

3x3Tb WD Red and 1 3Tb Seagate Ironwolf in RAID 10

 

Offsite Backup (at work :P) for the most important files:

1U chassis

220w FSP PSU

Asrock Rack e3c224d2i mITX

Intel Core i3 4370 (Which I plan to install later in the Home NAS)

2x4Gb DDR3 1600 ECC

3x1Tb Hitachi 2.5" drives in RAID 5 (Gonna see if I can update to 4x1Tb RAID 5 this week)

 

In the picture below, the Home NAS is still in the 2U chassis but I swapped not long ago to the 4U chassis (and added the Seagate 3Tb drive) which dramatically improved the cooling of the drives.20170420_193101.thumb.jpg.e9d194db0fc4af33ba161552443551c7.jpg

 

And the setup of my rack as it was in april:

20170407_183614.thumb.jpg.903f176db0e62ba216e8e46ebb6fa4ea.jpg

 

It's a lot cleaner now, added 10Gb ethernet, rails for the 4U chassis (retailer had not the other 2 pair I need for the 2U chassis) and a proper KVM.

 

Some pictures of the renewed setup a bit later;)

Link to post
Share on other sites
On 19/05/2017 at 2:13 PM, GreasyGaming said:

--snip--

UPDATED

 

Added 9 more 8TB drives

Changed CPU

Added 10Gbit NIC

Added another HBA

 

CASE: 45Drives 

MB: Supermicro X9SCM 

CPU: Intel(R) Xeon(R) CPU E3-1275 @ 3.40GHz

HS: Standard

RAM: 32GB ECC Memory (4x 8GB PC3-12800)

HBA: 2 x HighPoint Rocket 750

SSD: Toshiba 120GB

HDD 1: 36x 8TB WD RED 5400 RPM

NIC: Onboard 2x 1Gigabit

NIC: HP Mellanox 10GbE Connext-2

Link to post
Share on other sites

Hi guys,

 

As promised, here are some more recent pictures of my setup;)

 

I finally have the last pair of rails for the 2U chassis.

 

Rack as it is now, the top still requires some work and shelves to serve for miscellaneous storage place

20170725_201902.thumb.jpg.beab464fea1dc181d310a3c796885c11.jpg

 

And here's a little close-up for the hardware bit:

 

20170725_201927.thumb.jpg.21545425f925b6a81eba17b56bb65eeb.jpg

 

From top to bottom:

Netgear access point

TP-Link TL-ER6020 VPN router

Cable manager (That kind of manager is a really a must have for a rack, wonder how I did without it for so long:P)

D-Link DGS 1510-20 L3 switch with 2 10Gb ports

RDP Server (not a Xeon anymore but Ryzen 7 1700X with 32Gb DDR4 2133 ECC)

NAS in it's 4U chassis with it's 4x3Tb backplane (Not a Ryzen CPU, it's the wrong sticker:P)

2 UPS's to protect everything.

I placed pieces of paper over the Power on LED's cause they are quite aggressive:P

Link to post
Share on other sites

Workstation/Gaming computer
Intel i7-2600K running at 4.5GHz
Corsair 16GB DDR3 1600MHz
MSI GTX 1070
Samsung 850 EVO 500GB SSD for Windows and games.
17TB HDD (1x2TB, 2x3TB, 1x4TB, 1x5TB)

 

NAS
HP ProLiant Microserver Gen8
G1610T
4GB
2x8TB (Seagate IronWolf)

FreeNAS

 

Fileserver
Intel Xeon 1230v2
8GB
15TB
Debian


WS: 2600K@4,5ghz - 16GB -  500GB SSD  - 19TB HDD - GTX 1070 8GB - XONAR DGX - 40" 3840x2160  - W10 Pro
LAPTOP: Lenovo ThinkPad X131e - 12GB - 120GB SSD - W10
NAS: HP MicroServer Gen8 - 16TB

Link to post
Share on other sites

I will make it on the odd list when I finally get my stupid hp dl380 g6 to run linux 


Good luck, Have fun, Build PC, and have a last gen console for use once a year. I should answer most of the time between midday and midnight.

NightHawk: i5 6600k @4.2, ASUS Z170m-plus, H105, 16gb corsair vengeance LPX, Strix GTX 970, Corsair RM750X, Samsung 500 gb 850 evo, Samsung 250gb 750 evo and 5tb Toshiba x300

My new NAS (remember to add link) HP DL380G6 with 4x146gb 10k drives and 4x300gb 10K drives, running FreeNAS

PSU White list:

 

 

Link to post
Share on other sites
On 6/1/2017 at 7:25 AM, GreasyGaming said:

Ok, someone else is running enterprise high speed spinners and the drive temps were astronomical!

 

Would be interested to know you get on.

Been running 10k and 15k RPM disks since SCSI Ultra Wide & Ultra 2, back then the full metal sleds HP used really helped. Now they are just plastic and require a bit more cooling, especially now that most are in 2.5" size. I've had disks too hot to even hold before, lack of airflow is a bitch lol.

Link to post
Share on other sites

QNAP TVS-EC880 w/ 8x 3TB @RAID6 (10Gb SFP+)

Supermicro server (esxi) w/ 2x 1TB SSD + 1x 5TB (10Gb SFP+)

 

Network backend is on a Ubiquiti US-16-XG 10Gb switch.  Most clients are served from AC WAPs and a Ubiquiti US-24. 

 

 

51CA566A-B350-489A-9439-2B4ECD6C3915.jpeg


Desktop: ASUS X99 WS | i7-6850k @ 4.5Ghz | 32GB Dominator 3000 | 2x Titan X (pascal) | 5.5TB SSD | LEPA G1600 | Corsair Air 540 | Acer Predator X34 | 2x Dell U2415

Laptop: HP ZBook 17 G4 | i7-7820hq | 16GB | P3000 | 1080P | 1TB 600P | 500GB 850 EVO

Laptop: Dell XPS 13 9343 | i5-5200u | 8GB DDR3 | 256GB SSD | 4k Touchsceen

Laptop: HP Elitebook Folio G1 | M7-6Y75 | 8GB LPDDR3 | 600p 512GB SSD | FHD Touchsceen

Server(ESXi): Supermicro | 2x E5-2640v2 | 128GB DDR3 ECC | 2x 1TB SSD | 2x 5TB 7.2k

HCI Dev Cluster: 3x Supermicro X9SRW-F | 3x E5-2640 | 3x 64GB DDR3 ECC | 3x 120GB SSD Cache + 500GB SSD Capacity

Nas: QNAP TVS-EC880 | E3-1246v3 | 16GB | 8x 3TB RAID6 | 2x 128GB SSD 

Nas: QNAP TS-451+ | J1900 | 8GB | 4x 4TB RAID6 (offsite backup)

Link to post
Share on other sites

Hardware

CASE: Rosewill RSV-L4411

PSU: Rosewill CAPSTONE-450

MB: SUPERMICRO MBD-X9SCL

CPU: Intel Xeon E3-1220v3

HS: Stock Intel heatsink

RAM: Mushkin Enhanced PROLINE 32GB DDR3 1600 ECC

RAID CARD 1: IBM M1015, Flashed with "IT" firmware

SSD: Intel 128gb (not sure of model, had it laying around)

HDD 1: 5x 4TB Seagate ST4000DM000

HDD 2: 4x 3TB WD red WD30EFRX

 

Software and Configuration:

My server is running FreeNAS 11u3.

My main array consists of 5 4TB disks in RaidZ1 using the 128GB SSD for SLOG with NFS and SMB shares.. The second array is RaidZ2 with a SMB share. It is joined to active directory to manage permissions to the smb shares. I'm not using any plugins/jails.

 

Usage:

I have a NFS share on bigboy mounted to my vmware lab for idle VMs that need large storage and a windows share for data. It is effectively silo storage for my hypervisors. I was using iSCSI but wanted to give NFSv4 a shot. I also store movies / torrents / data on bigboy. I host plex on a VM within my vmware lab and mount a SMB share, running it within a jail is a hassle*. I do have a second FreeNAS box with 3 SSDs striped for vms that need more giddyup (backed up routinely by veeam).

 

Backup:

I use veeam for my backups which mounts to a smb share on tinyboy. I also drop my manual backups of network devices to tinyboy. At this time I do not use any cloud storage but hoping to find a really good friend or a nearby co-location site to stick a server. I use replication to backup some of the datasets to the backup volume (tinyboy).

 

Additional info:

*When FreeNAS updates the FreeBSD kernel, the current jails do not get updated. As a result when you go to update your ports/plugins it wants the new kernel. To update the jail's kernel you have to delete it, update the template, and create a new jail. Too much of a headache.

Rack is a Startech 4POSTRACK25U (not sure if I'm allowed to link to where I bought it or not?)

The Rosewill case is great except that it's pointless as a "rackable" case... They made it too wide for rails of any kind. So it's sitting on a shelf -_-.

 

Photo's:

Just the rack in the dark

Spoiler

oPpYush.jpg

Rack with the lights on. Top left: Cable modem (150/10) Top Right: Custom box for pfSense

Stray cable shooting out the left above the switch - runs out to another power outlet where I'm using powerline ethernet. Drooping cable with PoE injector is used for a Ubiquiti AP-Pro mounted higher up on the wall.

Spoiler

wPXKot8.jpg

16 port netgear - has a nice web GUI, bought it for VLANs (ignore the poor unplugged blue cable) 

Dell C1100 (32GB, L5520) is off, was a ESXi node. May switch from the custom pfSense box to the C1100.

Dell R610 (96GB, X5670) running ESXi

IBM x3650m2 (32GB, X5670) running ESXi

FreeNAS-1 (what this post is about)

Spoiler

iKQTwMC.jpg

TP-Link switch is used for IPMI, piss poor cable management by yours truly (always changing things, hard to keep up). Little 8 port gigabit switch ontop of the TP-Link was something I was testing.

Spoiler

7ljoLwL.jpg

Cyber Power 1500 UPS, just bought new batteries for it few months ago.

Spoiler

j0bLzSN.jpg

 

Volume overview

Spoiler

OLfqZOE.jpg

Bigboy volume status

Spoiler

IUngvq3.jpg

 

Link to post
Share on other sites
On 9/29/2017 at 1:14 AM, Dark said:

-snip-

You mounted a rack server to the wall...

 

That's what I call... vertical thinking! <yeeeeaaaaaaahhhhhhhh!!!!!.wav>

 

Seriously though, super neat setup on that wall!

 

EDIT: actually, how did you mount them exactly?

Link to post
Share on other sites
33 minutes ago, MG2R said:

You mounted a rack server to the wall...

 

That's what I call... vertical thinking! <yeeeeaaaaaaahhhhhhhh!!!!!.wav>

 

Seriously though, super neat setup on that wall!

 

EDIT: actually, how did you mount them exactly?

Something like this I'm assuming: https://www.amazon.com/Startech-19-Inch-Vertical-Mountable-Server/dp/B001YHYVEY

Generally these are intended for switches and patch panels, but they can hold servers if you secure them to the wall properly.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×