From old hardware, a new NAS rises

[jimm_eh]

Hi all,

 

I'm a VFX artist, and I'm wrapping up a gig which ate up an unexpectedly huge amount of storage on my current server (a QNAP TSD-453Be with 4x8TB Reds in RAID-10). I still have room, but it got close. It's time for me to split off the work stuff onto its own separate server.

So, the questions I'd like to put to the experts is:

1. What NAS software setup would be best for my given hardware and use case?
2. Or should I spend the $ and build from different hardware entirely?

Use Case:

Dedicated file server ONLY  (No VM's, Plex servers or other server type applications).  
High performance (the closer to saturating 10GbE the better)
High reliability/redundancy
Data will be VFX work.  This would consist of lots of huge files, like 4k sequences (as EXR frames) and Houdini simulation data.

Working Environment: 10GbE network running from a Mikrotik CRS305-1G-4S+IN, with 2 Windows 10 workstations and the QNAP plugged in.  The new server gets the last port.

Current hardware:

Big old Cooler Master case with 9 bays (3.5")
Antec Earthwatts 650 PSU from 2010 -- checked the capacitors for any swelling, none found.
MSI H77A-g43 motherboard with an Ivy Bridge 3570k CPU
16GB RAM (32GB max)  No ECC option
LSI Megaraid 9207-8i 6Gbs SAS card (up to 8 SATA drives) ---------->
Mellanox MCX311A-XCAT CX311A ConnectX-3  SFP+ Ethernet ----> these two occupy all slots of usable bandwidth for this purpose.
256GB Sandisk Extreme system SSD (remember Sandforce?)
All with UPS power backup

Available drives:
2 4TB WD Reds from 2015, with about 4 years always-on operation.

2 4TB WD Reds from 2019, with only a few months total operation.

1 4TB HGST Coolspin with about 3.5 years always-on operation.
If I run with these, I'm thinking of picking up 2 more 5400 4TB's to set up 4x2 Raid 6 with 1 spare.

My main question is which NAS software setup would be best for this? 

FreeNAS would be my first choice, but two things make me hesitate there: no ECC memory, and max 32GB on this hardware. FreeNAS only uses RAM for writeback caching of asynchronous writes, and does not support SSD for that use.  From what I understand, the ZIL/SLOG are a different beast, not a write cache per se.

Alternatives I know of are Unraid, and a brew-my-own mdadm Linux setup. My skills there are what I'd call "medium".

I'm about a week from completing this gig and plan to get started soon on this build.  Your input is appreciated!

 

Edited January 23, 2021 by jimm_eh

[Electronics Wizardy]

3 minutes ago, jimm_eh said:

2 WD Reds from 2015, with about 4 years always-on operation.

2 WD Reds from 2019, with only a few months total operation.

1 HGST Coolspin with about 3.5 years always-on operation.
If I run with these, I'm thinking of picking up 2 more 5400 4TB's to set up 4x2 Raid 6 with 1 spare.

What size are the drives? If there under about 4tb id just get new ones.

 

Id probably try to get a array of identical drives if you can, just makes it nicer to setup and configure.

 

3 minutes ago, jimm_eh said:

FreeNAS would be my first choice, but two things make me hesitate there: no ECC memory, and max 32GB on this hardware. FreeNAS only uses RAM for writeback caching of asynchronous writes, and does not support SSD for that use.  From what I understand, the ZIL/SLOG are a different beast, not a write cache per se.

Freenas doesn't need any more ram than anouther os does. the 1gb/tb rule is bs, and should be pretty good for performance here.

 

ECC isn't need in ZFS more than any other filesystem. ECC is always nice to have, but ZFS doesn't need it more than other systems.

 

4 minutes ago, jimm_eh said:

Alternatives I know of are Unraid, and a brew-my-own mdadm Linux setup. My skills there are what I'd call "medium".

If you want to diy linux, id go zfs for snapshots, and I just like woring with it more. Performance will be fine aswell, still hdd limited.

 

 

[jimm_eh]

17 minutes ago, Electronics Wizardy said:

What size are the drives? If there under about 4tb id just get new ones.

 

 

 

All 4TB, 5400 RPM. Updated to reflect it.

Yeah, I've thought about waiting for the next Best Buy deals on 8-16TB Easystores, but I already have these sitting around and don't expect future gigs to eat up space quite like this one did.

[Maverickfftytwo]

Look into Unraid.

 

Your system is pretty similar to an old system that I have and I am going to be using unraid for photo storage.

[Electronics Wizardy]

34 minutes ago, jimm_eh said:

All 4TB, 5400 RPM. Updated to reflect it.

Yeah, I've thought about waiting for the next Best Buy deals on 8-16TB Easystores, but I already have these sitting around and don't expect future gigs to eat up space quite like this one did.

Freenas/truenas core is likely your best option here, but id replace drive with biggers ones of about the same size if you want the performacne.

 

28 minutes ago, Maverickfftytwo said:

Look into Unraid.

 

Your system is pretty similar to an old system that I have and I am going to be using unraid for photo storage.

good option, but performnce isn't great as its normally limited to the speed of a single drive.

[Switchboy]

Go for openmediavault with snapraid and mergerfs. It will grant the flexibility of unraid while being debian under the hood. OMV is Linux rather than FreeBSD based. This means it is more likely to run reliable on more various hardware.

 

OpenNAS/freeNAS is very expensive and inflexible because it uses ZFS which is a bit overkill in the homeserver envioroment IMO. It does not allow you to add a single drive to a data storage pool. Rather you have to add a software raid array. This means that for every amount of hardrives you add you will have the pay the ‘ZFS tax’ and dedicate at least one drive for redundancy! And even worse: if one of your arrays in the pool suffers catastrophic drive failure from which it cannot recover the ENTIERE DATA POOL is gone! So the entire data pools is as redundent as the weakest array added to the pool!

 

With snapraid and mergerFS you can access every drive independably if you wish so even in the event of catastrophic drive failure not ALL of the data is lost. The system allows you to add a single drive to a pool and it allows for a certain flexibility for the amount of drives you reserve for parity data.

Of course, you can also install docker or any other hypervisor you wish for all your virtualization needs. And unlike unraid it is free as in free beer and as in free speech.

 

A good tutorial to set this up: https://www.networkshinobi.com/snapraid-and-mergerfs-on-openmediavault/

 

I’ve been running it on i5-3570K with 24GB of ram and four 8TB Seagate ironwolfs and a 500gb SSD.
It runs several services:
- NAS
- Resillio Sync
- Plex server with hardware decoder (970 with patch to unlock the transcode limit) https://github.com/keylase/nvidia-patch
- PiHole
- Radarr/Sonarr/Lidarr/Bazarr/Qtorrent (for sailing the high seas)
- Nextcloud
- openVPN

 

[jimm_eh]

On 1/25/2021 at 8:34 AM, Switchboy said:

Go for openmediavault with snapraid and mergerfs. It will grant the flexibility of unraid while being debian under the hood. OMV is Linux rather than FreeBSD based. This means it is more likely to run reliable on more various hardware.

 

 

I hadn't much heard of openmediavault, I'll have to take a look at that.