Storage OS sugestions

[Ralms]

Greetings people,

 

So a little background regarding my setup and plans. I was running a Windows Server 2012 R2 with Storage Spaces for around 2 years now and I hate it with a passion now and refuse to use it again, its just terrible.

Ive been trying and testing Unraid for the last month but I just been having tons of problems and issues with it, like random GUI hangs, random stop responding to network requests, the last update completely ruined all my Docker containers forcing me to reinstall the containers, etc etc etc 

On top of this I dont like the way Unraid is setup because its 1 machine trying to do a lot of stuff. Than I cant have VMs running without having the array started, if I need to stop the array for some reason, Im forced to stop everything. Its stupid, that was my main mistake with Windows Server and Im running from it, trying to have Vms dedicated to 1 task for easier management. 

 

I've been looking into many solutions but nothing really caught my eye in terms of fitting my needs.
Im looking for a Storage Server that is flexible and has parity, so here is what I looked into: 

SnapRaid - Really like the concept but its still an Array so its performance its limited to each disk since when reading a file, that file will only be on a single disk, not taking advantage of all disks.

FreeNas - was really nice, but I wanna run Raid-Z1 (equivelent to Raid 5) but I cant have all disks available when doing the Setup because I dont have anywhere to put my data on. I was going to start with 3 drives but then if I wanted to expand the Raid into my full 5 drives, I cant. On top of that Im planning to expand to Raid 6 in the future. 

XPEnology - I dont want to use it because not even its developers believe its stable, so I wont trust my data to it regardless if its based on Synology OS. 

Storage Spaces - Already used it in the past, in Parity mode, its terrible, very very slow. 

Unraid - Very unstable and unreliable. On top of that is trying to do everything at once. 

 

Any sugestions?

[cbc02009]

Sorry FreeNAS won't work for you, That's definitely what I was going to suggest. I love the web GUI on it.

[Electronics Wizardy]

What hardware are you using?

 

Id give btrfs a shot on linux.

1 hour ago, Ralms said:

Than I cant have VMs running without having the array started

Umm.. You need to have the storage setup for the vm's to run usign that storage.

 

What do you need to do with it?

[Ralms]

1 minute ago, Electronics Wizardy said:

What hardware are you using?

 

Id give btrfs a shot on linux.

Umm.. You need to have the storage setup for the vm's to run usign that storage.

 

What do you need to do with it?

The VMs run on dedicated SSD. 
The storage is mainly for Media, backups, etc. 

[Electronics Wizardy]

1 minute ago, Ralms said:

The VMs run on dedicated SSD. 
The storage is mainly for Media, backups, etc. 

what hardware are you using?

 

How much storage do you need? What speed is this network your using?

[Ralms]

1 minute ago, Electronics Wizardy said:

what hardware are you using?

 

How much storage do you need? What speed is this network your using?

Sorry I missread your first reply.

My current hardware is a P6X58D-E with a Intel i7 970 and 12GB DDR3 Ram. But since it doesnt have VT-D I'm looking into getting a dedicated machine for storage. 
I ran into a mistake of buying a Dell 2950 III with 2 QuadCore Xeons and 32GB of ECC Ram but it consumes way too much power. So Im looking for a single Socket Motherboard to use those CPUs and RAm. 

My Disks are 4 x Seagate NAS ST3000VN000 3TB and 1 x WD Red WD30EFRX 3TB.

 

My network is all Gbit, but I will have a 4 port Intel Pro/1000 PT with Link Aggregation to the switch and other machine (server) will use a dual connection in Link Aggregation too. The clients run in Gbit. 

[Electronics Wizardy]

2 minutes ago, Ralms said:

So Im looking for a single Socket Motherboard to use those CPUs and RAm. 

There aren't any, and there still power hungy cpu's and ram. Id toss them.

 

2 minutes ago, Ralms said:

it doesnt have VT-D I

This won't help storage at all.

 

What vms are you running?

 

Id probably install centos and use btrfs for storage and kvm for vm.s

[Ralms]

1 minute ago, Electronics Wizardy said:

There aren't any, and there still power hungy cpu's and ram. Id toss them.

 

This won't help storage at all.

 

What vms are you running?

 

Id probably install centos and use btrfs for storage and kvm for vm.s

My idea with VT-D was to run the storage server on a VM on top of ESXi with the disks in Passtru to the VM. 

 

What do you mean with btrfs? I know its a file system, but I'm trying to understand how to use it to manage the array, dont know linux that well (I want though)

[Electronics Wizardy]

1 minute ago, Ralms said:

What do you mean with btrfs? I know its a file system, but I'm trying to understand how to use it to manage the array, dont know linux that well (I want though)

Its a file system that supports raid. You can put all the drives in and use it to run raid and make snapshots,

 

Id use kvm for the vm's its normally faster than esxi and its fully free.

[samiscool51]

the best option i can give you to use a software or hardware RAID and go that route

if you can't find one for your needs then you might need a more professional look on your case (someone who works with a storage server for a living e.g. an IT admin)

but wait, unity you work as one!

yes, but i use windows server with storage spaces so i would be useless for your case.

i think some of the mod's on the forums work as IT admins, but i have never asked before, so i don't know.....

[leadeater]

9 hours ago, Ralms said:

Storage Spaces - Already used it in the past, in Parity mode, its terrible, very very slow. 

Parity mode requires SSDs in the pool in journal mode to have good performance. Without SSDs if you need performance mirror configurations should be used, with the down side of usable space.

[leadeater]

@Ralms

Have a look in to Nutanix Community Edition, has basically all the features of the paid version with a cap on the maximum number of hosts allowed in a cluster. We're currently moving all our virtual infrastructure to Nutanix, ESXi as the hypervisor. We have around 1400 VMs, with a large mix of workloads.

[samiscool51]

1 hour ago, leadeater said:

1400 VMs

0_0

but the question is...

CAN THEY RUN CRYSIS?!?!

prob not but if they can, thats a lot of horse power you have there...

use it responsibly and wisely...

(plays minecraft on host machine) (i did that one with one of out servers at work, it's stupid and fun!  )

[Ralms]

5 hours ago, leadeater said:

Parity mode requires SSDs in the pool in journal mode to have good performance. Without SSDs if you need performance mirror configurations should be used, with the down side of usable space.

Yes but that on top of the fact that you cant change anything when a Virtual Disk is created was the nail in the coffin for me. I had 4 Drives in Parity and I wanted to add a SSD for cache but it wouldnt let me unless I recreate the entire thing. It was a pain when Storage spaces only knows to take more data drives to not let you remove them later even if they are empty, etc. I had many headaches with it. 

[Ralms]

5 hours ago, leadeater said:

@Ralms

Have a look in to Nutanix Community Edition, has basically all the features of the paid version with a cap on the maximum number of hosts allowed in a cluster. We're currently moving all our virtual infrastructure to Nutanix, ESXi as the hypervisor. We have around 1400 VMs, with a large mix of workloads.

So I went to Nutanix website and it seems very interesting. 
Im just confused regarding feature set. So I assume you were recomending the Community Edition. 
Im really liking what Im finding but Im worried, wont I need to passtru the HardDrives to Nutanix (or Prism) so it has total control? Im asking that because my CPU doesnt have VT-D. 

 

But thank very much for the sugestion. 

[leadeater]

1 minute ago, Ralms said:

Yes but that on top of the fact that you cant change anything when a Virtual Disk is created was the nail in the coffin for me. I had 4 Drives in Parity and I wanted to add a SSD for cache but it wouldnt let me unless I recreate the entire thing. It was a pain when Storage spaces only knows to take more data drives to not let you remove them later even if they are empty, etc. I had many headaches with it. 

I'm pretty sure you can add an SSD for caching after the fact, add it to the pool and change it's mode to journal. Were you actually trying to add a SSD storage tier to an existing virtual disk? That does require a complete rebuild.

 

I don't disagree with your wanting to move away from it, it's not the most flexible software based storage out there or even cheap for that matter. There are also tons of hidden options/tweaks/optimizations that need to be done as well which are very poorly documented.

 

Be aware there are similar downsides to ZFS. You can't add more disks to an existing vdev, you can add disks to a zpool but only as a new vdev. This makes initial planning quite important since you can't just do adding disks as you please.

[leadeater]

4 minutes ago, Ralms said:

So I went to Nutanix website and it seems very interesting. 
Im just confused regarding feature set. So I assume you were recomending the Community Edition. 
Im really liking what Im finding but Im worried, wont I need to passtru the HardDrives to Nutanix (or Prism) so it has total control? Im asking that because my CPU doesnt have VT-D. 

 

But thank very much for the sugestion. 

Oh damn that is a problem. Yes you need to pass everything through to the Nutanix CVM controller VM. With ESXi you can passthrough individual disks, you need to go in to the advanced options of the host configuration and allow local disks to be used for RDMs.

 

You can do the same with Hyper-V, you likely aren't wanting to use that, and Nutanix's own hypervisor is based on KVM so what ever that can do defines what you'll be able to do disk wise.

 

Edit:

Yes Community Edition as that one is free, the other ones cost money and being an enterprise focused product it isn't cheap.

[Ralms]

3 minutes ago, leadeater said:

I'm pretty sure you can add an SSD for caching after the fact, add it to the pool and change it's mode to journal. Were you actually trying to add a SSD storage tier to an existing virtual disk? That does require a complete rebuild.

Yes I was. 

4 minutes ago, leadeater said:

Be aware there are similar downsides to ZFS. You can't add more disks to an existing vdev, you can add disks to a zpool but only as a new vdev. This makes initial planning quite important since you can't just do adding disks as you please.

That is the main reason I wont be able to use it. 

[leadeater]

Quick question, how much data do you currently have. Wondering due to the problem of needing a place to keep that data during reconfiguration.

[Ralms]

2 minutes ago, leadeater said:

Quick question, how much data do you currently have. Wondering due to the problem of needing a place to keep that data during reconfiguration.

Around 5 TB, so Im using 2 Disks out of the 5.. I can free one of those by copying the data to other random disks I have laying around if needed. 

[leadeater]

5 minutes ago, Ralms said:

Around 5 TB, so Im using 2 Disks out of the 5.. I can free one of those by copying the data to other random disks I have laying around if needed. 

Looks like Nutanix does support adding single disks at a time so could suite you rather well .

https://threewhistlesfulltime.wordpress.com/2017/01/27/nutanix-community-edition-install-single-node-cluster/

 

[Ralms]

2 minutes ago, leadeater said:

Looks like Nutanix does support adding single disks at a time so could suite you rather well .

Can you install Nutanix directly on a machine? Being a host effectivly? 

[leadeater]

4 minutes ago, Ralms said:

Can you install Nutanix directly on a machine? Being a host effectivly? 

Yes by the looks of it Community Edition uses their own KVM hypervisor, Acropolis. I've only used the paid products which are actual server hardware and we install ESXi, this may be a paid only feature.

 

Second video explains the installation rather well, basically create USB installer then boot and install.

[Ralms]

2 minutes ago, leadeater said:

Yes by the looks of it Community Edition uses their own KVM hypervisor, Acropolis. I've only used the paid products which are actual server hardware and we install ESXi, this may be a paid only feature.

 

Second video explains the installation rather well, basically create USB installer then boot and install.

Yeah I was watching the second video.

Man you just showed me what I was looking for, this looks awesome. Really liking it
Just need to double check how Nutanix handles HardDrive failures, etc. 
But regardless, even if I dont use this for Storage in the end, I will most likely use it for the rest. 

[leadeater]

Just now, Ralms said:

Yeah I was watching the second video.

Man you just showed me what I was looking for, this looks awesome. Really liking it
Just need to double check how Nutanix handles HardDrive failures, etc. 
But regardless, even if I dont use this for Storage in the end, I will most likely use it for the rest. 

Try out their online 2 hr demo, might be able to simulate HDD failures etc.