Storage Solution Help

[dark_xzyph3r]

Hey guys, I am wanting to upgrade my Storage in my server (Wanting to build a new one all together) and I was thinking of getting 2 raid controllers and having 4 or more 120GB SSD's in raid 0 on one and have about 10 2TB HDD's on the other in raid 0 but have the SSD's cache the mechanical HDD's. I will be running 10Gb from it to my main rig. Yes I know Raid 0 is dumb but I have offsite backups and this is really just for experimentation.

[mineblaster]

what dose you storage server currently have cpu(s) motherboard etc

[dark_xzyph3r]

1 minute ago, mineblaster said:

what dose you storage server currently have cpu(s) motherboard etc

 

In my sig, I am going to build a new one soon with 2 Intel Xeon X5550 on a SuperMicro X8DTH-iF

[Beeeyeee]

You want to avoid that many drives at once if you can. 

if not, your going to need one heck of a RAID card.. and you better not be kidding about those daily offsite backups. because as soon as theres a sudden power loss, your ENTIRE computer and all its RAID arrays dies. you lose everything, its unrecoverable.

 

honestly for the SSD's you should just get the Samsung 950 pro. way faster than all those i'd bet.

[dark_xzyph3r]

1 minute ago, Beeeyeee said:

You want to avoid that many drives at once if you can. 

if not, your going to need one heck of a RAID card.. and you better not be kidding about those daily offsite backups. because as soon as theres a sudden power loss, your ENTIRE computer and all its RAID arrays dies. you lose everything, its unrecoverable.

 

Yeah I know the risk  (Insert movie reference that I forgot) 

[Beeeyeee]

Just now, dark_xzyph3r said:

 

Yeah I know the risk  (Insert movie reference that I forgot) 

good, just making sure you know that:)

[dark_xzyph3r]

1 minute ago, Beeeyeee said:

honestly for the SSD's you should just get the Samsung 950 pro. way faster than all those i'd bet.

I already have the SSD's though

[mineblaster]

would go the likes of software raid like limetech UnRaid or a hardware raid using most likely an lsi controller

 

[dark_xzyph3r]

1 minute ago, mineblaster said:

would go the likes of software raid like limetech UnRaid or a hardware raid using most likely an lsi controller

 

 

Yeah I was thinking of using unraid and just using HBA controllers so unraid could get all control. Could ESXI do the same? I have a copy on a flash drive and I dont have to pay for the software...

[mineblaster]

i haven't used either yet but plan to use unraid soon ive almost enough cash save 12 3tb and ill add some sort of ssd drive at a later date if your going software i would say an sas hba controller as it provides better control of your drives and they are pretty cheap on ebay

 

so if you aquire your hardware and sas hba controller first and test ESXI first since you already have a copy

and if you dont like ESXI im pretty sure theres trials of unraid 

 

and if you decide to go hardware raid after the board can do 8 drives allready and maybe an ibm m1015 raid for ssd and the last drives (im pretty sure the lsi manger will put them together)

[scottyseng]

4 hours ago, dark_xzyph3r said:

Hey guys, I am wanting to upgrade my Storage in my server (Wanting to build a new one all together) and I was thinking of getting 2 raid controllers and having 4 or more 120GB SSD's in raid 0 on one and have about 10 2TB HDD's on the other in raid 0 but have the SSD's cache the mechanical HDD's. I will be running 10Gb from it to my main rig. Yes I know Raid 0 is dumb but I have offsite backups and this is really just for experimentation.

I think if you're going to put that much down on SSDs, you'd be better suited to a Intel 750 or a Samsung 950 NVMe drive.

 

The cost of the RAID controllers, SSDs, and batteries for the RAID card would probably add up to the cost of a Intel 750.

 

Ah, never mind, I just read that you already own the SSDs. Hmm, I would say to use software RAID for them...something that supports TRIM.

 

I wouldn't recommend RAID controllers if you plan on putting the SSDs in RAID on them. They don't usually have TRIM support so the performance suffers over a long time. The hard drives would be fine on hardware RAID cards though. If you plan on getting hardware RAID, you also need the RAID card batteries to go along with them. If the power goes out on the hardware RAID cards with the on board batteries, you have no risk of losing the array / corrupt data.

 

Unraid with HBA cards is a good alternative as well.

 

If you go software RAID, then make sure you get a UPS for your NAS.

[leadeater]

4 hours ago, dark_xzyph3r said:

 

Yeah I was thinking of using unraid and just using HBA controllers so unraid could get all control. Could ESXI do the same? I have a copy on a flash drive and I dont have to pay for the software...

ESXi doesn't have any inbuilt storage solutions to create storage pools from disks that is free. VMware has a technology called VSAN but requires 3 physical servers minimum to use it. I get round this issue by creating a storage VM and passthrough HBA's with SSD's and HDD's and use Windows Storage Spaces. This is NOT the best thing to do but due to my very specific requirements works for me. Also I run 10Gb from my server to desktop and host my Steam library on it. I get 2GB/s on my 4x 850 Pro pool and 1GB/s read and 500MB/s write on my 2x 840 Pro 5x 3TB Auto Tier pool, raw performance (1GB/s network limited).

 

You can do what I do directly with Hyper-V rather than the using the VM but I need ESXi/vCenter for the testing I do.

 

If you want to run both VMs and storage off the same box then the options I'm aware of are: Hyper-V Server, unRAID, Nutanix Community Edition. Or anything similar to these I have not listed, can't know of everything or remember them all.

 

Either way don't go down the hardware RAID route unless there is a requirement to do so, costs much more and no real benefit to doing so.

[dark_xzyph3r]

17 hours ago, leadeater said:

ESXi doesn't have any inbuilt storage solutions to create storage pools from disks that is free. VMware has a technology called VSAN but requires 3 physical servers minimum to use it. I get round this issue by creating a storage VM and passthrough HBA's with SSD's and HDD's and use Windows Storage Spaces. This is NOT the best thing to do but due to my very specific requirements works for me. Also I run 10Gb from my server to desktop and host my Steam library on it. I get 2GB/s on my 4x 850 Pro pool and 1GB/s read and 500MB/s write on my 2x 840 Pro 5x 3TB Auto Tier pool, raw performance (1GB/s network limited).

 

You can do what I do directly with Hyper-V rather than the using the VM but I need ESXi/vCenter for the testing I do.

 

If you want to run both VMs and storage off the same box then the options I'm aware of are: Hyper-V Server, unRAID, Nutanix Community Edition. Or anything similar to these I have not listed, can't know of everything or remember them all.

 

Either way don't go down the hardware RAID route unless there is a requirement to do so, costs much more and no real benefit to doing so.

 

2 GB/s is impossible on 10 Gb... It only scales up to 1.2 GB/s..

 

 

17 hours ago, scottyseng said:

I think if you're going to put that much down on SSDs, you'd be better suited to a Intel 750 or a Samsung 950 NVMe drive.

 

The cost of the RAID controllers, SSDs, and batteries for the RAID card would probably add up to the cost of a Intel 750.

 

Ah, never mind, I just read that you already own the SSDs. Hmm, I would say to use software RAID for them...something that supports TRIM.

 

I wouldn't recommend RAID controllers if you plan on putting the SSDs in RAID on them. They don't usually have TRIM support so the performance suffers over a long time. The hard drives would be fine on hardware RAID cards though. If you plan on getting hardware RAID, you also need the RAID card batteries to go along with them. If the power goes out on the hardware RAID cards with the on board batteries, you have no risk of losing the array / corrupt data.

 

Unraid with HBA cards is a good alternative as well.

 

If you go software RAID, then make sure you get a UPS for your NAS.

I think that I might just grab an hba card so I will have more sata ports and just use either free nas in a VM...

 

I always have my server connected to a ups

[Qwerty2323]

Aint 120 ssd priced bad at least at my country 250 price is dam close to 120s

[dark_xzyph3r]

17 hours ago, scottyseng said:

I think if you're going to put that much down on SSDs, you'd be better suited to a Intel 750 or a Samsung 950 NVMe drive.

 

The cost of the RAID controllers, SSDs, and batteries for the RAID card would probably add up to the cost of a Intel 750.

 

Ah, never mind, I just read that you already own the SSDs. Hmm, I would say to use software RAID for them...something that supports TRIM.

 

I wouldn't recommend RAID controllers if you plan on putting the SSDs in RAID on them. They don't usually have TRIM support so the performance suffers over a long time. The hard drives would be fine on hardware RAID cards though. If you plan on getting hardware RAID, you also need the RAID card batteries to go along with them. If the power goes out on the hardware RAID cards with the on board batteries, you have no risk of losing the array / corrupt data.

 

Unraid with HBA cards is a good alternative as well.

 

If you go software RAID, then make sure you get a UPS for your NAS.

I think that I might just grab an hba card so I will have more sata ports and just use either free nas in a VM...

 

I always have my server connected to a ups

 

13 hours ago, dark_xzyph3r said:

I think that I might just grab an hba card so I will have more sata ports and just use either free nas in a VM...

 

I always have my server connected to a ups

Nvm about this post lol my phone said that I never sent it before

 

13 hours ago, Qwerty2323 said:

Aint 120 ssd priced bad at least at my country 250 price is dam close to 120s

They were $25 each

[scottyseng]

3 hours ago, dark_xzyph3r said:

I think that I might just grab an hba card so I will have more sata ports and just use either free nas in a VM...

 

I always have my server connected to a ups

I heard that some VM software don't allow the OS to touch hardware (FreeNAS needs to touch the drives) directly, so you might have issues with FreeNAS. I'm not too well versed into the world of virtual machining though, so take what I say with a grain of salt.

[leadeater]

7 hours ago, dark_xzyph3r said:

 

2 GB/s is impossible on 10 Gb... It only scales up to 1.2 GB/s..

 

Yes which is why I said it was 1GB/s network limited. Guess you missed that in my post.

 

Quote

(1GB/s network limited).

 

[dark_xzyph3r]

7 hours ago, leadeater said:

Yes which is why I said it was 1GB/s network limited. Guess you missed that in my post.

 

 

Ahh yeah, sorry was like 3 am lol

 

9 hours ago, scottyseng said:

I heard that some VM software don't allow the OS to touch hardware (FreeNAS needs to touch the drives) directly, so you might have issues with FreeNAS. I'm not too well versed into the world of virtual machining though, so take what I say with a grain of salt.

I thought that hba cards just pass drives through to the host?

[leadeater]

3 minutes ago, dark_xzyph3r said:

I thought that hba cards just pass drives through to the host?

It works with ESXi if you passthrough the HBA to the VM, you need VT-d support for this to work. Generally speaking this is not officially supported by ZFS/Storage Spaces etc but does work, that how I do my storage VM currently. I've got tow HBA passed through to it.

 

If you are going to install FreeNAS directly on the hardware then you have nothing to worry about.

[dark_xzyph3r]

Just now, leadeater said:

It works with ESXi if you passthrough the HBA to the VM, you need VT-d support for this to work. Generally speaking this is not officially supported by ZFS/Storage Spaces etc but does work, that how I do my storage VM currently. I've got tow HBA passed through to it.

 

If you are going to install FreeNAS directly on the hardware then you have nothing to worry about.

Yeah I would just install free nas but using a dual quad core Xeon with hyperthreading system is soon overkill for that lol so I was thinking of either unraid or esxi and just pass all the drives to a VM for storage ie free nas then have all my VMS on a 2tb drive that backups to the network share coming from the free nas vm

[scottyseng]

6 minutes ago, dark_xzyph3r said:

Yeah I would just install free nas but using a dual quad core Xeon with hyperthreading system is soon overkill for that lol so I was thinking of either unraid or esxi and just pass all the drives to a VM for storage ie free nas then have all my VMS on a 2tb drive that backups to the network share coming from the free nas vm

Yeah, if you can figure out how to pass the HBAs to the VM, you'd be good. I'm not too knowledgeable about VMs though.

 

Yeah, that would be overkill. I have a E5-2695V3 (14 core, 28 thread) and I'm just using it to run Windows Server 2012 R2 as a basic NAS...the CPU laughs at 1-5% load for most of its life so far.

[LinusTech]

29 minutes ago, dark_xzyph3r said:

Yeah I would just install free nas but using a dual quad core Xeon with hyperthreading system is soon overkill for that lol so I was thinking of either unraid or esxi and just pass all the drives to a VM for storage ie free nas then have all my VMS on a 2tb drive that backups to the network share coming from the free nas vm

I haven't looked back since we switched to unRAID, but I wouldn't recommend running FreeNAS in a VM on it (or in general unless you really know what you're doing) 

 

From my experience storage controller passthrough can be kinda flaky. 

 

Honestly I'm really happy with the performance of the unRAID storage array in its own anyway. Using SSDs to hide the slow array write speeds makes it as fast as I'll need for quite some time. 

[leadeater]

26 minutes ago, scottyseng said:

Yeah, if you can figure out how to pass the HBAs to the VM, you'd be good. I'm not too knowledgeable about VMs though.

 

Yeah, that would be overkill. I have a E5-2695V3 (14 core, 28 thread) and I'm just using it to run Windows Server 2012 R2 as a basic NAS...the CPU laughs at 1-5% load for most of its life so far.

Damn, can I have that CPU

[scottyseng]

3 minutes ago, leadeater said:

Damn, can I have that CPU

It was a engineering sample from ebay. I got it for $600. It works perfectly fine though....just completely laughs at everything I attempt to put it under load (aside rendering / stress tests). Also fun to remote connect to the NAS at college and have the less tech savvy people think my laptop has a 14 core CPU (Yes, people do believe this too...).

[dark_xzyph3r]

33 minutes ago, scottyseng said:

Yeah, if you can figure out how to pass the HBAs to the VM, you'd be good. I'm not too knowledgeable about VMs though.

 

Yeah, that would be overkill. I have a E5-2695V3 (14 core, 28 thread) and I'm just using it to run Windows Server 2012 R2 as a basic NAS...the CPU laughs at 1-5% load for most of its life so far.

I can take that off your hands