Windows Storage Space Query

[Shadowsong]

Hi everyone, i'm looking for a bit of advice on Windows Storage Spaces, if my planned setup is achiveable and some guidance on how to set it up, specifically the powershell I will need.

The storage setup -

• Windows Server 2019
• 6 x 3TB HDDs
• 3 x 500GB SSD's
• 1 x 500GB PCI SSD

The plan -
 

• Single storage space including all of the disks (want to use tiering)
• 1x 400GB SSD-Only Virtual Disk Striped and formatted in ReHS (For VMs)
• 1x 8TB HDD/SSD Virtual Disk Parity and formatted in ReHS (For video editing files, pictures, Plex Video files, ISO's etc)
• 1x #TB HDD/SSD Virtual Disk Parity and formatted in NTFS (For files not supported by ReHS)

So the first question is if you think that seems like it will be possible? If so do you have any recomendations for improvements/optimizations, specifically around columns (not sure what to set!) and if parity would be suitable for HDD/SSD tiering? Can different file systems be used across the same storage pool?

Thanks!

[Electronics Wizardy]

This should work fine

 

Id probably but the vms on the pcie ssd only, then set everything else to collumn size of 3.

 

Id be tempted to go mirror here, just so its easier to deal  with a failed drive.

 

Id go fewer bigger hdds.

[Shadowsong]

9 minutes ago, Electronics Wizardy said:

This should work fine

 

Id probably but the vms on the pcie ssd only, then set everything else to collumn size of 3.

 

Id be tempted to go mirror here, just so its easier to deal  with a failed drive.

 

Id go fewer bigger hdds.

Great that's good to know thanks

 

If the VM's were on the PCIE SSD Only how would that work in terms of resiliency? Also are you able to choose a specific physical disk for data to live on when setting up a virtual disk, i thought it would only have visbility of the storage pool? (Not set my server up yet so not been able to see this running)

 

Have you had any experience with mirror accelerated parity? I read here - https://social.technet.microsoft.com/Forums/windows/en-US/4a366ee1-f84e-4bf0-8dd8-c80c76022b20/mirror-accelerated-parity-support?forum=winserverfiles that it would be possible on a single server via powershell.

 

I take it drive failiure in a parity scenario with storage spaces is a bit of a nightmare?

[Electronics Wizardy]

3 minutes ago, Shadowsong said:

If the VM's were on the PCIE SSD Only how would that work in terms of resiliency?

There would be none, if that drive failes, you lose all the data.

 

3 minutes ago, Shadowsong said:

Also are you able to choose a specific physical disk for data to live on when setting up a virtual disk, i thought it would only have visbility of the storage pool? (Not set my server up yet so not been able to see this running)

When you make a pool, you can pick the specific disks to put the data on.

 

4 minutes ago, Shadowsong said:

Have you had any experience with mirror accelerated parity? I read here - https://social.technet.microsoft.com/Forums/windows/en-US/4a366ee1-f84e-4bf0-8dd8-c80c76022b20/mirror-accelerated-parity-support?forum=winserverfiles that it would be possible on a single server via powershell.

I think you can run it on a single server, but I haven't ran it for a bit. Try it in a vm

 

4 minutes ago, Shadowsong said:

I take it drive failiure in a parity scenario with storage spaces is a bit of a nightmare?

Not really, its pretty easy to replace. Go play with it in a vm.

 

[Shadowsong]

5 minutes ago, Electronics Wizardy said:

There would be none, if that drive failes, you lose all the data.

 

When you make a pool, you can pick the specific disks to put the data on.

 

I think you can run it on a single server, but I haven't ran it for a bit. Try it in a vm

 

Not really, its pretty easy to replace. Go play with it in a vm.

 

Excellent thanks, I think i will do as you say with the VM's on the PCI-E SSD and just ensure backups are done regularly. In this case i assume i just don't include the PCIE SSD in pool so it remains standalone, should i then just keep it this way or is it still worth making a second pool just with that disk? I will go test it out in a VM and get a feel for how things work. 

 

On the performance side of things, if i keep with the parity option for the two larger pools is there any issue with me still using tiering in there? I know parity has terrible performance compared to mirror in storage spaces and am not sure if the SSD tiering will help that much, other than making the SSD's work harder? 

[Electronics Wizardy]

1 hour ago, Shadowsong said:

Excellent thanks, I think i will do as you say with the VM's on the PCI-E SSD and just ensure backups are done regularly. In this case i assume i just don't include the PCIE SSD in pool so it remains standalone, should i then just keep it this way or is it still worth making a second pool just with that disk? I will go test it out in a VM and get a feel for how things work. 

 

On the performance side of things, if i keep with the parity option for the two larger pools is there any issue with me still using tiering in there? I know parity has terrible performance compared to mirror in storage spaces and am not sure if the SSD tiering will help that much, other than making the SSD's work harder? 

tiering works fine in parity. And you can add a write buffer on ssds, so it will still write pretty fast.

 

Id just format the pcie ssd for vms only.

 

Are you buying these parts new or already have them? I would change some things if you were buying them new.

 

[leadeater]

9 hours ago, Shadowsong said:

1x #TB HDD/SSD Virtual Disk Parity and formatted in NTFS (For files not supported by ReHS)

What files are not supported by ReFS? Far as I know there isn't anything, maybe some odd vendor recommendations but other than that Windows and Apps don't care if it's ReFS vs NTFS.

 

9 hours ago, Shadowsong said:

• Single storage space including all of the disks (want to use tiering)
• 1x 400GB SSD-Only Virtual Disk Striped and formatted in ReHS (For VMs)
• 1x 8TB HDD/SSD Virtual Disk Parity and formatted in ReHS (For video editing files, pictures, Plex Video files, ISO's etc)

You're going to need to use PowerShell and define your storage teirs manually for this, fairly simple process and good guides online how to do it. If you get stuck I can give you the exact commands, I'd rather by lazy in the first instance and let you learn how first.

 

9 hours ago, Shadowsong said:

1 x 500GB PCI SSD

I would set this SSD are Journal mode to use as Write-Back Cache, not a lot of use for this with only one of them. Just remember you cannot use the disk that has the OS on it in the Storage Pool so if this disk is that then it's not an option.

 

9 hours ago, Shadowsong said:

Have you had any experience with mirror accelerated parity? I read here

Yes I use it. There's two answers for this 1) It's work great and been very fast 2) Horribly slow and worse than pure HDDs in mirror.

 

I've got 2 servers running identical configuration but with different hardware, one much newer and using good SSDs (Samsung Pros) and an older one using kinda junk SSDs (ADATA and low tier Sandisk). The older worse hardware configuration had horrific performance, worse than a single HDD, exactly why no idea but it was bad.

 

So it's really important you do sustained long term benchmark before going ahead and using it, confirm it's actually working well. I recommend IOmeter and running multiple different I/O sizes and read/write ratios and have test lengths of about 30 minutes.

 

9 hours ago, Shadowsong said:

I take it drive failiure in a parity scenario with storage spaces is a bit of a nightmare?

It's very important to have an extra disk connection free on your system to add the replacement drive for the failed, this will make you're life WAY easier. Storage Spaces wants a replacement drive present before it'll eject a failed drive, you can work around it but plan ahead and you'll have a much better time. Replacement is easy and only a few clicks.

 

9 hours ago, Shadowsong said:

I think i will do as you say with the VM's on the PCI-E SSD and just ensure backups are done regularly. In this case i assume i just don't include the PCIE SSD in pool so it remains standalone, should i then just keep it this way or is it still worth making a second pool just with that disk?

If you don't realistically plan on adding a second PCIe SSD then don't add it in to a pool, otherwise do so because you can't do it if you're using it. Dedicated SSD for the VM storage is a good use for this though, it's that or journal but how effective utilization wise as a journal disk is a bit questionable.

 

9 hours ago, Shadowsong said:

On the performance side of things, if i keep with the parity option for the two larger pools is there any issue with me still using tiering in there? I know parity has terrible performance compared to mirror in storage spaces and am not sure if the SSD tiering will help that much, other than making the SSD's work harder? 

If they are half way decent SSDs you won't wear them out.

 

 

I think this test was 6 SSD (mirror) + 3 HDD (parity), I can get updated test later with for sure known config.

[Shadowsong]

Thanks both, given me a much better idea of what my setup my look like. Most of this is exisiting kit, with a few additions so unfortunately can't really swap much of it. I will defintily test the mirrored parity setup though. Is there a space benefit to mirrored parity over standard mirror or do they both half the usable space? 

 

3 hours ago, leadeater said:

What files are not supported by ReFS? Far as I know there isn't anything, maybe some odd vendor recommendations but other than that Windows and Apps don't care if it's ReFS vs NTFS.

I think i may have been getting confused on this one for some reason thinking it was a file limitation, where as it's more specific things that certain apps may want to use like Plex with it's meta/artwork etc as hard links don't work. For these sort of scenarios, do you think it's still worth having a small NTFS virtual disk from the large pool, or just keep those few exceptions on the OS SSD?

 

Lastly then, for my virtual disk setup. The plan was to either have one large ReFS disk, or that plus a small NTFS disk for support. Is there any benefit or decrement to having multiple virtual disks instead? 

 

For example would there be any performance difference if i had backups going to one virtual disk and video files going to another? Or would it be no different than them just going to the same virtual disk in different folders? 

[leadeater]

29 minutes ago, Shadowsong said:

Is there a space benefit to mirrored parity over standard mirror or do they both half the usable space?

The fast tier is SSD and mirror, usually two-way mirror, so 50% usable capacity. Slower tier is HDD and parity, single or dual, so is much more capacity efficient (what that is is based on number of disks). Capacity is additive of tiers so if you create a virtual disk of 1TB size with 100GB of that being two-way mirror SSD and 900GB being single parity using 6 disks total raw capacity used is 200GB SSD and 1080GB HDD.

 

For the same configuration using two-way mirror total raw would be 2TB, ~800GB more or 60% more efficient.

 

The benefits aren't really any different to RAID, similar space efficiency rules apply.

 

29 minutes ago, Shadowsong said:

I think i may have been getting confused on this one for some reason thinking it was a file limitation, where as it's more specific things that certain apps may want to use like Plex with it's meta/artwork etc as hard links don't work.

Plex works fine. The Plex setup I have uses hard links, I think that isn't actually for Plex though but for sonarr/radarr. I'm not sure I didn't actually set it up just host the VM.

 

29 minutes ago, Shadowsong said:

Lastly then, for my virtual disk setup. The plan was to either have one large ReFS disk, or that plus a small NTFS disk for support. Is there any benefit or decrement to having multiple virtual disks instead? 

 

For example would there be any performance difference if i had backups going to one virtual disk and video files going to another? Or would it be no different than them just going to the same virtual disk in different folders? 

Multiple virtual disks will perform better and also allows better space utilization, particularly when you add new disks to the pool. It's a little complicated so I'd go do some research in to column sizes and the impact that has when adding disks and data placement.

 

Always consider first manageability, don't create virtual disks just because you can, have a reason. Your backups is a good example, for that I would not tier and only use single or dual parity with HDDs.

 

Something to help you on your way to understanding how Storage Spaces works is that disks themselves do not matter as much, they are just placement targets for data chunks. Everything in Storage Spaces works on data chunks, these chunks are placed across disks in the pool based on resiliency settings, column size and other rules such as Storage Tiers.

 

Tradition RAID works at the disks/block layer and doesn't have strong hardware abstraction at the storage subsystem, Storage Spaces/ZFS/BTRFS all work at the data layer and have strong hardware abstraction and therefore don't greatly care what the storage device is or it's size, these are just factors it uses rather than directly relies on.

[Shadowsong]

Thanks again @leadeater very useful. I think ill go ReHS across all my virtual disks then, thinking about it anything that may end up needing NTFS will be inside my VM's anyway so it's not really an issue. 

 

So after all of that and pending some testing, my thoughts are to go with the following and in additon to backups also seperating out my video files into a different virtual disk.

• 500GB PCI SSD in ReHS Standalone (VMs)
• 1TB Virtual Disk in ReHS Single or Parity (undecided) HDD only (Backup)
• 1TB Virtual Disk in ReHS Mirroed-Parity SSD/HDD (Files)
• #TB Virtual Disk in ReHS Mirrored-Parity SSD/HDD (Video)

The above would be across the 6 x 3TB drives and 3 x 500GB SSDs.

 

Does that seem like a sensible approach to you or any further recomendations/optimizations to be had? 

[leadeater]

On 12/5/2019 at 3:44 AM, Shadowsong said:

Does that seem like a sensible approach to you or any further recomendations/optimizations to be had? 

Yep looks good to me

[Shadowsong]

On 12/6/2019 at 10:35 AM, leadeater said:

Yep looks good to me

Do you happen to have those powershell commands you used for setting up your storage spaces or a good article explaining the setup? Especially the mirroed parity side of things. I've got the server up and running now and been playing around with the GUI but it's limited. The powershell i've found online works, but still not given me all the info I wanted. 

 

So far i have a 14.55TB storage pool created, made up of 5x3TB HDD's and 2x500GB SSD's. I have used powershell to set the tiers, but when checking found the "templates" in there as well. They ok to be left or should they be removed? 

 

 

Thanks again for the help!

[leadeater]

5 hours ago, Shadowsong said:

Do you happen to have those powershell commands you used for setting up your storage spaces or a good article explaining the setup? Especially the mirroed parity side of things. I've got the server up and running now and been playing around with the GUI but it's limited. The powershell i've found online works, but still not given me all the info I wanted. 

https://www.starwindsoftware.com/blog/configure-a-resilient-volume-on-windows-server-2016-using-storage-spaces

 

5 hours ago, Shadowsong said:

I have used powershell to set the tiers, but when checking found the "templates" in there as well. They ok to be left or should they be removed? 

Leave them, storage tiers don't actually do anything until you define them when creating a virtual disk. They are just rule sets and have no effect on the pool at all by themselves.

[Shadowsong]

6 hours ago, leadeater said:

https://www.starwindsoftware.com/blog/configure-a-resilient-volume-on-windows-server-2016-using-storage-spaces

 

Leave them, storage tiers don't actually do anything until you define them when creating a virtual disk. They are just rule sets and have no effect on the pool at all by themselves.

Excellent thanks, just what i needed! 

 

I'm now setup - 

 

So i understand this last bit correctly - For the creation of the virtual disks, If i were to setup a tiered disk with say 50GB SSD and 200GB HDD. That would effectively give me the mirror-accelerated parity correct? 

 

Lastly, in the GUI it still gives me options for mirror, parity etc when making a virtual disk. However with powershell (using the command in his guide) it doesn't set resiliency and i assume relys on the underlying resiliency of the tiers you create? 

 

New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName "VirtualDisk" -StorageTiers @($ssd_tier,$hdd_tier) -StorageTierSizes 100GB, 300GB

 

 

[leadeater]

2 minutes ago, Shadowsong said:

Lastly, in the GUI it still gives me options for mirror, parity etc when making a virtual disk. However with powershell (using the command in his guide) it doesn't set resiliency and i assume relys on the underlying resiliency of the tiers you create?

Correct, the tiers set the resiliency. The PowerShell command has a resiliency parameter you can set but that is for the entire virtual disk which is why you set it on the tier when using tiers, otherwise you would specify it.

[Shadowsong]

4 minutes ago, leadeater said:

Correct, the tiers set the resiliency. The PowerShell command has a resiliency parameter you can set but that is for the entire virtual disk which is why you set it on the tier when using tiers, otherwise you would specify it.

Perfect thanks, so when creating my virtual disks if i have one i only want to be on "cold" storage I can just set it only to the HDD tier. 

 

One final question, do you have any recomendations on setting the write cache higher than the default 1GB? Have you adjusted yours?

[leadeater]

46 minutes ago, Shadowsong said:

One final question, do you have any recomendations on setting the write cache higher than the default 1GB? Have you adjusted yours?

Leave it as default, write back cache in Storage Spaces isn't that good so best to just leave it default.

[Shadowsong]

1 hour ago, leadeater said:

Leave it as default, write back cache in Storage Spaces isn't that good so best to just leave it default.

Thanks

 

Here are my results if anyone is interested in the future, mirrored parity seems excellent!

OS: Windows Server 2019

FileSystem: ReHS

HDD_Tier consists of 5x3TB HDD's 

SSD_Tier consists of 2x500GB SSD's

 

Parity Only - Backups Drive

 

 

Mirror-Accelerated Parity - Files Drive

 

 

[Shadowsong]

@leadeater Do your mirror accelerated parity drives generate the same optimization error as i get on mine? Do you know if this is anything to be concerned about?

 

The volume Files (E:) was not optimized because an error was encountered: The operation requested is not supported by the hardware backing the volume. (0x8900002A) 

 

Not found anything concrete online suggesting what this may be, but i'm wondering if it's purely because it's not needed as ReHS handles this on the fly and not by a defrag task?

[leadeater]

What command and parameters are you running?

[leadeater]

Optimize-Volume -TierOptimize

Assuming you're doing this then no you don't need to, that is for if you use NTFS which doesn't support multi-resilient volumes so has to move data between tiers on command/schedule. ReFS can do it on the fly as needed so doesn't need that at all.

[Shadowsong]

3 minutes ago, leadeater said:

What command and parameters are you running?

Sorry should have been clear, these are happening automatically and gernerating event errors. 

 

 

Defrag schedule however is disabled. 

[leadeater]

30 minutes ago, Shadowsong said:

Sorry should have been clear, these are happening automatically and gernerating event errors. 

 

 

Defrag schedule however is disabled. 

Ok, so there is a default scheduled task under one of the folders which gets enabled when you create a tiered virtual disk. Either find and disable it, adjust the command it runs or just ignore it.

[leadeater]

[Shadowsong]

Ah great, that explains it. Cheers!