Tiered storage in Win10

[teknohpile]

edit: I want to do something like this (Drobo 5D3) on my existing Win10 machine without spending $700

 

TLDR: I'm trying to make a simple and cost-effective hybrid drive solution (fast and high capacity) for my mid-range gaming computer. I want to use Storage Spaces to make it work but as I understand it Win10 doesn't have support in the GUI to do this.

 

I have 4x 2TB WD blue drives, a 250GB m.2 NVMe drives, 2 SATA SSDs (one is the system disk, the other I bought to use for caching with my onboard RAID controller) and 6 total SATA ports on my motherboard.

After seeing just how bad Intel Rapid Storage Technology can be I've decided to try Storage Spaces in Windows 10. 4 disks set up with parity is already performing better than the raid controller on my motherboard. But I still want to add the SSD as a cache for the RAID. According to what I'm reading this is not supported in the GUI (link below). I may be able to use powershell to do this, but the example uses equal quantities of HDDs and SSDs to make it work. Nothing as simple as "assign 1 SSD to RAID as cache". So, can this be done with 1 SSD and 4 HDDs. Maybe I should just copy the game I want to play onto the NVMe or just buy a new Drobo with thunderbolt and be done with it.

 

Edited August 18, 2019 by teknohpile
added parts used for project

[Electronics Wizardy]

You can have mixed levels in storaga espaces, just tryin making a new virtual disk and see what happens.

[teknohpile]

I have the SSD added to the pool, but the GUI also suggests that it's trying to split the data evenly across the pool. Doesn't seem like the SSD is being used as a cache drive. The performance is better than the Intel RST, but still really bad.

 

[teknohpile]

the answer is probably in here:

https://docs.microsoft.com/en-us/powershell/module/storage/new-storagepool?view=win10-ps

but it's hurting my brain . inexperienced with powershell so it's gonna take me a while 

[Electronics Wizardy]

1 hour ago, teknohpile said:

the answer is probably in here:

https://docs.microsoft.com/en-us/powershell/module/storage/new-storagepool?view=win10-ps

but it's hurting my brain . inexperienced with powershell so it's gonna take me a while 

what command did you use to make the virtual disk?

 

Don't use the gui, it sucks

[Mr.Humble]

 

[teknohpile]

3 hours ago, Mr.Humble said:

 

Thank you. It looks like this example is just 1 HDD and 1 SSD, which should be fairly straight forward following that guide. I need to build a stripe with parity using 4 HDDs and accelerate with only 1 SSD. the guide makes it sound like you need equal quantities of HDD and SSD to use tiered storage in windows 10

[teknohpile]

13 hours ago, Electronics Wizardy said:

what command did you use to make the virtual disk?

 

Don't use the gui, it sucks

I threw it together with the gui. Still reading up on powershell. 

[Electronics Wizardy]

6 minutes ago, teknohpile said:

I threw it together with the gui. Still reading up on powershell. 

You can't do it with the gui in windows 10, you need power shell. Try something like this 

 

Make the tiers with this 

Get-StoragePool Pool1 | New-StorageTier –FriendlyName SSDTier –MediaType SSD

Get-StoragePool Pool1 | New-StorageTier –FriendlyName HDDTier –MediaType HDD

 

Mkae the storage tier variables

 

 

$SSD = Get-StorageTier -FriendlyName SSDTier

$HDD = Get-StorageTier -FriendlyName HDDTier

 

THen make the virtual disk

 

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Simple –StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB -WriteCacheSize 1GB

[teknohpile]

7 minutes ago, Electronics Wizardy said:

You can't do it with the gui in windows 10, you need power shell. Try something like this 

 

Make the tiers with this 

Get-StoragePool Pool1 | New-StorageTier –FriendlyName SSDTier –MediaType SSD

Get-StoragePool Pool1 | New-StorageTier –FriendlyName HDDTier –MediaType HDD

 

Mkae the storage tier variables

 

 

$SSD = Get-StorageTier -FriendlyName SSDTier

$HDD = Get-StorageTier -FriendlyName HDDTier

 

THen make the virtual disk

 

Get-StoragePool Pool1 | New-VirtualDisk -FriendlyName Space1 -ResiliencySettingName Simple –StorageTiers $SSD, $HDD -StorageTierSizes 8GB, 32GB -WriteCacheSize 1GB

Thanks that's really helpful. I'm not great with linux cli, but I know bash WAY better than windows powershell. I'm starting from scratch here 

[Mr.Humble]

47 minutes ago, teknohpile said:

Thank you. It looks like this example is just 1 HDD and 1 SSD, which should be fairly straight forward following that guide. I need to build a stripe with parity using 4 HDDs and accelerate with only 1 SSD. the guide makes it sound like you need equal quantities of HDD and SSD to use tiered storage in windows 10

Look at this post at the end of that thread: 

You clearly can do it with just 1 SSD

[teknohpile]

2 minutes ago, Mr.Humble said:

Look at this post at the end of that thread: 

You clearly can do it with just 1 SSD

Sweet! exactly what I'm trying to do. very much appreciated.

[teknohpile]

Was hoping to use the m.2 drive, oh well. Must be a server-only feature. Unless there's something I need to fix in diskpart 

 

Edit: Success! fix here:  https://www.petenetlive.com/KB/Article/0001560

Edited August 16, 2019 by teknohpile

[teknohpile]

and.... now I'm stuck again 

 

it doesn't seem to like the

• @($ssd_tier,$hdd_tier)

[teknohpile]

I'm starting to doubt this will work. these should be perfectly valid commands, the errors just don't make sense. I guess I'll start from scratch tomorrow.

 

 

[BloodKnight7]

My best advice is to do it through PowerShell (ISE) with admin priviledges, you can autocomplete an see the suggestions for the commands there. If you have spare time let me know through a PM and we can do a remote session, Ill give you a hand.

[teknohpile]

15 minutes ago, teknohpile said:

alright, so simple resilience was fairly easy. No errors creating the tiered virtual disk. now I just have to figure out what I was doing wrong with parity. do the SSD and HDD tiers both need to be set to parity? maybe I set the columns wrong. :shrug:

 



 

With some help from @BloodKnight7 I got a simple volume working. Still need to get parity to work. probably something wrong with my logic on how many columns I need for parity to work right

[BloodKnight7]

Hey man, my apologies, I've been busier than expected all this year... Congrats on the advances, I'll figure out the command for parity when I get home. Ssd and HDD always go in the same pool for a hybrid build, so I would discard 100% having to separate them.

[teknohpile]

this whole time I just had to decrease the volume size further than I thought... still more capacity than a mirror though. ultimately just a slight modification to anoteco's process/

223GB NVMe

4x WD-Blue 1.82TB (2TB) drives. 

parity required that I keep adjusting the size of the HDD tier until the command was happy.

https://www.petenetlive.com/KB/Article/0001562

 

 

 

<create pool in GUI and rename to mypool>

Get-StoragePool mypool | Set-ResiliencySetting -Name Simple -NumberOfColumnsDefault 1
Get-StoragePool mypool | Set-ResiliencySetting -Name Parity -NumberOfColumnsDefault 3

#commands that were commented out are only neccessary if drives have "unspecified" media type

get-storagepool mypool | get-physicaldisk | FT FriendlyName, Size, MediaType

#get-storagepool mypool | get-physicaldisk | ? MediaType -eq "Unspecified" | Set-PhysicalDisk -MediaType HDD
#get-storagepool mypool | get-physicaldisk

New-StorageTier -StoragePoolFriendlyName mypool -FriendlyName SSD_Tier -MediaType SSD -ResiliencySettingName Simple
New-StorageTier -StoragePoolFriendlyName mypool -FriendlyName HDD_Tier -MediaType HDD -ResiliencySettingName Parity
$ssd_tier = Get-StorageTier -FriendlyName SSD_Tier
$hdd_tier = Get-StorageTier -FriendlyName HDD_Tier

 

#command should work with or without "@()"

New-VirtualDisk -StoragePoolFriendlyName mypool -FriendlyName "VirtualDisk" -StorageTiers @($ssd_tier,$hdd_tier) -StorageTierSizes 200GB, 4800GB

<go back to GUI for easier disk formatting>

 

Get-StorageTier

[BloodKnight7]

Glad you had success. Mirror is 50% capacity loss, Parity space loss is similar to a raid 5 configuration

[teknohpile]

yup. I'm happy with it. now I'm curious if I can change the caching method .
maybe some light reading I can do another day.

https://www.wepc.com/tips/ssd-cache/

 

Write-around SSD caching would probably be better for the life of the SSD. With the performance I'm seeing I would guess it's using Write-back SSD caching

[Electronics Wizardy]

11 hours ago, teknohpile said:

now I'm curious if I can change the caching method .

storage spaces is tiering, not caching, so the normal caching types dont apply

 

11 hours ago, teknohpile said:

Write-around SSD caching would probably be better for the life of the SSD.

dont worry about ssd life here, it wont be an issu

 

 

[vanst]

Hello teknohpile,

I would like to follow up - how your Storage Pool performed during the time? Is it still working great now?

I have set up a mirror pool from 2 4TB HDD. I filled the pool with 1 TB of data - most of them are raw images and ISO files. The write speed is good, but the read speed from the pool is not stable. At first, it is just around 1 to 10 MB/s. After some hours, the speed increased to from 20 ~ 90 MB/s.

Is this because I don't hava SSD cache? Or something wrong with the congfiguration? Could you advise!

 

[teknohpile]

On 3/22/2020 at 11:49 AM, vanst said:

Hello teknohpile,

I would like to follow up - how your Storage Pool performed during the time? Is it still working great now?

I have set up a mirror pool from 2 4TB HDD. I filled the pool with 1 TB of data - most of them are raw images and ISO files. The write speed is good, but the read speed from the pool is not stable. At first, it is just around 1 to 10 MB/s. After some hours, the speed increased to from 20 ~ 90 MB/s.

Is this because I don't hava SSD cache? Or something wrong with the congfiguration? Could you advise!

 

sorry, had a lengthy response typed up a few days ago and the forums went offline before I could hit send.

Short version is it worked well as a storage drive, but not great for working directly from that drive. Specifically random reads were not great.
  I'm switching right now to a much larger NVMe drive and I'll work directly from that, maybe with nightly backups to the storage. Mirrored NVMe might be a good option if you are doing critical work. Otherwise a real RAID controller or NAS running unraid or ZFS might have better performance.

[vanst]

Thanks for sharingg. Well, I have get rid of the Storage Space and use an NVME to put all my project files there. I used the hard drive just to back up my footage & final product one it was completed.

A question - how do you do the nightly back up. Is it manually or there is a software?