Performance storage for a VM server

[m9x3mos]

Hello all,

I am currently running a Dell T7500 as a VM server which houses about 12 VMs on it. Currently it is running Windows Server 2016 off a 120 GB ssd and then uses Storage Spaces with 1 120 GB ssd for cache tier and 2 750gb hdds for storage tier. I have noticed that is is really getting slow and when I reboot one of the VMs it takes about 8 minutes to come back up. 

I am trying to see if there is anyway I can speed that up. I have been playing with UnRaid recently and it seems like it will primarily run off the storage tier and not keep anything in cache. Is there any solution that would use SSD caching for frequently used bits of storage and move everything else to the slower tier? I think that is what Storage Spaces with Windows server is suppsed to do but it hasn't seem to keep up performance as of lately. 

Does anyone else run a home lab have any advice that isn't just buying more SSDs to run them off as dedicated?

[leadeater]

2 hours ago, m9x3mos said:

Currently it is running Windows Server 2016 off a 120 GB ssd and then uses Storage Spaces with 1 120 GB ssd for cache tier and 2 750gb hdds for storage tier. I have noticed that is is really getting slow and when I reboot one of the VMs it takes about 8 minutes to come back up. 

Is the SSD being used as a Journal (write-back cache) or actually Tiering? My guess would be that you are using it as a Journal because you would need 2 SSDs in the pool if the HDDs are in a two-way mirror, Storage Spaces won't allow a Tier setup where 1 tier is non resilient (simple) and the other one is (mirror or parity).

 

My best advice would be to get another 120GB SSD so you can have a Teired setup of Two-Way Mirror SSD and Two-Way Mirror HDD, additionally you can use powershell to set the pool paramater IsPowerProtected to true which will give a performance speed up but just know that does introduce potential data loss in a power cut.

 

Example Configuration below

New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName SSD_Tier -MediaType SSD -ResiliencySettingName Mirror
New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName HDD_Tier -MediaType HDD -ResiliencySettingName Mirror

$ssd_tier = Get-StorageTier -FriendlyName SSD_Tier
$hdd_tier = Get-StorageTier -FriendlyName HDD_Tier

New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName "VM" -WriteCacheSize 3GB -StorageTiers @($ssd_tier,$hdd_tier) -StorageTierSizes 100GB, 500GB

Set-StoragePool -FriendlyName Pool1 -IsPowerProtected $True

 

Another option is to create a new virtual disk only using the SSD tier and place VHDX files in that locaotion that you always want to be fast, configure as above but take out the HDD tier parts and it'll give you a virtual disk pinned to SSD.

 

Also make sure your virtual disks are formatted with ReFS as that is required for Multi-Resilient Virtual Disks, auto tiering of data to storage tiers.

https://charbelnemnom.com/2017/06/how-to-create-a-multi-resilient-volume-with-refs-on-standalone-server-in-windows-server-2016-ws2016-hyperv-storagespaces/

[m9x3mos]

I actually don't know if it is Journal or Tier. I set it in the UI for Windows Server during configuration. Is there anyway to tell?

I have 3 1tb drives I was wanting to swap in since the 750s are a bit old and one is showing smart errors. Can I do a mirrored ssd setup with a parity HDD tier?

 

40 minutes ago, leadeater said:

Is the SSD being used as a Journal (write-back cache) or actually Tiering? My guess would be that you are using it as a Journal because you would need 2 SSDs in the pool if the HDDs are in a two-way mirror, Storage Spaces won't allow a Tier setup where 1 tier is non resilient (simple) and the other one is (mirror or parity).

 

My best advice would be to get another 120GB SSD so you can have a Teired setup of Two-Way Mirror SSD and Two-Way Mirror HDD, additionally you can use powershell to set the pool paramater IsPowerProtected to true which will give a performance speed up but just know that does introduce potential data loss in a power cut.

 

Example Configuration below

New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName SSD_Tier -MediaType SSD -ResiliencySettingName Mirror
New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName HDD_Tier -MediaType HDD -ResiliencySettingName Mirror

$ssd_tier = Get-StorageTier -FriendlyName SSD_Tier
$hdd_tier = Get-StorageTier -FriendlyName HDD_Tier

New-VirtualDisk -StoragePoolFriendlyName Pool1 -FriendlyName "VM" -WriteCacheSize 3GB -StorageTiers @($ssd_tier,$hdd_tier) -StorageTierSizes 100GB, 500GB

Set-StoragePool -FriendlyName Pool1 -IsPowerProtected $True

 

Another option is to create a new virtual disk only using the SSD tier and place VHDX files in that locaotion that you always want to be fast, configure as above but take out the HDD tier parts and it'll give you a virtual disk pinned to SSD.

 

Also make sure your virtual disks are formatted with ReFS as that is required for Multi-Resilient Virtual Disks, auto tiering of data to storage tiers.

https://charbelnemnom.com/2017/06/how-to-create-a-multi-resilient-volume-with-refs-on-standalone-server-in-windows-server-2016-ws2016-hyperv-storagespaces/

 

[leadeater]

17 minutes ago, m9x3mos said:

Can I do a mirrored ssd setup with a parity HDD tier?

You can but Parity in Storage Spaces is rather slow so no good for VM hosting and more applicable to file shares and archiving, even tiered with SSD.

 

17 minutes ago, m9x3mos said:

I actually don't know if it is Journal or Tier. I set it in the UI for Windows Server during configuration. Is there anyway to tell?

There is, I'll go with simple way first and if that doesn't shed light we can jump over to a few PowerShell commands that will tell us.

 

You should see in Server manager under Storage Pools all the physical disks in the pool, under the usage column it'll say Automatic or Journal.

 

For the Virtual Disk there is a column called Tiered and it'll have a tick, like above, if tiering is configured.

[m9x3mos]

Mine says Automatic there currently

1 hour ago, leadeater said:

You can but Parity in Storage Spaces is rather slow so no good for VM hosting and more applicable to file shares and archiving, even tiered with SSD.

 

There is, I'll go with simple way first and if that doesn't shed light we can jump over to a few PowerShell commands that will tell us.

 

You should see in Server manager under Storage Pools all the physical disks in the pool, under the usage column it'll say Automatic or Journal.

 

For the Virtual Disk there is a column called Tiered and it'll have a tick, like above, if tiering is configured.

 

[leadeater]

10 minutes ago, m9x3mos said:

Mine says Automatic there currently

Under volumes is the Virtual Disk, V: drive, formatted with ReFS or NTFS?

[m9x3mos]

I don't see that under volumes but when I check the disk in File Explorer, it does show ReFS.

5 minutes ago, leadeater said:

Under volumes is the Virtual Disk, V: drive, formatted with ReFS or NTFS?

 

[leadeater]

42 minutes ago, m9x3mos said:

I don't see that under volumes but when I check the disk in File Explorer, it does show ReFS.

Ok, the best I can suggest is running Set-StoragePool -FriendlyName VirtualPool -IsPowerProtected $True and adding more HDDs to the pool to increase performance of the slower tier which is most likely causing the slowness, Two-Way Mirror is not limited to 2 disks and can use as many as you like. That PowerShell command should add a fair bit extra performance though.

[m9x3mos]

Thanks. I will give that a try and see how it works.

If i add a second SSD (have another 120gb one at home) and have 3 HDDs, would the above still work?

2 hours ago, leadeater said:

Ok, the best I can suggest is running Set-StoragePool -FriendlyName VirtualPool -IsPowerProtected $True and adding more HDDs to the pool to increase performance of the slower tier which is most likely causing the slowness, Two-Way Mirror is not limited to 2 disks and can use as many as you like. That PowerShell command should add a fair bit extra performance though.

 

[leadeater]

9 minutes ago, m9x3mos said:

Thanks. I will give that a try and see how it works.

If i add a second SSD (have another 120gb one at home) and have 3 HDDs, would the above still work?

 

Yea it will work, you will have to run an Optimize-Volume PowerShell job to redistribute the data across the new disks though.

[leadeater]

Also could you run in PowerShell the below and post the output.

 

Get-VirtualDisk -FriendlyName VirtualDisk | Get-StorageTier | ft FriendlyName, MediaType, Size

 

Should look like this:

FriendlyName                 MediaType          Size
------------                 ---------          ----
HMN-WS01_Games_FCSATA_01_HDD HDD       1100048498688
HMN-WS01_Games_FCSATA_01_SSD SSD        215016800256

[Mikensan]

Worth noting that you only have 7gb free, I know many file systems start performing very poorly once they near getting full. I haven't kept up with the times, but I can't imagine windows running very well without space for virtual ram / page file / swap. I know for vmware you can specify other disks for swap, unsure how it is handled in hyper-v.

 

The SSD also might not be good enough for things like async writes.

[m9x3mos]

On 1/2/2019 at 11:48 AM, Mikensan said:

Worth noting that you only have 7gb free, I know many file systems start performing very poorly once they near getting full. I haven't kept up with the times, but I can't imagine windows running very well without space for virtual ram / page file / swap. I know for vmware you can specify other disks for swap, unsure how it is handled in hyper-v.

 

The SSD also might not be good enough for things like async writes.

That is only the unallocated space from the partition after the disk was created. It looked like that as soon as I setup the drive. I don't think it is even half full.

[Mikensan]

On 1/4/2019 at 10:15 PM, m9x3mos said:

That is only the unallocated space from the partition after the disk was created. It looked like that as soon as I setup the drive. I don't think it is even half full.

But what allocated it? I've never used storage spaces so I'm very unfamiliar with it - was looking at @leadeater's screenshot as comparason and saw his wasn't full allocated so I thought yours had something going on? I see it's been a while - have you had any progress?

[m9x3mos]

On 1/16/2019 at 9:22 AM, Mikensan said:

But what allocated it? I've never used storage spaces so I'm very unfamiliar with it - was looking at @leadeater's screenshot as comparason and saw his wasn't full allocated so I thought yours had something going on? I see it's been a while - have you had any progress?

Still looking into things. Haven't had a lot of time lately. Changing that setting did seem to make it a bit faster.

On another machine at home I am testing three solutions with 3 x 3tb drives and a SSD. Doing tests with unraid, storage spaces tiered storage (parity), and Intel raid 5 setup with acceleration.