Storage Space issues with Server 2012 R2

[Th3Architect]

Hi All - 

 

So I'm having some issues with one of my Windows Server 2012 R2 machines - I decide to try out the Storage Spaces in the O/S and found it easy to setup and fairly easy to figure out the difference between pools and actual storage etc. It has been running on my server for nearly a year and is used to back up A LOT of data while I work on customers machines, ie reformats.

 

My issue is that I have 4 x 4TB drives in a two-way mirror so 16TB total 8TB actual, my storage pool is saying I need to add more drives as my pool is too full, I have cleared out SOOOO much data but the pool doesn't think I have, is there anyway to 'jump start' storage spaces into realising I have released all that space? I have attached screenshots so you can see.

 

Oh and I've just noticed the virtual drive showing up in Disk Management, I've seen this before but not on my machine and it was NOT there before the reboot I did about an hour ago so that doesn't have anything to do with my current issue but if anyone knows why it's doing it I'd love to know.

 

Thank you for any wisdom you might be able to impart, I really don't want to have to take this server offline at all

[BiggHertz]

Hi There,

 

What I'm fairly certain is happening, is that you're looking at the pool (aggregate)level. Which is an overview of pool resources being utilized. I'm assuming that when the original virtual disk was created from the pool, thin provisioning was selected, this only allocates resources from the pool as the demand on the associated volume as needed. However, when the contents of that volume are deleted, the volume does not release resources back to the pool. So if the volume grew to 10 tb for example, the virtual disk also grew to 10 tb, this would allocate 10tb from the associated pool. if you delete the contents of the volume, this does not shrink the virtual disk, and the virtual disk remains at 10 tb and still uses 10 tb from the pool.

 

Execute the following command in PowerShell and I'm fairly confidant that you will see the Provisioning type as "Thin"

 

get-virtualdisk | select * | ft FriendlyName,OperationalStatus,ProvisioningType,ResiliencySettingName,NumberOfColumns,allocatedsize,footprintonpool

 

You can remedy this by creating a new thinly provisioned virtual disk, transfer contents and detach/delete original virtual disk.

 

My suggestion is to leave it alone as the volume will eventually grow again.

 

What is more of a concern is that you mentioned that you have 4 x4tb drives and you stated it was in a 2 way mirror, yet the volume labeled as storage space shows 16 TB, If the information above is correct, you've purposely over-subscribed to your storage, which isn't the end of the world, just be mindful of this. Also when the time comes for adding addtional drives, one must add the number drives that correspond to the number of columns in the pool (also listed in the powershell script above).

 

Hope that helps!

 

-Jason

[Th3Architect]

Hi Jason,

 

Thanks for the reply, you were indeed right the provisioning type is showing as Thin. first question is 'Should this have been a different type of provisioning?' and secondly I went on the basis that my pool should be 16TB as 4x4TB = 16TB am I right in thinking I should have made my pool 8TB as I knew I was going to be mirroring them? What I was trying to do was not fool myself by making it larger than it actually was, to put it simply I just wanted it showing exactly what storage I had available to me rather than a pool twice the size of my actual storage, have done the complete opposite here?

 

And if you don't mind how would I go about creating a new thinly provisioned virtual disk, all my sata ports are full I think except maybe one, I will be backing up the data to larger external drives over the next day or two ready to carry out whatever it is I need to do just in case. I'm thinking of starting again with the storage pool if turns out to be awkward and if I do as per my info above how would you suggest setting it up.

 

Thank you for your advice BTW it's very much appreciated.

[BiggHertz]

Thanks for the reply, you were indeed right the provisioning type is showing as Thin. first question is 'Should this have been a different type of provisioning?'

 

     - Thick provisioning is great in scenarios where you want to: 

            A. Increase on demand provisioning performance (not necessarily as much of an improvement as it was in the past, with Server 2016,REFS and VHDX's all working on closing                        that performance gap).

            B. Increase Performance by controlling fragmentation of the pool.

    

     -Thin provisioning is great in scenarios where you want to:

          A. Keep virtual disk size as small as necessary (Server Virtualization management benefits greatly from this at a small trade off of on demand provisioning performance) as smaller files are faster to                        move during a migration.

          B. Having a large number of file shares that share the same aggregate(pool) of disks where fragmentation isn't of great concern and where more efficient space utilization is desired.

         C. Easier management as all changes to the storage volume occur from the sysadmins point of view, as they simply add the necessary resources and any OS side items don't even need any attention as they thought the space was there all the time, helps when you have 10s,100s, or 1000s of servers.

 

 

and secondly I went on the basis that my pool should be 16TB as 4x4TB = 16TB am I right in thinking I should have made my pool 8TB as I knew I was going to be mirroring them?

 

    - The scenario in which you have placed yourself in is called over-subscription. Where you advertise that you have more resources than on hand. In this case you are advertising that you have 16TB of formatted space on your volume, when in reality you only have 8 TB of useable space after you implemented the 2 way mirror redundancy. This is quite common practice. But must be kept in check as this provides a false sense of security of space available =)

 

 

What I was trying to do was not fool myself by making it larger than it actually was, to put it simply I just wanted it showing exactly what storage I had available to me rather than a pool twice the size of my actual storage, have done the complete opposite here?

     

      - Reading the above facts, i'll leave that decision up to you, not really a right or wrong decision here.

 

And if you don't mind how would I go about creating a new thinly provisioned virtual disk, all my sata ports are full I think except maybe one, I will be backing up the data to larger external drives over the next day or two ready to carry out whatever it is I need to do just in case. I'm thinking of starting again with the storage pool if turns out to be awkward and if I do as per my info above how would you suggest setting it up.

 

      - As i constantly have to do this, I use PowerShell =) But I would suggest that you user the Windows Server Manager, on the File and Storage Services, Volumes, Storage Pools, here you can see your virtual disks and make the changes as you see fit =)

 

Thank you for your advice BTW it's very much appreciated.

 

-You're most certainly welcome! Let me know if you have any additional questions!

 

-Jason

[Th3Architect]

Hi Jason, 

 

Thanks again for the help, so based on the information you've given me here -

 

   - Thick provisioning is great in scenarios where you want to: 

            A. Increase on demand provisioning performance (not necessarily as much of an improvement as it was in the past, with Server 2016,REFS and VHDX's all working on closing                        that performance gap).

            B. Increase Performance by controlling fragmentation of the pool.

    

     -Thin provisioning is great in scenarios where you want to:

          A. Keep virtual disk size as small as necessary (Server Virtualization management benefits greatly from this at a small trade off of on demand provisioning performance) as smaller files are faster to                        move during a migration.

          B. Having a large number of file shares that share the same aggregate(pool) of disks where fragmentation isn't of great concern and where more efficient space utilization is desired.

         C. Easier management as all changes to the storage volume occur from the sysadmins point of view, as they simply add the necessary resources and any OS side items don't even need any attention as they thought the space was there all the time, helps when you have 10s,100s, or 1000s of servers.

 

 

 

Im assuming with thin provisioning if I have a 100GB VD and I put 20GBs of Data in it, that would make it 20GB, if I then put another 40GB in it will make it 60GB, if I were to then delete the 40GB it will stay at 60GB not shrink back down to 20GB as you would expect to see on a normal HDD storage space.

 

If that is correct does this mean I should be using Thick provisioning and will both myself and my clients see this as the actual storage size along with the fact that when I delete data it will actually free up that space?

 

Thanks again and sorry for all the noob q's I'm trying to get my head around S2012 R2, thanks

[BiggHertz]

Hi Jason, 

 

Thanks again for the help, so based on the information you've given me here -

 

Im assuming with thin provisioning if I have a 100GB VD and I put 20GBs of Data in it, that would make it 20GB, if I then put another 40GB in it will make it 60GB, if I were to then delete the 40GB it will stay at 60GB not shrink back down to 20GB as you would expect to see on a normal HDD storage space.

 

If that is correct does this mean I should be using Thick provisioning and will both myself and my clients see this as the actual storage size along with the fact that when I delete data it will actually free up that space?

 

Thanks again and sorry for all the noob q's I'm trying to get my head around S2012 R2, thanks

 

Once the Storage Space Virtual Disk has been provisioned (thick/thin) the footprint of the pool will not automatically shrink the footprint on the pool. This can be accomplished manually(PowerShell use Resize-VirtualDisk Cmdlet), however if your pool has many virtual disks, and you continually grow and shrink virtual disk, fragmentation within the pool will occur.

 

If it is only 1 pool, with a low number of virtual disks, I would just stay thinly provisioned, less wasted space on the pool, leaving you with options like iSCSI, or separate redundancy types, like a 2 way mirror VD, and a 4 disk simple space for speed =) As long as the disk operations on each VD aren't parallel, you're in good shape to get both performance, as well as redundancy on the same pool.  Pretty cool!

 

-Jason 

[Th3Architect]

Hi Jason,

 

So based on the newly attached photo and knowing it's only 1 x pool, 4 x physical drives and what I assume would also be classed as 1 x Virtual Disk, would you say I'm safe to run the powershell resize command? And if so can you type out the exact command so I can just copy and paste into Powershell, thanks a bunch for this you'll be a life saver if I can do this without having to format and start again.

[brwainer]

Hi, I'm just going to add some of my thoughts here, some related to what you're discussing already and some of my own.

You can't resize a Storage Spaces Virtual Disk smaller - it only goes up. The Resize-VirtualDisk commandlet is written to be compatible with many types of virtual disk subsystems, not just Storage Spaces, so even though the commandlet won't complain about inputting a smaller size than you currently have, when it passes it off to Storage Spaces, you'll get an error. Now, this only affects the Advertised size of hte Virtual Disk, not how much space it uses on your physical disks.

If you are on Server 2012 R2 and not Windows 8/8.1, then you really should be using Server Manager to manage your Storage Spaces, not the control panel widget. I wasn't even aware that Server 2012 R2 had that widget available, I thought it was disabled to force you to use the Server Manager. The advantage of Server Manager is that you can see much more information before having to use PowerShell, and you can specify more settings when creating the Virtual Disks. Note that Server Manager calls it Storage Pools, not Storage Spaces.

I have a couple of ideas that you can use, from order of least amount of work to most amount of work. I'm defining work from the perspective of having to copy all your data, which with 4TB would take many hours.

• In PowerShell, run these commands to clean up the amount of physical space used by the virtual disk after it has expanded (The Optimize-Volume PowerShell commandlet is an alternate frontend to defrag that provides all the features expected of PowerShell, such as variables, but the output of the pure defrag command is cleaner)

  defrag D: /D /H /U /Vdefrag D: /K /H /U /V

  /D is a normal defrag, we need to use this to consolidate all the free spaces left from things getting removed. /K means to perform Slab Consolidation, which combines files onto the same drives slabs, wherever possible, thereby allowing slabs to be unallocated from the virtual disk. It also runs ReTrim automatically. Note that you might actually see Virtual Disk size/usage go up during the defrag, this is normal and will be reclaimed either automatically or when you Slab Consolidate and Retrim.

• Unless you are using them for a specific reason, clear out the Shadow Copies on your D volume (But NOT on your backup volume). Shadow copies are snapshots of the drive at specific times, and can use up considerable amounts of space because by design they wil keep around files that were deleted. The easiest way I have foud to do this is:

  - open an explorer window to "This PC"

  - right click your C drive and open the properties

  - go to the Shadow Copies tab

  - select the D: volume

  - Click Settings

  - Set the Maximum Size to the smallest it will let you, usually around 320MB or so

  - Click OK, and all shadow copies are cleared since you lowered the maximum, although sometimes the Properties doesn't update with the new Used amount.

  - Go back to Settings for D and reset it to something reasonable, like 50GBs and click OK on every window you opened.

  This will clear out all Previous Versions and Deleted Files from the D drive, but WILL NOT affect your backups volume, unless you clear the Shadow Copies on that volume directly (you really don't want to do this, it's part of how must backup systems store their data). I say to right click the C drive above because it doesn't like showing the Shadow Copies tab on drives stores on Storage Spaces, it's just a bug, and it's still accessible from the other drives.

  Finally, rerun "defrag D: /K /H /U /V" to make sure it's noticed the newly freed space.

The above two steps are related to reducing a Thin volume to as close to the actual amount of Used space as possible, You'll never get it closer than about 110% of the actual used size, until you get to the maximum size of the Thin volume. Steps beyond this will be related to fixing your advertised size mistake.

• In Disk Manager, shrink your D: drive to 8TB, or rather to 7.27TiB. Now you are only advertising available the amount of space you actually hav available. As long as the second 8TB of the virtual disk is left empty, you shouldn't see it using any space at all, however Windows will probably continue to howl at you that your Pool is low capacity (it will say you are low capacity as long as 75% of your physical space is being used on all but your two least occupied drives. In other words, it wants to have at least two drives at less than 75% capacity). If this isn't good enough for you, next move to:
• Create a new Thin volume of 8TB and move all your data over to it. It has to be a Move, not a Copy, because you won't have space for two copies of your data at once. The D: virtual drive will shrink as files are removed from it, although you may need to follow the two steps I gave you initially a few times during the process in case it doesn't shirnk automatically. Once all files and shares are moved over to the new volume, delete the D: volume and it's storage space.
• If you really want to get away from Thin provisioning, you'll have to do one of two things:

  - Easier: move all data off to another system or drive, with a 6TB or 8TB drive or a Storage Spaces or RAID array, delete your thinly provisioned volume, and create a thickly provisioned volume of the maximum size it will let you (which would be around 8TB or 7.27TiB). Move all your data back.

  - harder: create a 1TB Thick volume in your pool. Move 1TB of data to it. Increase the size of the voume to 2TB. Move another 1TB of data to it. Repeat until all data is moved, and then delete the thin volume.

  In either case Windows will still complain that your pool is low on capacity, but it doesn't matter since you don't have any Thin voumes that might expand on a moment's notice. Note that a single Thick volume using all of the available pool space is not the scenario Storage Spaces was built to address, you are probably better served in this case by using RAID1 from your motherboard or standard Windows Mirrored Volumes as have been available for more than a decade.

I have two servers using Storage Spaces with Server 2012 R2, and I've gotten a good handle on how to use it for the small business / prosumer. One server is using thin provisioned virtual drives on 12TB of disk space, while the other is using Thick provisioned drives with Storage Tiers and Writeback caches on 7TB of drive space and 512GB of SSD space. Storage Spaces works at the small pool capacity level, but it really is built for larger and more robustly designed storage networks.

[BiggHertz]

Hi, I'm just going to add some of my thoughts here, some related to what you're discussing already and some of my own.

You can't resize a Storage Spaces Virtual Disk smaller - it only goes up. The Resize-VirtualDisk commandlet is written to be compatible with many types of virtual disk subsystems, not just Storage Spaces, so even though the commandlet won't complain about inputting a smaller size than you currently have, when it passes it off to Storage Spaces, you'll get an error. Now, this only affects the Advertised size of hte Virtual Disk, not how much space it uses on your physical disks.

If you are on Server 2012 R2 and not Windows 8/8.1, then you really should be using Server Manager to manage your Storage Spaces, not the control panel widget. I wasn't even aware that Server 2012 R2 had that widget available, I thought it was disabled to force you to use the Server Manager. The advantage of Server Manager is that you can see much more information before having to use PowerShell, and you can specify more settings when creating the Virtual Disks. Note that Server Manager calls it Storage Pools, not Storage Spaces.

I have a couple of ideas that you can use, from order of least amount of work to most amount of work. I'm defining work from the perspective of having to copy all your data, which with 4TB would take many hours.

• In PowerShell, run these commands to clean up the amount of physical space used by the virtual disk after it has expanded (The Optimize-Volume PowerShell commandlet is an alternate frontend to defrag that provides all the features expected of PowerShell, such as variables, but the output of the pure defrag command is cleaner)

  defrag D: /D /H /U /Vdefrag D: /K /H /U /V

  /D is a normal defrag, we need to use this to consolidate all the free spaces left from things getting removed. /K means to perform Slab Consolidation, which combines files onto the same drives slabs, wherever possible, thereby allowing slabs to be unallocated from the virtual disk. It also runs ReTrim automatically. Note that you might actually see Virtual Disk size/usage go up during the defrag, this is normal and will be reclaimed either automatically or when you Slab Consolidate and Retrim.

• Unless you are using them for a specific reason, clear out the Shadow Copies on your D volume (But NOT on your backup volume). Shadow copies are snapshots of the drive at specific times, and can use up considerable amounts of space because by design they wil keep around files that were deleted. The easiest way I have foud to do this is:

  - open an explorer window to "This PC"

  - right click your C drive and open the properties

  - go to the Shadow Copies tab

  - select the D: volume

  - Click Settings

  - Set the Maximum Size to the smallest it will let you, usually around 320MB or so

  - Click OK, and all shadow copies are cleared since you lowered the maximum, although sometimes the Properties doesn't update with the new Used amount.

  - Go back to Settings for D and reset it to something reasonable, like 50GBs and click OK on every window you opened.

  This will clear out all Previous Versions and Deleted Files from the D drive, but WILL NOT affect your backups volume, unless you clear the Shadow Copies on that volume directly (you really don't want to do this, it's part of how must backup systems store their data). I say to right click the C drive above because it doesn't like showing the Shadow Copies tab on drives stores on Storage Spaces, it's just a bug, and it's still accessible from the other drives.

  Finally, rerun "defrag D: /K /H /U /V" to make sure it's noticed the newly freed space.

The above two steps are related to reducing a Thin volume to as close to the actual amount of Used space as possible, You'll never get it closer than about 110% of the actual used size, until you get to the maximum size of the Thin volume. Steps beyond this will be related to fixing your advertised size mistake.

• In Disk Manager, shrink your D: drive to 8TB, or rather to 7.27TiB. Now you are only advertising available the amount of space you actually hav available. As long as the second 8TB of the virtual disk is left empty, you shouldn't see it using any space at all, however Windows will probably continue to howl at you that your Pool is low capacity (it will say you are low capacity as long as 75% of your physical space is being used on all but your two least occupied drives. In other words, it wants to have at least two drives at less than 75% capacity). If this isn't good enough for you, next move to:
• Create a new Thin volume of 8TB and move all your data over to it. It has to be a Move, not a Copy, because you won't have space for two copies of your data at once. The D: virtual drive will shrink as files are removed from it, although you may need to follow the two steps I gave you initially a few times during the process in case it doesn't shirnk automatically. Once all files and shares are moved over to the new volume, delete the D: volume and it's storage space.
• If you really want to get away from Thin provisioning, you'll have to do one of two things:

  - Easier: move all data off to another system or drive, with a 6TB or 8TB drive or a Storage Spaces or RAID array, delete your thinly provisioned volume, and create a thickly provisioned volume of the maximum size it will let you (which would be around 8TB or 7.27TiB). Move all your data back.

  - harder: create a 1TB Thick volume in your pool. Move 1TB of data to it. Increase the size of the voume to 2TB. Move another 1TB of data to it. Repeat until all data is moved, and then delete the thin volume.

  In either case Windows will still complain that your pool is low on capacity, but it doesn't matter since you don't have any Thin voumes that might expand on a moment's notice. Note that a single Thick volume using all of the available pool space is not the scenario Storage Spaces was built to address, you are probably better served in this case by using RAID1 from your motherboard or standard Windows Mirrored Volumes as have been available for more than a decade.

I have two servers using Storage Spaces with Server 2012 R2, and I've gotten a good handle on how to use it for the small business / prosumer. One server is using thin provisioned virtual drives on 12TB of disk space, while the other is using Thick provisioned drives with Storage Tiers and Writeback caches on 7TB of drive space and 512GB of SSD space. Storage Spaces works at the small pool capacity level, but it really is built for larger and more robustly designed storage networks.

 

Very good info here +1 

 

Software defined storage solutions are becoming very common nowadays, almost ready for some small tenant production shops. Still some kinks indeed, most of the items in my environment are very large 20+ EMC flash and/or tiered arrays (SAN) and 100+ NetApps (File Level Storage) for these very reasons!

 

-Jason