I want to build Windows storage server

[Berk70]

Hello,

I have a windows pc with a lot of drives both internal and external. They are all shared with SMB and it works but it has no safety and it has become impossible to organize I have HDDs sized from 4TB to 10TB but I am willing to buy couple drives to make pool storage in windows.

It has to be in windows because I only have my desktop which is a gaming pc.

It would be excellent if I could set it with different capacity drives.

I have looked but found it too complex. I want a big single disk with reduntancy as a beginner to this. Do you have any guidance or suggestions?

Thanks.

 

I am on Windows 10 pro

12400f

2x16 gb ddr4 4000mhz ram

rtx 3090

1tb micron 2450 nvme ssd Windows drive

1tb wd sn730 nvme ssd

4tb sata + 5 tb sata + 6tb sata + 2x8tb usb + 10tb usb HDD

Edit: bought 4 16tb drives for this

[leadeater]

You can use HDDs of different sizes but the usable capacity of the pool for a single large volume is effected by this. The best approach for you is to create the pool with only the 4TB, 5TB & 6TB first then add all the other disks after that. During the creation with the first 3 disks create a volume that is single parity. Doing it this way makes the stripe/column number size small as possible which is good for data placement reasons later when you add the multiple larger disks that can take more data placements of the 2 data chunks and 1 parity chunk.

[Berk70]

14 hours ago, leadeater said:

You can use HDDs of different sizes but the usable capacity of the pool for a single large volume is effected by this. The best approach for you is to create the pool with only the 4TB, 5TB & 6TB first then add all the other disks after that. During the creation with the first 3 disks create a volume that is single parity. Doing it this way makes the stripe/column number size small as possible which is good for data placement reasons later when you add the multiple larger disks that can take more data placements of the 2 data chunks and 1 parity chunk.

single parity in windows storage spaces

I have to transfer all files and then create the pool with empty disks right

you said start with small disks and then add large ones for stripe column number I didnt understand it could you please explain or guide me somewhere i could read about this?

Thanks

[leadeater]

17 minutes ago, Berk70 said:

I have to transfer all files and then create the pool with empty disks right

Yes that is correct

 

17 minutes ago, Berk70 said:

you said start with small disks and then add large ones for stripe column number I didnt understand it could you please explain or guide me somewhere i could read about this?

If you create a pool with 8 disks the stripe/column size could default to a larger number. I forget what, I'd have to check the docs. But you can ensure it won't rather simply just be starting with fewer disks.

 

The way Storage Spaces works is data is broken up in to chunks of the column width, basically number of required devices for the volume configuration, and the data is placed on any devices that can satisfy this.

 

So if you have 2 disks of 1TB and 2 disks of 2TB you can fill the entire pool 100% with a single volume and the 2TB disks would have twice the data. This could however only happen in a two-way mirror with column size 1 (1 data + 1 copy, 2 total).

[Berk70]

4 minutes ago, leadeater said:

Yes that is correct

 

If you create a pool with 8 disks the stripe/column size could default to a larger number. I forget what, I'd have to check the docs. But you can ensure it won't rather simply just be starting with fewer disks.

 

The way Storage Spaces works is data is broken up in to chunks of the column width, basically number of required devices for the volume configuration, and the data is placed on any devices that can satisfy this.

 

So if you have 2 disks of 1TB and 2 disks of 2TB you can fill the entire pool 100% with a single volume and the 2TB disks would have twice the data. This could however only happen in a two-way mirror with column size 1 (1 data + 1 copy, 2 total).

Two way mirror is it like raid 1 ? Is it different than parity

4+5+6 tb disk in pool will it have 15tb total or only 12tb and also there is parity will it lose me 4tb or 6tb i really on novice.

thanks.

[leadeater]

7 minutes ago, Berk70 said:

Two way mirror is it like raid 1 ? Is it different than parity

Yes, but it's also like RAID 10 as well. Two-way Mirror can do stripping if there is enough disks in the pool and a larger than 1 column size is used. But the usable capacity is always 50%.

 

7 minutes ago, Berk70 said:

4+5+6 tb disk in pool will it have 15tb total or only 12tb and also there is parity will it lose me 4tb or 6tb i really on novice.

thanks.

I have a tip for you. Go in to disk manager and create the same number of virtual disks (VHDX) as you have but making them only a few GB in size but of the same relative size disks you intend to use i.e. 40GB, 50GB, 60GB, 2x 80GB, 100GB

 

Then mount and add these virtual disks to a Storage Pool and create a volume. You will be able to see the maximum size a single volume would be allowed to be and how much total pool is used.

 

Aka try before you buy

 

[Berk70]

13 hours ago, leadeater said:

Yes, but it's also like RAID 10 as well. Two-way Mirror can do stripping if there is enough disks in the pool and a larger than 1 column size is used. But the usable capacity is always 50%.

 

I have a tip for you. Go in to disk manager and create the same number of virtual disks (VHDX) as you have but making them only a few GB in size but of the same relative size disks you intend to use i.e. 40GB, 50GB, 60GB, 2x 80GB, 100GB

 

Then mount and add these virtual disks to a Storage Pool and create a volume. You will be able to see the maximum size a single volume would be allowed to be and how much total pool is used.

 

Aka try before you buy

 

I have done VHDX of 40 50 60gb and created a storage space with 97GB usable area test location is on an usb ssd but the test file i am copying is copying at 5-20mb a second I have 32gb ram and cpu usage is only at 3 percent is it normal for this to be very slow?

And even empty storage space have 7gb in it isnt it too much

Thanks.

Edit: I have tried it on my main ssd and it is faster uses up to 30 percent of cpu but it doesnt look like it is affected by that. 1.15gb write speed on the ssd is nice without the storage spaces it writes over 2 gb a second but on harddisks i guess the speed will match the disks and I wouldnt lose much.

Thanks for helping me i understand this much better.

Is the storage spaces screen normal it says it uses 7gb but it is empty

While doing the actual disks what are the benefits or harm of using VDHX filen on the disks rather then using the disk as a whole I want your opinion on it? (4tb disk gets 4tb VDHX file rather than the whole disk)

 

 

[leadeater]

52 minutes ago, Berk70 said:

I have done VHDX of 40 50 60gb and created a storage space with 97GB usable area test location is on an usb ssd but the test file i am copying is copying at 5-20mb a second I have 32gb ram and cpu usage is only at 3 percent is it normal for this to be very slow?

No, it'll be an artifact of using virtual disks and allocation unit sizes in how the volume you created was done. You will get better actual performance than this. Also when you do it for real and you format the volume don't use the Windows default allocation unit size, make it much larger and you'll get a lot better performance.

 

Do performance test before putting your data on it.

 

52 minutes ago, Berk70 said:

While doing the actual disks what are the benefits or harm of using VDHX filen on the disks rather then using the disk as a whole I want your opinion on it? (4tb disk gets 4tb VDHX file rather than the whole disk)

Greater risk of data loss and corruption and difficulty in managing the underlying storage. But mainly performance related. By default a VHDX is thin provisioned so when you are putting data in to the pool these backend virtual disks would have to grow in file size and allocate space on the disks which takes time and  that will cause performance loss. Then you have multiple layers of storage sectors and file system allocations.

 

It can be fine, or you can force the VHDX virtual disks to be created at full size, but overall not worth the hassle and risk of problems.

[Berk70]

15 hours ago, leadeater said:

No, it'll be an artifact of using virtual disks and allocation unit sizes in how the volume you created was done. You will get better actual performance than this. Also when you do it for real and you format the volume don't use the Windows default allocation unit size, make it much larger and you'll get a lot better performance.

 

Do performance test before putting your data on it.

 

Greater risk of data loss and corruption and difficulty in managing the underlying storage. But mainly performance related. By default a VHDX is thin provisioned so when you are putting data in to the pool these backend virtual disks would have to grow in file size and allocate space on the disks which takes time and  that will cause performance loss. Then you have multiple layers of storage sectors and file system allocations.

 

It can be fine, or you can force the VHDX virtual disks to be created at full size, but overall not worth the hassle and risk of problems.

Thanks for all.

How about drive failure if a drive goes bad what would i do

[leadeater]

1 hour ago, Berk70 said:

Thanks for all.

How about drive failure if a drive goes bad what would i do

You add another disk to the pool then tell the pool to remove the failed one. It will be marked as failed and you can only remove it after adding a replacement and not before, it won't let you.

 

Try it with your virtual disks, unmount or offline one of them in Disk Manager then try the replacement process with a new VDHX.

Edited June 15, 2023 by leadeater

[Berk70]

10 minutes ago, leadeater said:

You add another disk to the pool then tell the pool to remove the failed one. It will be marked as failed and you can only remove if after adding a replacement and not before, it won't let you.

 

Try it with your virtual disks, unmount or offline one of them in Disk Manager then try the replacement process with a new VDHX.

thanks for everything i will try

[Berk70]

 

On 6/15/2023 at 10:09 PM, leadeater said:

You add another disk to the pool then tell the pool to remove the failed one. It will be marked as failed and you can only remove it after adding a replacement and not before, it won't let you.

 

Try it with your virtual disks, unmount or offline one of them in Disk Manager then try the replacement process with a new VDHX.

Hello again.

I have purchased 4 of 16tb drives taken them from their external enclosure and put them in my pc had to but different cables too etc.

 

Right now i am at the last step of building the pool. I had error of 0x0000000 and restarted pc and ther is no error. it is creating the pool now.

 

But Storage spaces only have 38 TB space and with resiliency it will use all of the disks is that normal? ı am losing nearly 5TB more than 1 disk

 

I have tried writing to it and i am getting 350mb write speed single drive writes at 200 i was expecting 600 is this normal? Is there a setting to change ? or is it my cpu 12400f but usage is very low

Thanks

[leadeater]

25 minutes ago, Berk70 said:

But Storage spaces only have 38 TB space and with resiliency it will use all of the disks is that normal? ı am losing more than 1 disk

Yea, 16TB HDDs are actually 14.55TiB so it should be 43.65TB but there's also a little extra overhead I think with the RAID header and NTFS partitioning so 38.6TB does sound right, but a little less than I would have expected.

[Berk70]

25 minutes ago, leadeater said:

Yea, 16TB HDDs are actually 14.55TiB so it should be 43.65TB but there's also a little extra overhead I think with the RAID header and NTFS partitioning so 38.6TB does sound right, but a little less than I would have expected.

Well 14,5*3 is 43TB how can i lose that much is crazy for me

 

How about the performance it drops to 20 bm right now which is terrbile

[Electronics Wizardy]

44 minutes ago, Berk70 said:

Well 14,5*3 is 43TB how can i lose that much is crazy for me

 

How about the performance it drops to 20 bm right now which is terrbile

By default windows storage spaces sets the the number of columns to 3, so there is 2 data for ever parity. So 58 * .666 = 38.6 which is just bout what you see.

 

The write speeds are caused by the small cluster size on NTFS. If you bump that clusters size up to like 256k you will see much better perf than the 4k default.

[leadeater]

1 hour ago, Electronics Wizardy said:

By default windows storage spaces sets the the number of columns to 3, so there is 2 data for ever parity. So 58 * .666 = 38.6 which is just bout what you see.

I should have done the 66% math lol, oh well my guess at it being higher was wrong.

[Berk70]

@leadeater @Electronics Wizardy I thought it will use one drive for parity and rest would be data

 

If i increase the disk number the performance will not improve as i understand it? Can i change the column number? I have seen ltt stuff that loses 1 drive to parity if i recall correctly from bunch of drives. this home storage thing does not turn out as i expected.

 

I have reformatted with 1mb size crystaldiskmark result attached. file copy test result attached.

 

Is it possible to create 2 raid 0 from 4 drives for twice the performance with raid 1 each other so i have resiliency? speed is in between 350 and 80mb on big video files.

 

cpu usage is low but can it be cpu?

[Electronics Wizardy]

2 minutes ago, Berk70 said:

Can i change the column number?

Yup, it needs to be done in power shell though. Use the -numberofcolumns paramater.

 

 

3 minutes ago, Berk70 said:

Is it possible to create 2 raid 0 from 4 drives for twice the performance with raid 1 each other so i have resiliency? speed is in between 350 and 80mb on big video files.

Yup. Set it to mirror, then number of columns to 2 then you have a raid 10 basically.

 

3 minutes ago, Berk70 said:

cpu usage is low but can it be cpu?

Its probably the cluster size on ntfs. Set ntfs cluster size to 256k and make sure its the same as a data stripe with the virtual disk and you should get good write speeds.

[Berk70]

16 hours ago, Electronics Wizardy said:

Yup, it needs to be done in power shell though. Use the -numberofcolumns paramater

What is ideal for 4 drives?  if i set it at 4 will it write to 3 drives and use 1 for parity? will i gain space. What will happen if i add drives later on can i change it ?

 

16 hours ago, Electronics Wizardy said:

Its probably the cluster size on ntfs. Set ntfs cluster size to 256k and make sure its the same as a data stripe with the virtual disk and you should get good write speeds

I have already formatted the drive with 1mb size which was the highest.

 

edit:I thought with windows storage spaces this would be easy but deeper i go i am more confused. Some people have had issues . i have looked at a lot of guides and it just complicated. do you have anything else that i can run on my windows pc that will get me good speeds and a single drive parity

 

Edit2:Just noticed @Electronics Wizardyyour username here is the same as a youtube channel i have come to while searching for these is it yours?

 

Thanks for everything

[Electronics Wizardy]

7 hours ago, Berk70 said:

What is ideal for 4 drives?  if i set it at 4 will it write to 3 drives and use 1 for parity? will i gain space. What will happen if i add drives later on can i change it ?

 

The parity to data ratio will stay the same. It will just spread the data across more drive if you add more.

 

7 hours ago, Berk70 said:

edit:I thought with windows storage spaces this would be easy but deeper i go i am more confused. Some people have had issues . i have looked at a lot of guides and it just complicated. do you have anything else that i can run on my windows pc that will get me good speeds and a single drive parity

 

Yea its a interesting beast to get working right. Snapraid + drivepool is probalby the next best option, but its not realtime and not made for fast moving data. 

 

If you want parity and mixed drive sizes your making sacrifices unfortunately.

 

7 hours ago, Berk70 said:

 

Edit2:Just noticed @Electronics Wizardyyour username here is the same as a youtube channel i have come to while searching for these is it yours?

 

Yup thats me. 

[Berk70]

13 minutes ago, Electronics Wizardy said:

The parity to data ratio will stay the same. It will just spread the data across more drive if you add more

So if i set it to 4 capacity will not change but will it be faster ? or slower? and if i lose a drive with different column numbers wahat would change?

 

Another thing is the optimization button on storage spaces what does it do?

 

16 minutes ago, Electronics Wizardy said:

If you want parity and mixed drive sizes your making sacrifices unfortunately.

with only same sized drives 4x16 do you have ther recommendation?

[Electronics Wizardy]

6 minutes ago, Berk70 said:

So if i set it to 4 capacity will not change but will it be faster ? or slower? and if i lose a drive with different column numbers wahat would change?

 

Another thing is the optimization button on storage spaces what does it do?

If you set number of columns to 4 you will get more capacity as there is basically 1 parity drive worth of data now. Should give you about 43.5TiB usable.

 

If you lose a drive you need to replace it, and it will show a missing drive. you need at lease the number of columns worth of drive to have it in good condition.

 

ANouther options would be a raid card, but that needs new hardware.

 

OPtimization spreads the data across the different drives(I think). Its good if you add a new drive to the pool so all the drives have equal usage.

[leadeater]

13 hours ago, Electronics Wizardy said:

OPtimization spreads the data across the different drives(I think). Its good if you add a new drive to the pool so all the drives have equal usage.

Correct, if you add a new drive later the pool will not balance old data unless you tell it to i.e. Optimization.

[Berk70]

Thanks a lot @leadeater and @Electronics Wizardy

 

Unfortunately I still havent migrated to the storage space.

 

I have seen your guide and experimenting. @Electronics Wizardy

 

With 4x16tb disks if i set the column size to 4 and 256kb and ntfs is formatted as 1mb i get 20mb write speeds with 75% capacity

 

With same 4 drive but column set to 3 256kb and ntfs is formatted as 512kb i get 80 sometimes sometimes 300 but i only have 66% of capacity

 

So I had bought another 16tb drive My drives cost more than my desktop and laptop right now...

 

new-virtualdisk -storagepoolfriendlyname "Depolama havuzu" -FriendlyName parity -UseMaximumSize -ResiliencySettingName parity -Numberofcolumns 5 -Interleave 256kb

 

With this settings i am getting 80% space and good speeds. NTFS is formatted as 1MB

 

Commands for copy pasting if you know what you doing! This is not a guide!

 

To see the storage pool

get-storagepool

To see the virtualdisk that are created from the storage pool

get-virtualdisk

Delete the virtualdisk storage pool will stay

get-virtualdisk | remove-virtualdisk

Make 3 disk storage pool.* NTFS shlould be formatted as 512kb*

new-virtualdisk -storagepoolfriendlyname "Depolama havuzu" -FriendlyName parity -UseMaximumSize -ResiliencySettingName parity -Numberofcolumns 3 -Interleave 256kb

Make 5 disk storage pool.* NTFS shlould be formatted as 1MB*

new-virtualdisk -storagepoolfriendlyname "Depolama havuzu" -FriendlyName parity -UseMaximumSize -ResiliencySettingName parity -Numberofcolumns 5 -Interleave 256kb

*Depolama havuzu should be name of your storage if english it is "Storage pool"

 

[leadeater]

27 minutes ago, Berk70 said:

With this settings i am getting 80% space and good speeds. NTFS is formatted as 1MB

Nice, ~700MB/s is more along the lines of what I was expecting. Unfortunate it required so many disks in the pool to achieve such a huge jump but it seems to be very sufficient now.