storage spaces MS Storage Spaces - Plex - SSD Cache

[ObiWanMonk]

Ok i will start off by explaining the situation, Im a serious media junky/photographer/Videographer who keeps all his raws on  Archive and want to add a layer of protection in instead of just using cold storage.

 

I have a Large scale Plex library that i need Parity to give some fail over safety. Not got matching drives so rules Raid out in the normal sense. Currently running around 14TB of Media from 2x 8TB WD Golds with 44TB potentially available.

Looked into FlexRaid, ZFS, UnRaid, Raid 1-5 as an option but haven't been able to settle on a hard decision.

 

Windows Storage spaces certainly looks interesting, Next to no press coverage about it, Appears to have ability to add SSD's as a write cache and read speeds i would imagine would be absolutely fine as far as Plex goes on the network but not sure. I Cant justify a full custom server atm so need to make do with hot swap bays and my Lian Li Cube with Icydock hotswap drives (4x http://www.icydock.com/goods.php?id=157) and a 6x bay 2.5" enclosure (http://www.icydock.com/goods.php?id=151)

 

If anybody has any good ideas as far as what would be best to maximise performance given my situation.

 

For not my Drives are as follows:

2x 8TB WD Gold Enterprise 

2x 6TB Seagate Enterprise 

2x 4TB WD SE

2x 4TB WD Green

5x 2TB WD Green (currently in a drobo but can be transferred and made usable)

 

Totally open to any advice given! Thanks! 

[Electronics Wizardy]

Storage spaces parity has very slow writes. Like 40mBs slow. It's that's fine use it. Just start adding drives. You can can use the ssd as a cache drive if you want it faster. 

[ObiWanMonk]

24 minutes ago, Electronics Wizardy said:

Storage spaces parity has very slow writes. Like 40mBs slow. It's that's fine use it. Just start adding drives. You can can use the ssd as a cache drive if you want it faster. 

Nope defo not acceptable! need something that gives at least native speeds. I hear DrivePool is very good as well and very customiseable? 

[Electronics Wizardy]

2 minutes ago, ObiWanMonk said:

Nope defo not acceptable! need something that gives at least native speeds. I hear DrivePool is very good as well and very customiseable? 

Your on Windows and can't change that right?

 

If you want parity do drive pool + snap raid. 

[ObiWanMonk]

4 minutes ago, Electronics Wizardy said:

Your on Windows and can't change that right?

 

If you want parity do drive pool + snap raid. 

Sadly im stuck on windows as i need the pne system that does everything. its only as a file server, Plex, SABNZB, Sonarr and then just some light gaming which never effects performance really anyway as it wont be linekd to the pool. Can you run snap raid on windows?! (sorry if a stupid question)

[brwainer]

38 minutes ago, Electronics Wizardy said:

Storage spaces parity has very slow writes. Like 40mBs slow. It's that's fine use it. Just start adding drives. You can can use the ssd as a cache drive if you want it faster. 

While the first part is true, the second is not. If you want to use tiered virtual disks in storage spaces, the SSDs have to be present in storage spaces from the beginning, there is no way to convert a non-tiered disk to a tiered one. If you add SSDs later, then you have to create new virtual disks and copy the data over.

 

@ObiWanMonk you can totally go with Storage Spaces, that's what I'm using. If you were to go this route, I have a few recommendations:

• Use Server2016, not anything older nor consumer windows. Server2012R2 isn't as refined for Storage Spaces and ReFS, and the UI in regular Windows is very limiting (although you can actually do anything on regular Windows using PowerShell). EDIT: I recommend a Server2016 Essentials license if you don't have a server key yet
• Have at least 2 Sata SSDs, 4 or more preferred (excluding your boot disk). With 2016 you can have each tier be a different type, e.g. SSDs mirrored and HDDs Parity. Also with at least 2 SSDs in the pool, by default all virtual disks will get a 1GB writeback cache on the SSDs (mirrored) and you can define a larger writeback cache at the time of creation.
• Parity virtual disks are fine for anything that will be read more than written to - e.g. Your Plex library, and medium/long term archives. For anything that is actively being worked on, I recommend a Mirrored virtual disk. For more performance you can make it both mirrored and striped (in storage spaces terms, this would be "Number of data copies 2, Number of columns 2") which is akin to RAID10 and requires 4 disks (and 4 SSDs if tiered).
• Don't worry about tiering for anything that would be randomly accessed or where you would read the whole file at once (large video files, particularly your Plex library)
• Use ReFS unless you have something that doesn't like it. For example, Plex doesn't mind having the source files be on ReFS, but the folder where it stores all the data about your library has to be NTFS, otherwise you have an issue with downloading metadata, particularly posters and other images. I delt with that by having a volume for the library and a volume for the metadata.

[brwainer]

9 minutes ago, ObiWanMonk said:

Nope defo not acceptable! need something that gives at least native speeds. I hear DrivePool is very good as well and very customiseable? 

Once you put all your drives into the pool, you choose what type of redudnacy (2x mirror, 3x mirror, parity with 1 disk redudnacy, parity with 2 disk redundancy) on each virtual disk you make. So you can have a working disk that is mirrored and an archive disk that is parity. Also, if you add in SSDs, you can manually create a Parity virtual disk that has a very large writeback cache. All writes hit this cache first and are written to the parity as it goes on.

[Electronics Wizardy]

13 minutes ago, ObiWanMonk said:

Sadly im stuck on windows as i need the pne system that does everything. its only as a file server, Plex, SABNZB, Sonarr and then just some light gaming which never effects performance really anyway as it wont be linekd to the pool. Can you run snap raid on windows?! (sorry if a stupid question)

could you run linux on the host and windows in a vm. zfs seems like a good solution here or merger fs + snapraid.

[leadeater]

@ObiWanMonk

 

Also a Storage Spaces user and same advice as @brwainer, use Windows Server 2016.

 

The secret sauce that makes Server 2016 work better is Multi Resilient virtual disks, ReFS required. This adds real time tiering with the addition of all writes, including modifying existing data, hitting the faster tier.

 

Currently I have two servers running Windows Server 2016 Storage Spaces, one with 6 Samsung Pro SSDs and 3 HDDs and another server with 2 SSDs and 6 HDDs.

 

The other thing to be aware of is how Storage Spaces pooling actually works. When you add disks in to a pool the pool itself doesn't define the the resiliency/RAID configuration at all, you do that when creating virtual disks within the pool. What that means is you can have a virtual disk using a portion of your SSDs (mirror) and HDDs (parity) for archive type storage and then another using SSD (mirror) and HDD (mirror) for tasks that requires higher performance or even a pure SSD virtual disk like I do for hosting VMs.

 

Setting up those custom virtual disks and their tiers does require that you use PowerShell to do it properly as it can't be done in the GUI.

 

Another thing to be aware of is in Storage Spaces the resiliency settings are done on the data chunks not on disks, that means mirror spreads multiple copies across as many disks as you like so it's not RAID 1 or RAID 10 but it is closer to RAID 10 in concept.

 

As you can see if you get it right Storage Spaces can give you amazing performance, get it wrong and it will be shit.

[ObiWanMonk]

19 hours ago, Electronics Wizardy said:

14 hours ago, leadeater said:

@ObiWanMonk

 

Also a Storage Spaces user and same advice as @brwainer, use Windows Server 2016.

 

The secret sauce that makes Server 2016 work better is Multi Resilient virtual disks, ReFS required. This adds real time tiering with the addition of all writes, including modifying existing data, hitting the faster tier.

 

Currently I have two servers running Windows Server 2016 Storage Spaces, one with 6 Samsung Pro SSDs and 3 HDDs and another server with 2 SSDs and 6 HDDs.

 

The other thing to be aware of is how Storage Spaces pooling actually works. When you add disks in to a pool the pool itself doesn't define the the resiliency/RAID configuration at all, you do that when creating virtual disks within the pool. What that means is you can have a virtual disk using a portion of your SSDs (mirror) and HDDs (parity) for archive type storage and then another using SSD (mirror) and HDD (mirror) for tasks that requires higher performance or even a pure SSD virtual disk like I do for hosting VMs.

 

Setting up those custom virtual disks and their tiers does require that you use PowerShell to do it properly as it can't be done in the GUI.

 

Another thing to be aware of is in Storage Spaces the resiliency settings are done on the data chunks not on disks, that means mirror spreads multiple copies across as many disks as you like so it's not RAID 1 or RAID 10 but it is closer to RAID 10 in concept.

 

As you can see if you get it right Storage Spaces can give you amazing performance, get it wrong and it will be shit.

 

Ok thats amazing! The problem i have above all is this is going to be done on my main rig which is a do everything system, windows server i dont think is really the best option,  i did read however you can still do everything you need via windows 10 pro & powershell to get it up and running in the same way but is the GUI the only thing that missing?

 

I did look into DrivePool and SnapRAID a bit more. certainly an option but the idea that windows storage spaces actually does everything is the thing that is most intriguing. 

Is it possible for NVMe SSD's to be used as Cache Drives for the main pool??

[leadeater]

24 minutes ago, ObiWanMonk said:

Ok thats amazing! The problem i have above all is this is going to be done on my main rig which is a do everything system, windows server i dont think is really the best option,  i did read however you can still do everything you need via windows 10 pro & powershell to get it up and running in the same way but is the GUI the only thing that missing?

 

I did look into DrivePool and SnapRAID a bit more. certainly an option but the idea that windows storage spaces actually does everything is the thing that is most intriguing. 

Is it possible for NVMe SSD's to be used as Cache Drives for the main pool??

Sort of, Windows 10 doesn't support the advanced features of Storage Spaces like tiering. You can however add SSDs to the pool and set their mode to Journal which means they will cache writes but not reads.

 

Far as Windows Servers goes there isn't actually much difference, really it's just a cost thing.

 

If possible just add a few disks, 4-6, and two SSDs to a Windows 10 system and try it out to see what the performance is like. If it's not up to par then move on to trying DrivePool and SnapRAID. Finally you can download Windows Server 2016 trial and try out the more advanced Storage Spaces configuration. Once you know how each of them performs you can make an assessment on which is your best option regarding performance, cost and usability.

[NZLaurence]

I will just jump in here and add that storage spaces isn't really something designed to run on your desktop machine. A lot of the advanced stuff that makes it worth running is missing from win10. (Like tiers).

 

For what you are doing I would consider a reasonable second hand hardware raid card as well, if you don't already have one. Basic raid 10 or 1 give good performance with some redundancy for your main desktop.

And building out a smaller server with lots of drives as an archive machine running freenas or storage spaces or something. As an archive machine performance may not be as important.

[ObiWanMonk]

Kinda thinking it might be time to move my workstation over to windows Server 2016!  Do i actually lose anything by going over from Win10 Pro or is it just gains and stability??

 

What i did wonder was my options as far as raid goes as i have a complete hotch potch of drives. 2x 8TB, 2x 6TB, 2x 4TB, 2x 2TB. I did wonder if i am better having my drives mirrored to the same size drives and then pool those mirrors and integrate a NVMe SSD Cache for write cache. Only get half storage but got redundancy across everything. 

Any thoughts folks??

 

Is the general thoughts that Storage Spaces DIrect is the way to go for if i defect to Server 2016? if not use drive pool??

[brwainer]

2 hours ago, ObiWanMonk said:

Kinda thinking it might be time to move my workstation over to windows Server 2016!  Do i actually lose anything by going over from Win10 Pro or is it just gains and stability??

 

What i did wonder was my options as far as raid goes as i have a complete hotch potch of drives. 2x 8TB, 2x 6TB, 2x 4TB, 2x 2TB. I did wonder if i am better having my drives mirrored to the same size drives and then pool those mirrors and integrate a NVMe SSD Cache for write cache. Only get half storage but got redundancy across everything. 

Any thoughts folks??

 

Is the general thoughts that Storage Spaces DIrect is the way to go for if i defect to Server 2016? if not use drive pool??

Storage Spaces* is the way to go if you can add 2-4 Sata SSDs and/or 2 NVMe SSDs (I don't think you can write cache with just one device, that would give you a risk of data loss). Storage Spaces Direct is a very different thing that you don't need to be thinking about.

 

Don't mix multiple different types of drive management. "I did wonder if i am better having my drives mirrored to the same size drives and then pool those mirrors" - this sounds like you are thinking about doing a mirror with hardware RAID, and then putting those virtual drives into Storage Spaces. This is bad for two reasons: 1. If you have a power outage, there may be data that Storage Spaces thinks is written to disk, but the hardware RAID still isn't done with. This would be lost. 2. It is well known and proven that most hardware RAID controllers do RAID 1 very naively - they literally write the data to both disks can call it a day. Storage Spaces and ZFS (not sure about others like btrfs) employ additional checksums so that when data from a mirrored virtual disk is read, they know whether it is corrupted and can compare data from the other drive (with Storage Spaces this requires the filesystem to be ReFS).

 

You can definitely use Server versions of Windows for a workstation, I've done it in the past with 2012R2. Aside from enabling the full GUI, there are other things you can tweak to make it more friendly for use this way. See http://www.windowsworkstation.com/Win2016/ - I used their 2012R2 guide back when I ran that as my workstation OS.

 

If you decide to move to Server2016, definitely use Storage Spaces. Anything that is a 3rd party addition to Windows would be inferior. Even if you stay on Windows 10 Pro, I personally would continue using Storage Spaces. As leadeater said,

On 5/11/2017 at 9:56 AM, leadeater said:

You can however add SSDs to the pool and set their mode to Journal which means they will cache writes but not reads.

you just have to set the Usage by powershell, and do this before making any virtual disks.

[JoeyDee86]

Sorry, I know this is an old thread, but I wanted to post this so people know who find it in a google search.

 

Just as an FYI, I’ve been running real-time tiering on my desktop with Windows 10 Ent for over a year now. Knock on wood, performance has been great. My only complaint is that you need to allocated the storage when you create the disk, aka, no dynamically expanding volume. This was a huge issue for me when I realized that my downloads folder, which is p2p, should not be on my tiered drive, as that data is going to eat up my performance tier. So, I moved them to a normal mirror volume to solve that, but I was very low on space because of the forced allocation of the tiered drive...

 

Has anyone gotten nvme caching to work correctly though with a single server? 

[dalekphalm]

On 5/12/2017 at 10:47 AM, ObiWanMonk said:

Kinda thinking it might be time to move my workstation over to windows Server 2016!  Do i actually lose anything by going over from Win10 Pro or is it just gains and stability??

 

What i did wonder was my options as far as raid goes as i have a complete hotch potch of drives. 2x 8TB, 2x 6TB, 2x 4TB, 2x 2TB. I did wonder if i am better having my drives mirrored to the same size drives and then pool those mirrors and integrate a NVMe SSD Cache for write cache. Only get half storage but got redundancy across everything. 

Any thoughts folks??

 

Is the general thoughts that Storage Spaces DIrect is the way to go for if i defect to Server 2016? if not use drive pool??

Another alternative is FlexRAID - I used to use it extensively for 2-3 years, but a couple years ago, switched over to ESXi and FreeNAS when I got my new server and was able to properly plan a matching set of HDD's.

 

FlexRAID is quite flexible (duh), and speed wise, you're looking at single HDD native speeds (80-100 MB/s, maybe more depending on your HDD's). You can mix and match HDD sizes, create "virtual" HDD's from several smaller HDD's, etc.

 

@AshleyAshes is essentially our resident expert on that, these days.

 

It's cheaper than a Server 2016 license, and in many ways, might be more suitable to your needs.

 

Worth exploring, at the very least.

[AshleyAshes]

17 minutes ago, dalekphalm said:

FlexRAID is quite flexible (duh), and speed wise, you're looking at single HDD native speeds (80-100 MB/s, maybe more depending on your HDD's). You can mix and match HDD sizes, create "virtual" HDD's from several smaller HDD's, etc.

 

@AshleyAshes is essentially our resident expert on that, these days.

I also love that with FlexRAID, I can just add more drives as I have demand.  so I only have to purchase drives when I need them.    Though if drives larger than 8TB ever hit the $/GB sweet spot I'll have to swap out the parity drive first.  But even that's not a HUGE problem since any OLD parity drive can be repurposed into a storage drive.

[dalekphalm]

1 minute ago, AshleyAshes said:

I also love that with FlexRAID, I can just add more drives as I have demand.  so I only have to purchase drives when I need them.    Though if drives larger than 8TB ever hit the $/GB sweet spot I'll have to swap out the parity drive first.  But even that's not a HUGE problem since any OLD parity drive can be repurposed into a storage drive.

Yep, having to plan your HDD expansion (or purchases, in the first place) is goddamn annoying for people who can't just drop $1000 on 6 new HDD's all in one go.

 

That was by far my favourite thing about FlexRAID.

[AshleyAshes]

1 minute ago, dalekphalm said:

Yep, having to plan your HDD expansion (or purchases, in the first place) is goddamn annoying for people who can't just drop $1000 on 6 new HDD's all in one go.

Yeah.  Granted such systems offer vastly superior protection and are ideal for enterprise or just very important data.  There's totally situations I'd never advise someone to use FlexRAID in.

[dalekphalm]

1 minute ago, AshleyAshes said:

Yeah.  Granted such systems offer vastly superior protection and are ideal for enterprise or just very important data.  There's totally situations I'd never advise someone to use FlexRAID in.

Yeah I'm not exactly gonna replace our ESXi + EqualLogic SAN with FlexRAID

 

I'd probably get fired.