Cache Drive ??

[StarTrek133]

Hey Guys, 

I need some help .. I am currently building a Windows 11 Media / NAS box .. I have a spare 256gb SSD that I want to use as a cache drive if I can .. 

 

I have a 256gb M2 drive for the OS and then a Raid Array for all my data .. So hoping I can make a cache to help the Raid Array to not slow down or chug .. if that makes sense .. 

 

Thoughts ? 

[Bombastinator]

It at least used to be doable. You’d set up everything in a raid array so it looked like one drive, and then use options to use the ssd as cache.  Whether it would make things functionally faster would depend on the connection you were using though.  If the connection is slow enough that the HDDs could saturate it by themselves it would do no good at all.

[cmndr]

On 9/16/2022 at 2:02 PM, Bombastinator said:

It at least used to be doable. You’d set up everything in a raid array so it looked like one drive, and then use options to use the ssd as cache.  Whether it would make things functionally faster would depend on the connection you were using though.  If the connection is slow enough that the HDDs could saturate it by themselves it would do no good at all.

It'll depend on the use case.

 

Even then, there's  A LOT of use cases where adding a cache might take the array read speed from 10MBps to 100MBps...

----

As far as using the drive as cache... there's minimal harm to be done in doing so, assuming you have an acceptable backup strategy. My anecdotal experience of adding a cache to a budget NAS was that it made it night/day faster, even over a 1Gbps interface. Thumbnail generation got A LOT better, as did random directory searches (after warming up the cache by searching for ".")

 

 

 

 

 

[Bombastinator]

36 minutes ago, cmndr said:

It'll depend on the use case.

 

Even then, there's  A LOT of use cases where adding a cache might take the array read speed from 10MBps to 100MBps...

----

As far as using the drive as cache... there's minimal harm to be done in doing so, assuming you have an acceptable backup strategy. My anecdotal experience of adding a cache to a budget NAS was that it made it night/day faster, even over a 1Gbps interface. Thumbnail generation got A LOT better, as did random directory searches (after warming up the cache by searching for ".")

 

 

 

 

 

That’s a fast enough connection.  It’s a lot faster than many types of wifi though.  I don’t know what the connection to the NAS is.

[StarTrek133]

Thanks Everyone for the advice .. 

 

I am just hoping it will help the raid card not lag or hiccup when its trying to read and write .. So that is why I thought a Cache drive would help .. I also need to know how to create that cache drive as its something I have never done before .. I would assume its in computer management but also could be wrong .. 

 

Thanks .. 

[tkitch]

IDK if Win11 supports that in storage spaces, but TrueNAS Core and similar system do, and very easily at that.

 

[StarTrek133]

Thanks @tkitch for the advice .. Sadly I tried to do a TrueNas Scale system and after a week of trouble shooting and trying to get help from the community we never got any of the SMB shares to work and the VM's would not start either .. So I gave up and went back to what I know which is windows based .. Which is a shame as I didn't want to spend the extra $$$$ on a new raid card but what was I going to do .. 

[tkitch]

Be very very careful with hardware RAID anymore....

 

 

(IDK why you'd have had a ton of SMB issues in Truenas?  I setup Truenas Core, and got SMB running basically instantly.)

[StarTrek133]

Hey @tkitch I don't know why either .. I had a bunch of people helping me in the fourms and no one could understand what was going on .. I would create the pools , datasets, user, permissions, everything like it says in the video and when you try to access from a windows machine you just keep getting the username and password prompt over and over and over again .. no matter what you did never worked .. 

 

Also I could never get a vm to run either .. keep saying the drive wasn't there when you went to the pool it was there .. So I don't understand it was my first try at TrusNAS and I really wanted it to work .. As I wanted to not have to deal with a hardware raid card .. But what else can I do .. 

[SVThuh]

I know you tried TrueNas, but have you tried UnRaid?  Adding a cache drive is super easy, scheduling the mover is a piece of cake as well as scheduling TRIM with the SSD as well.  Supports VM's and Docker.  Might be worth it to give it a shot, if you don't like it, its as easy as going back to plan B.. or C or whichever one you're on now.

[StarTrek133]

Thanks @SVThuh I did not think of UnRaid as I thought you had to pay for that .. But if it will let me do raid , and a windows VM .. then it might be worth looking into .. 

 

Hhmm interesting .. 

[cmndr]

On 9/18/2022 at 11:20 PM, Bombastinator said:

That’s a fast enough connection.  It’s a lot faster than many types of wifi though.  I don’t know what the connection to the NAS is.

That anecdote would've been over 1Gbe.

If you're pulling a steady 100MBps from a NAS it'll be "OK" for most things.
If there's no cache and the speed drops to 0.5MBps it'll SUCK.

 

It won't matter if your max speed is 1000MBps if the drives will only do 0.5-5MBps when they're doing a lot of random IO.

[Bombastinator]

18 minutes ago, cmndr said:

That anecdote would've been over 1Gbe.

If you're pulling a steady 100MBps from a NAS it'll be "OK" for most things.
If there's no cache and the speed drops to 0.5MBps it'll SUCK.

 

It won't matter if your max speed is 1000MBps if the drives will only do 0.5-5MBps when they're doing a lot of random IO.

The first sentence doesn’t make sense to me.  Doesn’t mean it doesn’t make sense.  I seem to be failing to parse it though.

Really small files will slow down a HDD or any other device.  Part of the price paid for 64bit addressing. If you’re moving around a lot of 4k files and you do it repeatedly the speed of an SSD can make that up again. Assuming you can pull them from cache.  Cache doesn’t help if the data isn’t in cache in the first place.  Kind of depends what one is doing I guess.  

[cmndr]

19 hours ago, Bombastinator said:

The first sentence doesn’t make sense to me.  Doesn’t mean it doesn’t make sense.  I seem to be failing to parse it though.

Really small files will slow down a HDD or any other device.  Part of the price paid for 64bit addressing. If you’re moving around a lot of 4k files and you do it repeatedly the speed of an SSD can make that up again. Assuming you can pull them from cache.  Cache doesn’t help if the data isn’t in cache in the first place.  Kind of depends what one is doing I guess.  

TLDR: caching is amazing at keeping the worst case scenario on HDDs from happening and can often get you an "SSD" like experience, even over a 1Gbe connection.

 

I had previously referred to a personal anecdote of a "dumb" SSD cache (LIFO queuing on a 5 year old ARM based QNAP NAS) making an HDD based NAS much snappier. The bottleneck wasn't the network, it was the HDDs.

 

What you're doing (use case, attributes of your data) WILL matter of course. It affects how quickly/easily the cache will fill, how much you need, how sensitive the system is to cache misses, and so on. The ideal case scenario is where you have all drives working together (think 2000MBps SSD handling 90% of hits and it being maxed at 2GBps and a 4 HDD array capable of hitting 800MBps also hitting its limit for something like 2500-2800MBps total throughput - the medium-bad scenario is you avoid the HDDs array grinding to a halt at 0.5MBps)

For the sake of argument, assume that "streaming workloads" are fast enough (think reading single big files where all the data is more or less stored contiguously and you're getting 100-200MBps off the disks and saturating 1Gbe networking) off of HDDs. What HDDs suck at is random reads. Up to 1000x slower and you can FEEL it.

If you have something like ZFS (or PrimoCache) 16-64GB of cache (RAM) will get you a 50+% cache hit rate on most set ups, even upon a fresh reboot where there's basically nothing in the RAM. Mostly just because metadata (info on where ON the HDD the data is stored) is relatively easy to fit in a cache and prefetching is REALLY good.

If you're using something dumber than ZFS (think last in first out, aka most recently used - which is common on some cheaper NAS's built in OS), then you'll need more capacity. Using a cheepo 256GB SSD and setting aside 200GB or so for caching you can realistically get 50-99.9% cache hit rates most of the time. This means you'll have "good enough" performance for loading/generating thumbnails, searching modestly sized directories, etc.
()
In practice relatively little data is "hot". On a 1TB dataset, 80% of the reads might come from just 10% of the data (illustrative numbers).
If your cache is smart enough to get 90% of the hot data cached and 1% of the not-hot data cached, then ~100GB cache total would get you would result in a hit rate of around 75% (sometimes higher, closer 100%, sometimes lower - ideally with it biased towards the hits mostly being to small blocks of data and large streaming workloads will NOT being cached - so you get SSD speeds when HDDs are at their weakest and the HDDs will stream at ~200MBps each when they're at their strongest).

 

Here's an extreme case where I basically have cache for days (32GB RAM + 118GB optane on NAS + 58GB optane locally) for a 4 HDD array on a 10Gbe NAS...
I'm getting 4x better random IO, on average, than a Samsung 980Pro on a local machine. (note - this benchmark is fundamentally flawed for assessing performance this way but "gut feeling" checks don't have it much different than a decent nvme drive or even a 1.5TB optane drive on a local system - it's "good enough")