Linux caching solutions

asquirrel · November 6, 2020

What I want to build: Home NAS to store and (possibly) transcode movies (IE: I already own the movies, I will transcode them to h265 (until av1 becomes better) and then play them back; no need for plex as I don't need the live transcoding/streaming). It will also run several crytocurrency full nodes (ETH, BTC, XMR). This warrants at least two tiers of storage. SSD for ETH (because the devs decided on a really dumb storage architecture -- they store each transaction individually; billions of 64 byte strings, rather than storing blocks like BTC does) and HDD for the movies/BTC/XMR. In the case of movies, these files are massive (10+GB each) I want to be able to move them around (to/from my desktop) in a reasonable amount of time, which means the 100MB/s or so I could get from RAID/unRAID/etc is not ideal.

I looked into bcache, but in every sequential write test I can find, it basically runs no faster than the HDD under it, or at best doubles the performance of that HDD, for sub-cache-size writes. This is not what I want. If I have a 200GB SSD cache in front of a 10TB hard drive, and the cache is empty, I want to write 200GB to that cache at 550MB/s (sata 6 limit) before I see a speed drop-off to what the backing HDD can handle. "But the cache needs metadata..." it should be flushing to disk during that transfer, and if it can't move 10GB to the HDD by the time 200GB gets written, so help me...

There doesn't seem to be a linux caching solution that does this. Can I just not find it? Am I searching for the wrong thing? Am I doomed to just have one LUN of pure SSD just for moving files there, and writing my own daemon to destage data out of the cache? What's the deal?

Edit: Movie transcodes. Whether I transcode them on my desktop or on the NAS is yet to be determined. For a long time my 'NAS' was a mini-PC with a 13W peak power draw and a large external hard drive. I'm interested in upping my game slightly with the Zen 3 launch (a 5600X would be dope). But it might also make sense to just get a lower power usage CPU for the NAS and get myself a 5900X in my desktop, then do the transcodes on my desktop first and put the results on the NAS. *shrug* I haven't decided.

Electronics Wizardy · November 6, 2020

3 hours ago, asquirrel said:

in a reasonable amount of time, which means the 100MB/s or so I could get from RAID/unRAID/etc is not ideal.

What speeds do you want? What network speeds do you have to your nas? Normally you rlimited by 1gbe.

3 hours ago, asquirrel said:

I looked into bcache, but in every sequential write test I can find, it basically runs no faster than the HDD under it, or at best doubles the performance of that HDD

There is some tweaking, but most caching isn't made to improve sequentical speeds as hdds are already pretty good at that. Try tweaking bcache.

But raid should help a good amount here, you can easy get 500+mBs from a few hdds