How to over-complicate your storage, because you can't stop playing with software that looks fun..

[RobFRaschke]

I recently lost a 3TB drive that had a LOT of my legacy data on it. Within a week prior to that, I lost my remote back-up of said legacy data. All of my actually important data was duplicated across two drives, but not everything. I find myself very frustrated by this occurrence as I went without a major drive failure for a LONG time. I have drives that have 10 years of service on them without a failure, and were only retired due to being outdated in size(80GB, 320GB, 500GB >10 years, 3x 1TB drives >5years) and now I have drives failing left and right.

 

So, I'm re-working my storage solution, again. As it stands, the hardware I'll be using is:

 

1xWD Black NVMe 250GB drive for boot, OS and critical applications

 

1xSamsung 860 Evo 1TB SATA SSD for cache drive

 

1x Seagate ES.2 3TB drive for game drive

 

2x WD Red 8TB drives for legacy storage

 

And the software will be Drivepool, duplicating all legacy data across both 8TB drives, with an 800GB Write through cache on the 860 EVO. And PrimoCache on the remaining 200GB of the 860 EVO to read and write cache the 3TB game drive. I'll then be adding another layer of Drivepool on top that will make both sets of drives appear as a single drive, with a single letter, but with each of the primary 11 folders being given specific sub-pools of drives to write data to. Because hey, why not? Now, the order of operations for migrating to this drive structure is actually very important to minimize the chance at data loss and causing a PEBKAC error.

 

I've recieved and installed the two WD Red NAS drives(from WD Elements external drives for $140+tax ea.) and gotten them installed. The default partitioning looked good, so I left it alone. I've added both drives to my existing drive pool, which had the 860 Evo write caching for the remaining 3TB drive, and am now removing the 3TB drive, which will re-allocate the data to the 8TB drives.

 

I can then install primocache and reboot the system for it to take effect. I'll set up the caching using primo-cache for the 860 EVO and 3TB drive, and migrate all my game data to that drive set.

 

I can then enable folder duplication on the entirety of the remaining data on the original drivepool, which now has the 860 EVO for write cache, and 2x8TB drives for data storage. DrivePool, using the SSD optimizer plug-in will write incoming data to the 860 EVO, then as it writes out to the spinning disks, will automatically duplicate the data for me.

 

I can then create a new pool in DrivePool, assign the game drive, which will be seen as a single drive, and the legacy storage pool, which it will see as a single drive. Unfortunately, I then have to manually move the data from the drive or pool, into the newly created pool folder to make it a part of the new top tier pool, then prior to letting the system balance the pool across all drives, assign drive locations to each of the 11 top folders to ensure they stay on their correct, respective drives or pools. This will put all the data on a single drive letter for me, without any Windows dynamic links.

[RobFRaschke]

So, after running in the previously stated setup for a week and monitoring cache hit rates, the 500GB partition that was set in PrimoCache for read and write caching of the dual 8TB drives was having decent cache hits on reads, but not caching writes at all, removed the cache and then set it back up  in as close to the same order as I did on the 3TB drive(which is working perfectly, 89% cache hits overall, mostly on reads, but some writes as well) it still wasn't caching writes, which is absolutely the more important portion for that drive set.

 

So I gave in and removed the cache, deleted that partition and created a new simple partition and added it back to DrivePool as a write only cache drive and tested it on a write and it's definitely write caching properly with DrivePool. So the storage at this time is:

 

3TB Seagate 7200RPM drive, read/write cache in PrimoCache to half of 1TG 860 EVO, then added to a top tier Pool with below drive set.

 

2x8TB WD Elements external drives(kept exteranal on USB3 because it's not hurting performance AT ALL, I don't have space in my current case, and I can retain the original warranty on them) and 1/2 1TB 860 EVO in DrivePool with 860 EVO handling write cache and 100% of data duplicated across both drives. Writes at full SSD speed up to ~500GB and reads are striped across both drives, so surprisingly fast(seen ~250MB/s during sustained reads).

 

If I get a chance on my next day off, I'll crystal disk individual drives and pools of drives. Not sure if it'll be at all consistent on the top tier pool, because the data could go basically anywhere or everywhere, but the two separate sets should be benchmarkable. 

[seagate_surfer]

Thanks for using Seagate for your labs!  I think I'll never find the answer to why we like to complicate life hahaha. PrimoCache is a good option because it is not limited to only one specific technology, I played with it for a bit but did not give it a real use on a daily basis so thanks for sharing...  

[RobFRaschke]

5 hours ago, seagate_surfer said:

Thanks for using Seagate for your labs!  I think I'll never find the answer to why we like to complicate life hahaha. PrimoCache is a good option because it is not limited to only one specific technology, I played with it for a bit but did not give it a real use on a daily basis so thanks for sharing...  

Hey! Of course, I actually love seagate drives, had replaced all my WD drives in the last update, but just got TOO good of a deal on the WD element external drives. I was looking for seagate external drives and price just wasn't quite there. Using an ES.3 currently, my 3+ year old barracuda just bit the dust, which is no small part of what brought about this upgrade.