1 filesystem path, 2 storage objects & locations | Like a reverse NTFS Junction??

[JazzMac251]

Here's a new one for me...

tldr;

"Hey, computer. See this folder called 'C:\Parent'? You write a file here, also write it to 'D:\Child' as well. DO NOT write it to one and then copy it to the other. This is slow and stupid. Tell both drives to write the file simultaneously."

 

The Issue:

I'm a part time video editor. I have a Dropbox folder share set up with one of my regular clients, and that folder lives on a RAID10 array of HDDs which basically acts as a cold storage solution. I have a 1TB NVMe SSD that houses all my active projects. Whenever I get to work, I end up having to manually move the files from the sync'd Dropbox folder on the RAID HDDs over to the SDD. They send me nearly 3/4 of a TB of footage every week, so this takes a lot of time. If I don't remember to take care of it earlier in the week, just the file management alone can contribute significantly to time pressure issues. I'm trying to automate this process, but there are some interesting issues to deal with.

 

=============================
 

Attempted Solutions:

My first thought was to simply symlink the shared folder such that Dropbox thinks it's located within normal "Dropbox land," but in reality the folder lives on my SDD. But, this leaves me manipulating the client's master files directly from our shared folder. That's bad for two reasons; 1) Dropbox is going to try to sync every change I make in real-time across everyone's systems (annoying, but also huge potential for sync problems), 2) I don't want to mess with the master copies of the footage. Something would get lost, almost certainly. 

My second thought was to use some kind of local-machine folder syncing software (e.g. FreeFileSync), but that has issues too. There really isn't a way for the software to tell when files in the shared Dropbox folder have finished downloading. So, it ends up trying to sync the watched folders across the SDD and the HDD array in real time as the files are still downloading to the HDD array.  The high-throughput read AND write requests MASSIVELY tank both the Dropbox download AND the software folder sync at the same time. So, that's not great. 

My third idea was to look for an NTFS filesystem solution. My hope was that the data could be taken from system memory and written to both drive destinations simultaneously by the filesystem itself as the files download. This would avoid the bottleneck associated with the previous software solution above. Symlinks/Junctions are two linked PATHS to one set OBJECTS on disk - and I want the reverse of this - so I thought there might be a solution in that family. But, so far so bad. My searches have all come up empty.

My fourth idea (Jesus...) was to create a new mirrored volume across part of the HDD array and the NVMe SDD just for this use case. But, after thinking about it further, that won't work either! I don't actually want one group of files mirrored to two locations, I want one pipeline through which incoming data will be written to two otherwise independent locations. Creating a mirrored volume would not give me that pipeline. But, what it WOULD do is create a solution with all the same issues as using symlink...except this time I have a local copy I don't need. Great.

My last idea was to post here because I'm completely out of other ideas.  Any suggestions would be much appreciated! 

[Electronics Wizardy]

11 minutes ago, JazzMac251 said:

So, my third idea was to look for a filesystem solution. My hope was that the data could be taken from system memory and written to both drive destinations simultaneously by the filesystem itself as the files download - kind of like RAID1 but for a single folder. This would avoid the bottleneck associated with the previous software solution above. I thought there would be something in the symlink/junction family, but my searches have come up empty so far. 

Yea this doesn't exist on windows. Raid is done on a block level, not a file level, so a whole filesystem has to be on one level of raid.

 

11 minutes ago, JazzMac251 said:

My fourth idea (Jesus...) was to create a new mirrored volume across part of HDD array and the NVMe SDD just for this use case. But, that won't work either! I don't actually want one group of files mirrored in two locations, I want one pipeline through which incoming data will be written to two otherwise independent locations. This solution would have all the same problems as my first idea, but the files would be on two physical locations. That's not helpful

You can probably do with with the syncing programs, and have it go only way, so the new footage is on your ssd, but changes don't get copied back to dropbox.

 

Then edit the copy thats on your ssd, and copy the finished files over when your done.

[Pandur]

You can mount drives (and raid volumes) as a folder instead of drive letter in Windows. This is just a partial fix at best of course. But it does allow for a raided folder on a different drive.

[JazzMac251]

43 minutes ago, Electronics Wizardy said:

You can probably do with with the syncing programs, and have it go only way, so the new footage is on your ssd, but changes don't get copied back to dropbox.

 

Then edit the copy thats on your ssd, and copy the finished files over when your done.

The issue is that the syncing program will try to copy the files from the HDDs over to the SDDs as they are being downloaded, and this slows both processes down to a crawl.

19 minutes ago, Pandur said:

You can mount drives (and raid volumes) as a folder instead of drive letter in Windows. This is just a partial fix at best of course. But it does allow for a raided folder on a different drive.

Interesting! I hadn't thought of that. Can you elaborate a little bit more on how that helps? I think there's something I'm not seeing.

[Electronics Wizardy]

9 minutes ago, JazzMac251 said:

The issue is that the syncing program will try to copy the files from the HDDs over to the SDDs as they are being downloaded, and this slows both processes down to a crawl.

You cn have it so that it won't sync until its done downloading. Either wait for the lock to be removed, or just wait for the file not to be touched. Shouldn't be too complex of a script

 

[Pandur]

2 minutes ago, JazzMac251 said:

Interesting! I hadn't thought of that. Can you elaborate a little bit more on how that helps? I think there's something I'm not seeing.

You have a complex problem honestly. I was mostly commenting with regards to "can I raid a folder". And you can make your C:\parent folder a separate raid 1 array. That still leave the whole download and copy simultaneously problem. You either need a raid 1 that is fast enough to work on directly, meaning we are probably into the external thunderbolt raid 1 nvme territory. Or you'd need to download the files to it and then put it on your nvme and back again. It is a bit of catch 22 honestly.

 

Maybe look into having a NAS that can automatically synchronise the Dropbox files locally. You still have to copy the files to and from your nvme to do the actual work, but that would happen locally. The NAS should stay up to date without your interaction the rest of the time.

[JazzMac251]

1 hour ago, Electronics Wizardy said:

You cn have it so that it won't sync until its done downloading. Either wait for the lock to be removed, or just wait for the file not to be touched. Shouldn't be too complex of a script

 

I may have to go down that road. I'm very surprised the NTFS filesystem can't handle this functionality natively. It's basically just the inverse of an NTFS Junction. 

I had tried to go the software route in the paste. But, I had a very hard time telling the software what property to look for that would indicate whether the download was still in progress. What is this "lock" of which you speak?