Sync Program?

[caleb1]

Looking for a good automatic file sync software?

Looking to backup to a network location, and since the total size I would be backing up would be quite large, it would take too long to copy all the files and overwrite.

So I would like to find a software that can "sync" the folders and copy individual files as they are changed, so as not to backup everything at once.

I did an internet search, but there are many options....  Thought I would see if anyone knows of a good one?

 

Thanks.

[dag_dg]

Specifically what you're asking for on per-file changes; glusterfs will handle this though I'd wager your OS is likely Windows. "DFS" is about the closest thing I know of that will do what you want if you have multiple machines.

From my experience dealing with replicating data though in most situations it's better to do periodic syncs rather than real-time syncs even if you have a ridiculous number of small files. For my purposes I generally just use rsync for this task but for windows robocopy has served me well. If you can sync at the filesystem layer using something like ZFS with snapshot replication that can be super flexible, robust, and handy.

[caleb1]

12 hours ago, dag_dg said:

Specifically what you're asking for on per-file changes; glusterfs will handle this though I'd wager your OS is likely Windows. "DFS" is about the closest thing I know of that will do what you want if you have multiple machines.

From my experience dealing with replicating data though in most situations it's better to do periodic syncs rather than real-time syncs even if you have a ridiculous number of small files. For my purposes I generally just use rsync for this task but for windows robocopy has served me well. If you can sync at the filesystem layer using something like ZFS with snapshot replication that can be super flexible, robust, and handy.

Thanks for the input.  It doesn't neccicarally have to be real time, a scheduled sync every few hours or every day would be fine.

But my main thing is I want it to be able to just copy the filed edited since last sync, so that it does not take 18 hrs to copy ALL my files every day.

I'll take a look at rsync and robocopy.

[dag_dg]

For sure. Rsync if you're running Linux systems is amazing and in general handles large amounts of small files well though as with all things it does have its limits based on hardware performance(iops). Robocopy seems to be pretty popular with my Windows-focused colleagues when they don't want to get into setting up something like DFS.

Another project to check out is "unison"

 

https://www.cis.upenn.edu/~bcpierce/unison/

 

I like this one for bidirectional syncs; Any time you are doing a two way sync there's always a chance of conflicts popping up but I've found unison in general does a good job of making it easy to handle things.

[LogicalDrm]

Adding the two software I use. Resilio Sync free which is intended for LAN, but could be configured for over the Internet too. And FreeFileSync which I mainly use for semi-manual backups (it has automated features as well).