How to Use Rsync on Microsoft Windows for Cross-platform Automatic Data Replication

[Windows7ge]

Rsync is a very powerful data replication tool for synchronizing files between two locations. This can be performed locally on one system, locally on the network (LAN), or remotely over the Internet (WAN). As great of a tool as Rsync is, the fully functional version is only available on Linux. There are less functional 3rd party versions for Windows such as Grsync but this guide is going to discuses how it's possible to use Rsync on Windows so that synchronization over the network & Internet are possible. 

 

1. Preparing for and Installing the Linux Terminal

First introduced in Windows Build 1604 with the Anniversary Update and later stabilized in Windows Build 1703 with the Fall Creators Update a feature was included in Windows known as the Windows Subsystem for Linux(WSL). This feature is disabled by default so we need to enable it. Go to the stat menu and search for the Control Panel. From there navigate to Programs > "Turn Windows features on or off". Scroll down the list until you find Windows Subsystem for Linux. Tick the box. Click OK, and you'll be prompted to restart your computer.

 

 

After your computer is done restarting you need to open an elevated command prompt. You can do this by going to Start > Search: CMD > Right Click : Command Prompt > Run as administrator > Yes. Once you're in type:

bash

 

This should prompt you if you'd like to install Ubuntu on Windows. However, if you get this error like I did:

 

Follow the address https://aka.ms/wslstore this will take you to the Microsoft Store and list to you all the available distros to choose from:

 

For this guide we're going to stick with the popular user friendly Ubuntu but if you like you should be able to pick any other flavor you prefer. Click on Ubuntu (or other) and click on Install:

 

You may have to click Launch once the download is done but once it is you should be greeted by a prompt tell you that Ubuntu is installing:

Be patient, this will finish on it's own.

 

Once it's ready you'll be prompted to create a UNIX username & password:

I've just gone with test_sys for the sake of the tutorial. Once this is done you should be greeted by a green prompt that says your-username@your-computername:~$

 

For best reliability & performance we're going to perform a package update. This will updates many components of our terminal which can be achieved by running the two following commands (if prompted for a password use the password you created):

sudo -i
apt-get update && apt-get upgrade

sudo -i makes us root which is necessary to have full access to every directory that we need. Ordinarily we would use sudo followed by our command (sudo apt-get update && apt-get upgrade)  to get root privileges but this is not sufficient. The command will run partially then error. This update will take a few minutes, be patient.

 

2. Setting Up Automatic Login over SSH with Passwordless Public/Private Key Authentication

With updates done, and in order for automatic replication to work (over a network/Internet) we need a way to automatically authenticate with our server. The server can be local(LAN) or remote(WAN) but to perform the initial configuration password authentication needs to be enabled. For this example I'm going to use a local FreeNAS server. What we're going to do right now is create a public/private key pair using the following command:

ssh-keygen

It is going to ask you for a series of inputs including where to save it & to password protect the key. If you have a directory where you want the files you can put that in or just hit enter for the default .ssh directory. For password protection DO NOT PUT ONE IN. Leave the field empty and just hit enter. A passwordless key pair will be generated:

 

Once this is done we need to put the public key on our server. It's possible to copy the file yourself and move it to the server via WebUI (in the case of FreeNAS) but with password authentication enabled we can use a more direct route. The command below will have to be edited to suit your use case but will follow this structure.

ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.247

Be sure to specify the user on the remote server "user@" without this the terminal will attempt to authenticate with your local account and fail.

 

If you haven't established a connection with this server before you'll receive this warning:

If you trust the server say Yes.

 

With this passwordless public/private key authentication is configured and you'll be given the following prompt:

Disable password authentication on the server and attempt to login:

ssh root@192.168.0.247

If done correctly you won't be prompted for a password:

 

Using the passwordless RSA key pair is still a very secure way of authentication. So long as a unauthorized user doesn't get the private key on our Ubuntu Terminal the server is safe.

 

3. Create the Rsync Script

There are many ways to write a script with many different functions but for the sake of this tutorial we're going to keep it very basic. First off we need to open a text editor within the Terminal. Our two major options are Vim & Nano. For this example we're going to use nano as it's the simpler of the two. At the same time we can name our file. I'll call mine backup.script:

nano backup.script

 

At the very base of our script is our primary command:

#!/bin/bash
rsync

 

The first line here is common for scripts as it tells *NIX what type of interpreter we want it to run. The next line is the base of our command "rsync". From here we need to add our switches. Not going over all of them some of the extras we're interested in are:

-P - Combines --progress (Shows you the progress of each file being transferred) & --partial (If a large file transfer is interrupted this keep the remote file intact and resumes the transfer when a connection is reestablished)

-z - Compresses files while they're being transferred which helps speed up the overall transfer of files. Many file formats cannot be compressed though including (7z, avi, bz2, deb, g,z iso, jpeg, jpg, mov, mp3, mp4, ogg, rpm, tbz, tgz, z, zip)

-h - Instead of displaying the transfer speeds in bytes -h will show you it in the converted KB/s, MB/s etc

--stats - Give you a summary of how the transfer went when it's done

--log-file - Saves how the sync went to a log file so you can refer to it later if anything went wrong.

It's important to note that in this environment everything is case sensitive. -p is not the same as -P, -Z is not the same as -z

 

If you want more detail on rsync open the Terminal and run the command:

man rsync

 

With this we can construct our rsync command:

#!/bin/bash
rsync -avzhP -e 'ssh -p 22' --delete  --stats --log-file=/mnt/c/users/test_sys/backup-`date +”%F-%I%p”`.log /mnt/c/somefiles root@192.168.0.247:/mnt/test/

You will have to determine your own directories for your log-file, what local folder you want to copy, & what remote directory you want to place them in but your command should look similar to this. If you save and exit nano (Ctrl + O then Ctrl + X) copy/paste just the rsync line and paste it into your terminal it should run and give you and output similar to this:

 

If you set the log-file to your desktop you should also find it there:

backup-Year-Month-Day-HourAM/PM, this is what the code in the script creates for the file name.

 

Next we need to take our two lines and make it a file Linux can execute. This can be achieved by first checking the files current permissions. The command for this is:

ls -l backup.script

Your output should look like:

 

Right now the permissions are:

Owner: Read/Write

Group: Read/Write

Other: Read/Write

 

We need the Owner to be able to execute this file. Depending on your use case you may want Group or Other to have access to the file but for this example Owner is the only one who needs any access so we're going to use the command:

chmod 700 backup.script

Checking our file again:

Our script has now turned green and the Owner now has execute permissions. We can test the file by running the command:

./backup.script

If we wrote it correctly it will execute and close by itself without issue. We can check the log file to verify that it worked correctly.

 

4. Making the Script Execute by Itself With Windows Task Scheduler

To make our script perform a routine backup on it's own we can use Windows Task Scheduler. This can be found by going to Start > Search: "Task Scheduler". When the Task Scheduler opens click "Action" (near the top left) and from the drop down click on "Create Task...". This will bring up the following menus:

 

Each persons configuration will be different so I can't fully go over how to setup each tab but I can go over the basics:

 

General - Where you'll pick a name for this task, and a description(optional), you can also chose if it will only run when your user is logged in our not.

Triggers - This is where you will chose what will cause the task to be ran. For us "On a schedule" will be our choice. Here you can pick the frequency at which our task will be ran.

Actions - This is where we will setup our script to be executed. I will explain how in a moment.

Conditions - Offers some advanced features such as start delay or if you don't want the task to run if the system is on battery.

Settings - Offers miscellaneous configuration options.

 

Go to the Actions tab and click "New..."

For Action we want to start a program. Under Program/script click "Browse..." on the right. What we're going to do is use CMD.exe to run our script.

CMD.exe will be located at C:/Windows/System32/cmd.exe, click on it and hit Open

We now need to tell CMD what it is we want it to do and we can do that by adding an argument. In the "Add arguments (optional):" field I'm going to use:

-cmd /C wsl /home/test_sys/backup.script

Breaking this down:

-cmd - Tells cmd.exe that we want it to execute a command on it's own when it starts up

/C - This tells cmd.exe to autmatically close after it finishes executing our command. You can use /K to make it stay open.

wsl - Windows Subsystem for Linux. This starts our default Terminal

/home/test_sys/backup.script - This is the absolute path within our Terminal to our script. Even though the Terminal opens in our home directory using the relitive path "./backup.script" will not work. Absolute path is nessasary.

With that you can click OK.

 

Once you're done personalizing your Task you can click OK to create it. If you right-click the Task you can Run it to verify it'll work when it executes according to your schedule.

 

Summary

This is not the only way that we could have accomplished rsync on Windows. There are other options, other ways to setup our script, other ways to run it automatically, but this is quite ideal using readily available services included in Windows. If you're particularly script savvy you can take this to a whole additional level by extending the complexity of our script or running multiple scripts in parallel.

 

If you believe there is anything wrong with this guide. Anything that could be done better, improved, or simplified I'm open to all suggestions and will make appropriate revisions.

 

Hope this helps someone.