rsync backup script - Help

[panther420]

 A few weeks ago, I had a catastrophic failure with my Linux laptop that was caused by none other than my own stupid self, by issuing a command functionally similar to a sudo rm -rf --no-preserve-root, deleting everything under / seemingly unrecoverably (I tried a couple different backup utilities, and none of them worked). Anyway, I've moved on and reinstalled my OS, and I've come to terms with the (albeit relatively insignificant) data loss I've suffered. This time, however, I want to use a robust backup solution (something I completely lacked before). I've decided I want my script to rsync my home folder (pretty much all I really need) to my NAS and do incremental daily backups, but full weekly backups (so folders would be stored 2018-%w where %w is the week number in the year). That part is relatively straightforward but I have a few more goals that I'd like to accomplish with this script, and I'm not sure how to go about them:

 

1) Transfer files using an SSH destination to my server without having to log in or enter the password on my private key (because I might be away from my computer when the script runs)

2) Run the script while the laptop is sleeping/suspended (because chances are the script will run late at night when I'm sleeping, and I don't want to have to wake up the laptop just to run it). Is there a cronjob that can override suspension and proceed anyway? or an alternative that will accomplish the same goal of always having a backup for that day? Even if I don't use the computer that day?

3) Keep track of files over, say, 10gb that are unchanged and only back them up once (Since I'm only making a new folder every week, I'm referring to the previous week's folder) to save disk space.

[jde3]

Well.... to answer your question. edit your crontab (crontab -e) and add.

 

@daily rsync -az /home/user/ /mnt/backup

 

Each user has their own individual crontab you can do this on whatever user you want that preformed on. (note the / on the end of /home/user/)

 

but.......

 

I'd recommend maybe duplicity (deja dup in Gnome) as the rsync command is just a straight copy.. it isn't an incremental.

[panther420]

1 minute ago, jde3 said:

Well.... to answer your question. edit your crontab (crontab -e) and add.

@daily rsync -az /home/user /mnt/backup

 

Each user has their own individual crontab you can do this on whatever user you want that preformed on.

 

but.......

 

I'd recommend maybe duplicity (deja dup in Gnome) as the rsync command is just a straight copy.. it isn't an incremental.

Will a cronjob in a crontab run even when the system is sleeping?

[jde3]

Just now, panther420 said:

Will a cronjob in a crontab run even when the system is sleeping?

no. anacron can take care of that but.. again this is probably not what you want to do for a proper backup.

[panther420]

Just now, jde3 said:

no. anacron can take care of that but.. again this is probably not what you want to do for a proper backup.

Alright, not to be rude but I already said I figured out how to do a basic script such as the one you suggested where it just copies the files. You haven't really answered any of my three questions, which was what I was asking about. I'm also not looking for advice unless you believe dejadup or duplicity will accomplish all three of the things I wanted up there. Besides, I prefer sticking to CLI tools when I can. And rsync is sufficient for my needs, maybe I have the definition of an incremental backup wrong but all I need is a script that will copy my home folder to a directory of the week and update it daily, until the next week runs around. And that doesn't need to be debated.

[jde3]

Uh.. well umm..

 

1. rsync can use ssh so.. yeah just look into man rsync for the options for it, it works basically like scp.

3. rsync is a delta copy so it will never copy a unmodified file.

 

So you're asking kind of the wrong questions.

[panther420]

6 minutes ago, jde3 said:

Uh.. well umm..

 

1. rsync can use ssh so.. yeah just look into man rsync for the options for it.

3. rsync is a delta copy so it will never copy a unmodified file.

 

So your asking kind of the wrong questions.

Again, you haven't answered my questions. 

1. I'm well aware of the fact that it can use ssh, wouldn't it need me to enter a password or the password on my key? How would I do that if I'm not at the computer?

3. This one is turning out to be kind of a pain in the ass and your answer also doesn't tell me anything I didn't already know so I think I'm just going to exclude all files over a certain size limit upfront and copy big files manually (I won't have a lot of them on this laptop.)

[jde3]

I'm not going to tell you how to do it wrong. Your implementation you want to create is awful. There is no reason to contently re-create the same data. You are trying to write a script to create a incremental backup solution from scratch when many already exist.. anyone ever tell you not to re-invent the wheel?  ssh does not need a password on a private key.

 

I gave you my advice on the problem.. use duplicity, it's a cli very similar to rsync.

 

@daily duplicity /home/user scp://user@host/backup

[panther420]

10 hours ago, jde3 said:

I'm not going to tell you how to do it wrong. Your implementation you want to create is awful. There is no reason to contently re-create the same data. You are trying to write a script to create a incremental backup solution from scratch when many already exist.. anyone ever tell you not to re-invent the wheel?  ssh does not need a password on a private key.

 

I gave you my advice on the problem.. use duplicity, it's a cli very similar to rsync.

 

@daily duplicity /home/user scp://user@host/backup

Hey, I've been toying around with duplicity a little bit and I think you're right, it would be an easier solution that will manage everything for me. I'll probably just make one directory that does incremental backups forever (Is that a problem? Should you create full backups every so often? Is a week too much?). Sorry for being so harsh earlier.

[jde3]

5 minutes ago, panther420 said:

Hey, I've been toying around with duplicity a little bit and I think you're right, it would be an easier solution that will manage everything for me. I'll probably just make one directory that does incremental backups forever (Is that a problem? Should you create full backups every so often? Is a week too much?). Sorry for being so harsh earlier.

No, no problem. It does it's own versioning. When you restore or list a restore you can choose a point in time. (if it has it based on it's config)

[panther420]

11 hours ago, jde3 said:

No, no problem. It does it's own versioning. When you restore or list a restore you can choose a point in time. (if it has it based on it's config)

Cool. I'll double check the config and just roll with it. Thanks!