Off side pi backup system

[Janik0000]

Hello,

My Situation:
I want to set up an unraid server as NAS and VM server.
Then I want to have a network drive/folder via SMB/samba where i can put my important stuff.
And then I want this "important" content to back up to an offside Raspberry Pi with something like 2 500gig hdd in a raid-1.
Not required but would be nice live sync of the files, every file that gets deleted goes in a folder called "deleted" on the pi so I can go there if something gets accidentally deleted, every time a file gets updated because it was edited the for example last "3" versions of this file get saved for let's say 2 weeks if I need it for what ever reason.

 

My Problem:
I don't know where to start, this kind of project I never did something like this before.
LTT showed something like this in one of their server videos but didn't explain how it works (that's not acutely a problem)
As it themes right now I'm just not experienced enough for that project I try to find a solution my self but any help would be very appreciated

My Solution:

I used rsync, it's a linux only command (thanks again "electronics wizardy")

type rsync --help when your in your terminal and you can see what all this extansions mean

thats the command i use, if you need a exaple if you have any questions you can ask me:

 

rsync -avzrhu -e  "ssh -i "whre the ssh key is stored"" --delete "source" --exclude lost+found  user@example.domain:"destination"

 

I didn't manage to get this thing with the delted folder up and runnig nor with the versions but i will, try to do it and if i find a solution i will update the post again

Edited April 23, 2020 by Janik0000
solution found

[Electronics Wizardy]

So the pi needs a way to send a recieve data, best is probably a vpn with certs, but something like sshfs is a lot simpler to setup, and probably fine for security.

 

You can use a program like rsync to manage the copying of files, and can do a lot of fancythings with keeping old versions and delete files.

[Janik0000]

24 minutes ago, Electronics Wizardy said:

So the pi needs a way to send a recieve data, best is probably a vpn with certs, but something like sshfs is a lot simpler to setup, and probably fine for security.

 

You can use a program like rsync to manage the copying of files, and can do a lot of fancythings with keeping old versions and delete files.

thanks m8, i will check it out

[Salv8 (sam)]

17 minutes ago, Electronics Wizardy said:

-snip-

yea i'd do this, the USB connectors will be a bottle neck thanks to USB 2.0, if you have a PI 4 you can just connect them up to the 3.0 ports and call it a day.

if you want some expandability (or just faster storage in general if you really need it) the radxa ROCK Pi SATA HAT will alow 2.5" drives to be connected via the PI's GPIO header's.

i can't find any data saying if there are any data speed limitations but it probbly is faster then the 3.0 USB ports.

i would go this route if you have like $50-60 spare and do care about the future of your backup NAS

[Electronics Wizardy]

Just now, Salv8 (sam) said:

yea i'd do this, the USB connectors will be a bottle neck thanks to USB 2.0, if you have a PI 4 you can just connect them up to the 3.0 ports and call it a day.

if you want some expandability (or just faster storage in general if you really need it) the radxa ROCK Pi SATA HAT will alow 2.5" drives to be connected via the PI's GPIO header's.

i can't find any data saying if there are any data speed limitations but it probbly is faster then the 3.0 USB ports.

i would go this route if you have like $50-60 spare and do care about the future of your backup NAS

speed wise, I wouldn't worry too much for a offsite nas, Im gonna guess with isp will be the limit.

[Salv8 (sam)]

Just now, Electronics Wizardy said:

speed wise, I wouldn't worry too much for a offsite nas, Im gonna guess with isp will be the limit.

depends on the data you are transfuring lots of little file's (such as documents and images), it will be fine with what he is doing.

but if he had files that are over 200MB (movies or photoshop PSD's) then the transfer will slow down.

even if his ISP didn't limit it, i wouldn't be suspised if the USB ports were the issue.

my main backup for my movies and tv shows has this issue, when i did my first test on it, i used a bat script i wrote to write lots of little files to it (txt files about 10 KB) and it wrote 10 gb's in about a min, a ep of ducktales (2017), took 3, a 300 mb file took longer then a mutiple files worth 10gb. (this was over 3.0 as well)

hdd's are great for writing lots of small files, they really struggle when the files get bigger.

and since USB slows it down even further, it really can add up.

 

for what i'm storing it's fine, i can wait 10 mins for a 3GB file to be backed up.

it may be an issue for him and my potential solution will ensure that it is mitigated so that he doesn't have to worry about a backup taking too long since the longer a backup takes, the more of a chance that the master copy can fail and he potentially lose his data.

you don't want a partial backup now do ya?

[Janik0000]

50 minutes ago, Electronics Wizardy said:

best is probably a vpn with certs

I think i'll go with an proxy i could run the proxy on my vps.

8 minutes ago, Salv8 (sam) said:

depends on the data you are transfuring lots of little file's (such as documents and images), it will be fine with what he is doing.

but if he had files that are over 200MB (movies or photoshop PSD's) then the transfer will slow down.

even if his ISP didn't limit it, i wouldn't be suspised if the USB ports were the issue.

my main backup for my movies and tv shows has this issue, when i did my first test on it, i used a bat script i wrote to write lots of little files to it (txt files about 10 KB) and it wrote 10 gb's in about a min, a ep of ducktales (2017), took 3, a 300 mb file took longer then a mutiple files worth 10gb. (this was over 3.0 as well)

hdd's are great for writing lots of small files, they really struggle when the files get bigger.

and since USB slows it down even further, it really can add up.

 

for what i'm storing it's fine, i can wait 10 mins for a 3GB file to be backed up.

it may be an issue for him and my potential solution will ensure that it is mitigated so that he doesn't have to worry about a backup taking too long since the longer a backup takes, the more of a chance that the master copy can fail and he potentially lose his data.

you don't want a partial backup now do ya?

I only want backups of my documents, pictures, gamesaves, scrpits and stuff like that so mostly a few mb per file. i baught all my movies and tv shows i use on my plex server on dvd or blu ray so i don't need a second digital copy of them.

25 minutes ago, Electronics Wizardy said:

i would go this route if you have like $50-60 spare and do care about the future of your backup NAS

i'm think i got an raspi 3+ some wehre lying around so i will probably just go with that one becouse i really don't need fast speeds and if something sould hapen to my data the backup location will only be a few km's away so i could easly drive there take the drives and copy the data via usb 3.0 on to my pc if it should take to long to download.

[Electronics Wizardy]

29 minutes ago, Salv8 (sam) said:

hdd's are great for writing lots of small files, they really struggle when the files get bigger.

Thats odd, I have seen that hdd love big files, and really stuggle with small ones, as with big files, its much more sequentical.

 

I haven't found usb to be a issue with hdds, its well over what a hdd can handle, usb2 may limit you, but with pi's your now limited by 100m ethernet aswell.

 

13 minutes ago, Janik0000 said:

i'm think i got an raspi 3+ some wehre lying around so i will probably just go with that one becouse i really don't need fast speeds and if something sould hapen to my data the backup location will only be a few km's away so i could easly drive there take the drives and copy the data via usb 3.0 on to my pc if it should take to long to download.

Yea do the first sync on site, really speeds things up.

 

14 minutes ago, Janik0000 said:

I think i'll go with an proxy i could run the proxy on my vps.

I don't think a proxy is really want you want here. You probaby want a vpn here, and just run it from the home connection, unless you need a vps. 

[Janik0000]

20 minutes ago, Electronics Wizardy said:

but with pi's your now limited by 100m ethernet aswell

and that will be plenty because my upload is 5MB and my download 12MB. And it will rather be limited by the the persons internet connection where i left the pi. but anyway it's 5 am in my country so i think im no going to sleep a while.