Search the Community

This was my previous post, still haven't done nothing about it and i thought of this idea just now. Ik i can use ddrescue but i really want to preserve the HDD as is- rsync is dumb in this regard caz it cant ignore bad sectors , then wipe and repair the HDD with Bad sectors.

i drop kicked my pc, and my /home drive has bad sectors (has been flagged to fail soon) now. i rsync it but in a span of 14 hours, it managed to copy 270 gigs (got 6.7gb/1tbh) and ofc it spends about 20 mins trying to copy 1kb of file. on disks i see like 2002 bad sectors. i don't know what to do. Dual boot windows-Linux/deb got separate drives for both OS- one is a Toshiba(/home) but oddly enough it reported those bad sectors like after 4 days pior to the kick, its old tbh. the other is a WD blue 3 years of use of the shelf. also very new to Linux but am comfortable with terminal and vim. also Data recovery does not exist where i live, or the the whole damn country.

Rsync is a very powerful data replication tool for synchronizing files between two locations. This can be performed locally on one system, locally on the network (LAN), or remotely over the Internet (WAN). As great of a tool as Rsync is, the fully functional version is only available on Linux. There are less functional 3rd party versions for Windows such as Grsync but this guide is going to discuses how it's possible to use Rsync on Windows so that synchronization over the network & Internet are possible. 1. Preparing for and Installing the Linux Terminal First introduced in Windows Build 1604 with the Anniversary Update and later stabilized in Windows Build 1703 with the Fall Creators Update a feature was included in Windows known as the Windows Subsystem for Linux(WSL). This feature is disabled by default so we need to enable it. Go to the stat menu and search for the Control Panel. From there navigate to Programs > "Turn Windows features on or off". Scroll down the list until you find Windows Subsystem for Linux. Tick the box. Click OK, and you'll be prompted to restart your computer. After your computer is done restarting you need to open an elevated command prompt. You can do this by going to Start > Search: CMD > Right Click : Command Prompt > Run as administrator > Yes. Once you're in type: bash This should prompt you if you'd like to install Ubuntu on Windows. However, if you get this error like I did: Follow the address https://aka.ms/wslstore this will take you to the Microsoft Store and list to you all the available distros to choose from: For this guide we're going to stick with the popular user friendly Ubuntu but if you like you should be able to pick any other flavor you prefer. Click on Ubuntu (or other) and click on Install: You may have to click Launch once the download is done but once it is you should be greeted by a prompt tell you that Ubuntu is installing: Be patient, this will finish on it's own. Once it's ready you'll be prompted to create a UNIX username & password: I've just gone with test_sys for the sake of the tutorial. Once this is done you should be greeted by a green prompt that says your-username@your-computername:~$ For best reliability & performance we're going to perform a package update. This will updates many components of our terminal which can be achieved by running the two following commands (if prompted for a password use the password you created): sudo -i apt-get update && apt-get upgrade sudo -i makes us root which is necessary to have full access to every directory that we need. Ordinarily we would use sudo followed by our command (sudo apt-get update && apt-get upgrade) to get root privileges but this is not sufficient. The command will run partially then error. This update will take a few minutes, be patient. 2. Setting Up Automatic Login over SSH with Passwordless Public/Private Key Authentication With updates done, and in order for automatic replication to work (over a network/Internet) we need a way to automatically authenticate with our server. The server can be local(LAN) or remote(WAN) but to perform the initial configuration password authentication needs to be enabled. For this example I'm going to use a local FreeNAS server. What we're going to do right now is create a public/private key pair using the following command: ssh-keygen It is going to ask you for a series of inputs including where to save it & to password protect the key. If you have a directory where you want the files you can put that in or just hit enter for the default .ssh directory. For password protection DO NOT PUT ONE IN. Leave the field empty and just hit enter. A passwordless key pair will be generated: Once this is done we need to put the public key on our server. It's possible to copy the file yourself and move it to the server via WebUI (in the case of FreeNAS) but with password authentication enabled we can use a more direct route. The command below will have to be edited to suit your use case but will follow this structure. ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.247 Be sure to specify the user on the remote server "user@" without this the terminal will attempt to authenticate with your local account and fail. If you haven't established a connection with this server before you'll receive this warning: If you trust the server say Yes. With this passwordless public/private key authentication is configured and you'll be given the following prompt: Disable password authentication on the server and attempt to login: ssh root@192.168.0.247 If done correctly you won't be prompted for a password: Using the passwordless RSA key pair is still a very secure way of authentication. So long as a unauthorized user doesn't get the private key on our Ubuntu Terminal the server is safe. 3. Create the Rsync Script There are many ways to write a script with many different functions but for the sake of this tutorial we're going to keep it very basic. First off we need to open a text editor within the Terminal. Our two major options are Vim & Nano. For this example we're going to use nano as it's the simpler of the two. At the same time we can name our file. I'll call mine backup.script: nano backup.script At the very base of our script is our primary command: #!/bin/bash rsync The first line here is common for scripts as it tells *NIX what type of interpreter we want it to run. The next line is the base of our command "rsync". From here we need to add our switches. Not going over all of them some of the extras we're interested in are: -P - Combines --progress (Shows you the progress of each file being transferred) & --partial (If a large file transfer is interrupted this keep the remote file intact and resumes the transfer when a connection is reestablished) -z - Compresses files while they're being transferred which helps speed up the overall transfer of files. Many file formats cannot be compressed though including (7z, avi, bz2, deb, g,z iso, jpeg, jpg, mov, mp3, mp4, ogg, rpm, tbz, tgz, z, zip) -h - Instead of displaying the transfer speeds in bytes -h will show you it in the converted KB/s, MB/s etc --stats - Give you a summary of how the transfer went when it's done --log-file - Saves how the sync went to a log file so you can refer to it later if anything went wrong. It's important to note that in this environment everything is case sensitive. -p is not the same as -P, -Z is not the same as -z If you want more detail on rsync open the Terminal and run the command: man rsync With this we can construct our rsync command: #!/bin/bash rsync -avzhP -e 'ssh -p 22' --delete --stats --log-file=/mnt/c/users/test_sys/backup-`date +”%F-%I%p”`.log /mnt/c/somefiles root@192.168.0.247:/mnt/test/ You will have to determine your own directories for your log-file, what local folder you want to copy, & what remote directory you want to place them in but your command should look similar to this. If you save and exit nano (Ctrl + O then Ctrl + X) copy/paste just the rsync line and paste it into your terminal it should run and give you and output similar to this: If you set the log-file to your desktop you should also find it there: backup-Year-Month-Day-HourAM/PM, this is what the code in the script creates for the file name. Next we need to take our two lines and make it a file Linux can execute. This can be achieved by first checking the files current permissions. The command for this is: ls -l backup.script Your output should look like: Right now the permissions are: Owner: Read/Write Group: Read/Write Other: Read/Write We need the Owner to be able to execute this file. Depending on your use case you may want Group or Other to have access to the file but for this example Owner is the only one who needs any access so we're going to use the command: chmod 700 backup.script Checking our file again: Our script has now turned green and the Owner now has execute permissions. We can test the file by running the command: ./backup.script If we wrote it correctly it will execute and close by itself without issue. We can check the log file to verify that it worked correctly. 4. Making the Script Execute by Itself With Windows Task Scheduler To make our script perform a routine backup on it's own we can use Windows Task Scheduler. This can be found by going to Start > Search: "Task Scheduler". When the Task Scheduler opens click "Action" (near the top left) and from the drop down click on "Create Task...". This will bring up the following menus: Each persons configuration will be different so I can't fully go over how to setup each tab but I can go over the basics: General - Where you'll pick a name for this task, and a description(optional), you can also chose if it will only run when your user is logged in our not. Triggers - This is where you will chose what will cause the task to be ran. For us "On a schedule" will be our choice. Here you can pick the frequency at which our task will be ran. Actions - This is where we will setup our script to be executed. I will explain how in a moment. Conditions - Offers some advanced features such as start delay or if you don't want the task to run if the system is on battery. Settings - Offers miscellaneous configuration options. Go to the Actions tab and click "New..." For Action we want to start a program. Under Program/script click "Browse..." on the right. What we're going to do is use CMD.exe to run our script. CMD.exe will be located at C:/Windows/System32/cmd.exe, click on it and hit Open We now need to tell CMD what it is we want it to do and we can do that by adding an argument. In the "Add arguments (optional):" field I'm going to use: -cmd /C wsl /home/test_sys/backup.script Breaking this down: -cmd - Tells cmd.exe that we want it to execute a command on it's own when it starts up /C - This tells cmd.exe to autmatically close after it finishes executing our command. You can use /K to make it stay open. wsl - Windows Subsystem for Linux. This starts our default Terminal /home/test_sys/backup.script - This is the absolute path within our Terminal to our script. Even though the Terminal opens in our home directory using the relitive path "./backup.script" will not work. Absolute path is nessasary. With that you can click OK. Once you're done personalizing your Task you can click OK to create it. If you right-click the Task you can Run it to verify it'll work when it executes according to your schedule. Summary This is not the only way that we could have accomplished rsync on Windows. There are other options, other ways to setup our script, other ways to run it automatically, but this is quite ideal using readily available services included in Windows. If you're particularly script savvy you can take this to a whole additional level by extending the complexity of our script or running multiple scripts in parallel. If you believe there is anything wrong with this guide. Anything that could be done better, improved, or simplified I'm open to all suggestions and will make appropriate revisions. Hope this helps someone.

Hello, I’m new to the world of home servers so I’m looking for someone to check my homework. I was inspired by the Brian’s Econonas 2019 build. Dear Mods: if this is better suited for the New Builds and Planning forum, please move it there. 1. Budget & Location I’m located in Australia. My parts choices are based on what I’ve been able to find at my local store and I prefer to use commonly available parts. Budget of AU$700 (EDIT: ~ US$470) without drives. Open to small upgrades over time. 2. Use Case NAS for home file storage servicing 4 home users. Plex (if possible, eventually transcoding 3 x 1080p streams with ease). Resilio Sync (very important to me). SabNZB. Probably going with unRAID. (unlikely Plan B = Windows Server). Will be deployed with 5 X 8TB (2 of which are parity) drives and a 500 Gb cache SSD. Will eventually upgrade to 8 X 8 TB (2 of which are parity) drives and a 500 Gb mirrored cache + SSD for Plex metadata. Hoping it will last me 5-10 years. I want to experiment with running a Windows VM on this (among other tinkering), to potentially consolidate this NAS with my mom’s PC. But I haven’t looked too deeply into this aspect yet. Also, Folding@Home? This will sit in our home office with very tight space – the case I’ve selected is the biggest it can be. Silence is appreciated, but direct HDD cooling is paramount during hot weather (easily 40 deg C or 104 deg F). Dust resistance appreciated. 3. Parts List Case: Antec P101 Silent ($135) CPU: Ryzen 3200G ($145) Mobo: ASUS B450 Prime ($149) RAM: Corsair Vengeance LPX 8Gb ($69) PSU: Corsair RM650x ($169) TOTAL (before unRAID license, SATA cables and PCIE to SATA card): $667. P.S. Any recommendations for a good quality PCIE to SATA card are appreciated. Looking forward to your feedback. Thanks guys.

So I'm updating 3 Synology NAS units and I'm wondering. Should I stop the rsync process before updating DSM? Thanks

Hello all. So here is my situation. I have a Odroid XU4 (Ubuntu OS) board running as my NAS drive at home and I use it as automatic backup of my PC which works great. I use automatic robocopy script from my PC. I want to make a similar setup at my parents house to backup my backup out side of my house. I heard that rsync is one of the best options for that. Can someone help me with command line which will copy my backup location to my off site back up same way robocopy does? Thanks!

A few weeks ago, I had a catastrophic failure with my Linux laptop that was caused by none other than my own stupid self, by issuing a command functionally similar to a sudo rm -rf --no-preserve-root, deleting everything under / seemingly unrecoverably (I tried a couple different backup utilities, and none of them worked). Anyway, I've moved on and reinstalled my OS, and I've come to terms with the (albeit relatively insignificant) data loss I've suffered. This time, however, I want to use a robust backup solution (something I completely lacked before). I've decided I want my script to rsync my home folder (pretty much all I really need) to my NAS and do incremental daily backups, but full weekly backups (so folders would be stored 2018-%w where %w is the week number in the year). That part is relatively straightforward but I have a few more goals that I'd like to accomplish with this script, and I'm not sure how to go about them: 1) Transfer files using an SSH destination to my server without having to log in or enter the password on my private key (because I might be away from my computer when the script runs) 2) Run the script while the laptop is sleeping/suspended (because chances are the script will run late at night when I'm sleeping, and I don't want to have to wake up the laptop just to run it). Is there a cronjob that can override suspension and proceed anyway? or an alternative that will accomplish the same goal of always having a backup for that day? Even if I don't use the computer that day? 3) Keep track of files over, say, 10gb that are unchanged and only back them up once (Since I'm only making a new folder every week, I'm referring to the previous week's folder) to save disk space.

Hey guys, Okay, so I have a slight issue. I'm busy backing up data from one HP Prolient Microserver to another using Rsync over SSh. Problem is, the internet connection is pretty dismal (usually less than 1mbps upload in the office), so I thought it would be best to run the initial backup in the office and then just incremenal backups over the web, as the amount of data that changes on a daily basis isn't too much. Now here is where I hit my first issue: The Rsync does fine in the office and over the web, however, it seems to not see the files I copied in the office, so it wants to copy the entire bulk of files again, which is going to take ages. Am I just doing something wrong, or is it just the way Rsync is designed? If I run the backup in the office over the local network again, it just copies the changed files, so its definitely working in that regard. It just seems the change in the address of the rsync command causes the change to occur. The one server is running FreeNAS and the secondary server to which the data is copied runs Ubuntu Server.

TL;DR at the bottom. Basically, what the title says. I want to use rsync to have a remote back of a server. I know I can use ssh to connect the two, but the only way to do so over the internet that I've found is VPN's and port forwarding. I'd prefer neither because VPNs aren't free and port forwarding isn't the best in terms of security. From what I can tell, the answer is "you can't then", but I wanted to make sure, so I thought I'd ask here. If that is the answer, note that I already have access to a website/domain that has a static IP address that is accessible from the internet as it is. Is there a way I could use that? For example, the web server hosting the actual website is right beside the server I want to be using rsync with. They are on the same LAN. If I can access one in such a way, I'd imagine I could the other with SSH. However, I'm a bit of a noob at WAN networking. All I know is that part of what allows people to access the web server and use the website like they can is that our Domain Controller redirects them there. Is there a way to have SSH redirected to the correct rsync server? The two servers are currently on a LAN and replicating correctly as it is using SSH. I'm just needing help moving that to allow off-site backups. I know. Lots of specific questions. I'm just kind of lost in the WAN world, so yeah. Any help is appreciated. TL;DR: I require help to rsync using SSH over WAN/Internet without using VPN or port forwarding when it appears that the connection part is already set up in a way with our web server for access to the WAN/Internet, and I might be able to use something similar for this.