rsync is an utility for efficiently transferring and synchronizing files between 2 computers, it can also compare the modification times and sizes of files. It is commonly found on unix OSes. It was written in C as a single threaded application. The rsync algorithm uses a type of delta encoding, and is used for minimizing network usage.
We're going to do this tutorial on a proxmox server with 2 LXC Containers on the same local network:
We login via ssh in both of them:
root@debian-nginx2:~# apt search rsync
rsync/stable,now 3.1.3-6 amd64 [installed,automatic]
fast, versatile, remote (and local) file-copying tool
root@debian-nginx2:~# apt install rsync
root@debian-nginx2:~# which rsync
/usr/bin/rsync
Now let's create a sample directory with a bunch of extremely important data and try to send it to our second debian:
root@debian-nginx2:~# ls
masterdump.sql
root@debian-nginx2:~# mkdir myveryimportantdirectory
root@debian-nginx2:~# echo 'myveryimportant text' > myveryimportantdirectory/myveryimportanttext.txt
root@debian-nginx2:~# ls -l
total 476
-rw-r--r-- 1 root root 480150 Mar 20 10:29 masterdump.sql
drwxr-xr-x 2 root root 4096 Apr 2 07:00 myveryimportantdirectory
root@debian-nginx2:~#
Now that we have that, we will use rsync to send it over to debian1 but to do that we first need to install a ssh connection between the 2 hosts, and to make it more secure we will use SSH keys:
root@debian-nginx1:~# apt search openssh-server
Sorting... Done
Full Text Search... Done
openssh-server/stable,now 1:7.9p1-10+deb10u2 amd64 [installed]
secure shell (SSH) server, for secure access from remote machines
I have installed openssh pre-emptively but if you didn't then install the openssh-server package like so:
apt install openssh-server
rm /etc/ssh/sshd_config
wget https://raw.githubusercontent.com/ech1/serverside/master/ssh/sshd_config -O /etc/ssh/sshd_config
systemctl restart sshd
What my ssh config does is basically to allow the root login, and then refusing password authentifications to force the user to use SSH keys. This means that we will need to create the ssh keys of the slave node, and then we will need to copy the private key from the slave node to the master node.
So first let's create the ssh keys on our debian1 machine, create the ~/.ssh directory if it is not already there:
root@debian-nginx1:~# cd .ssh
-bash: cd: .ssh: No such file or directory
root@debian-nginx1:~# mkdir .ssh
root@debian-nginx1:~# cd .ssh
root@debian-nginx1:~/.ssh# ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/root/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_ed25519.
Your public key has been saved in /root/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:svD8ddV4wm64PWrRPUDr/z3xqk0AfS2zrFi5yccEGIo root@debian-nginx1
The key's randomart image is:
+--[ED25519 256]--+
| . |
| . . + . . |
| E . o + = .|
| . O * |
| . . S =.X.o|
| + o +.%.=.|
| + o B.B +|
| . . ..B.oo|
| . .+o+o=|
+----[SHA256]-----+
root@debian-nginx1:~/.ssh#
root@debian-nginx1:~/.ssh# cat id_ed25519.pub >> authorized_keys
Now that's done, we need to get the private key:
root@debian-nginx1:~/.ssh# cat id_ed25519
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACAR/s+1r8eJq5iHKi+JvCNyd9/vg5evPOC9HqdAKFbOkgAAAJiKlmf6ipZn
+gAAAAtzc2gtZWQyNTUxOQAAACAR/s+1r8eJq5iHKi+JvCNyd9/vg5evPOC9HqdAKFbOkg
AAAEA2aBZsMry0TeChASkAXPax2GEzEH26zcFmvGDaNzPiFRH+z7Wvx4mrmIcqL4m8I3J3
3++Dl6884L0ep0AoVs6SAAAAEnJvb3RAZGViaWFuLW5naW54MQECAw==
-----END OPENSSH PRIVATE KEY-----
into the ssh client's ~/.ssh directory, that way when they will need to ssh into our debian machine, they will automatically use the private ssh key. To send one file to another host on the same network, we can use python3's http module:
root@debian-nginx1:~/.ssh# cat id_ed25519
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACAR/s+1r8eJq5iHKi+JvCNyd9/vg5evPOC9HqdAKFbOkgAAAJiKlmf6ipZn
+gAAAAtzc2gtZWQyNTUxOQAAACAR/s+1r8eJq5iHKi+JvCNyd9/vg5evPOC9HqdAKFbOkg
AAAEA2aBZsMry0TeChASkAXPax2GEzEH26zcFmvGDaNzPiFRH+z7Wvx4mrmIcqL4m8I3J3
3++Dl6884L0ep0AoVs6SAAAAEnJvb3RAZGViaWFuLW5naW54MQECAw==
-----END OPENSSH PRIVATE KEY-----
root@debian-nginx1:~/.ssh# ip a | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 192.168.0.150/24 brd 192.168.0.255 scope global eth0
inet6 fe80::1cb1:50ff:fe1d:27dc/64 scope link
root@debian-nginx1:~/.ssh# python3 -m http.server 8080
root@debian-nginx2:~# cd ~/.ssh
-bash: cd: /root/.ssh: No such file or directory
root@debian-nginx2:~# mkdir ~/.ssh
root@debian-nginx2:~# cd ~/.ssh
root@debian-nginx2:~/.ssh# wget http://192.168.0.150:8080/id_ed25519
--2021-04-02 07:26:37-- http://192.168.0.150:8080/id_ed25519
Connecting to 192.168.0.150:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 411 [application/octet-stream]
Saving to: 'id_ed25519'
id_ed25519 100%[=================================================================================>] 411 --.-KB/s in 0s
2021-04-02 07:26:37 (27.9 MB/s) - 'id_ed25519' saved [411/411]
Now here you need to set the correct permissions onto the client's private ssh key because otherwise you will get this error :
root@debian-nginx2:~/.ssh# ssh root@192.168.0.150
The authenticity of host '192.168.0.150 (192.168.0.150)' can't be established.
ECDSA key fingerprint is SHA256:o7DMwIrXDCyQ3uzVdiUEHkViOXrJhcd47zLBc2U4uDU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.150' (ECDSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/root/.ssh/id_ed25519' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/root/.ssh/id_ed25519": bad permissions
you need to set the 600 permissions on the private ssh key and then you're able to login via ssh:
root@debian-nginx2:~/.ssh# chmod 600 id_ed25519
root@debian-nginx2:~/.ssh# ssh root@192.168.0.150
Linux debian-nginx1 5.4.78-1-pve #1 SMP PVE 5.4.78-1 (Mon, 30 Nov 2020 10:57:47 +0100) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Apr 2 06:57:25 2021 from 192.168.0.99
root@debian-nginx1:~# id
uid=0(root) gid=0(root) groups=0(root)
So now that we are able to login via ssh from debian2 TO debian1 we will be able to use rsync to send files from debian2 TO debian1
We will use rsync like so:
root@debian-nginx2:~# mkdir myveryimportantdirectory
root@debian-nginx2:~# echo 'myveryimportant text' > myveryimportantdirectory/myveryimportanttext.txt
root@debian-nginx2:~# ls -l
total 476
-rw-r--r-- 1 root root 480150 Mar 20 10:29 masterdump.sql
drwxr-xr-x 2 root root 4096 Apr 2 07:42 myveryimportantdirectory
root@debian-nginx2:~# rsync /root/myveryimportantdirectory root@192.168.0.150:/root/
skipping directory myveryimportantdirectory
root@debian-nginx2:~# rsync -a /root/myveryimportantdirectory root@192.168.0.150:/root/
root@debian-nginx2:~#
Now here we basically synchronized the very important directory using the -a flag between our 2 debian hosts (192.168.0.150 and 192.168.0.151) we can see it appear in the first debian host
root@debian-nginx1:~# ls
masterdump.sql
root@debian-nginx1:~# ls -l
total 476
-rw-r--r-- 1 root root 480150 Mar 20 10:29 masterdump.sql
drwxr-xr-x 2 root root 4096 Apr 2 07:42 myveryimportantdirectory
root@debian-nginx1:~# cat myveryimportantdirectory/myveryimportanttext.txt
myveryimportant text
And that's it! we have been able to use rsync to backup our important directory and it's contents into the other debian host.
Now one particular use case i have found for these rsync backups are to automatically run them at a certain time every day. For example on my nextcloud server i do daily backups at 3 AM. To do so i want a zipfile to have the name of the date at which the backup occured, and on top of it i want the backup to be sent over to another host, to do that we will first do the whole process manually after we create a backups directory on the debian1 host (which will recieve the backups that debian2 will send):
root@debian-nginx1:~# ls
masterdump.sql myveryimportantdirectory
root@debian-nginx1:~# mkdir backups
root@debian-nginx1:~# cd backups/
root@debian-nginx1:~/backups# ls -l
total 0
Now in here we want to recieve the backups from debian2, and we want them in a zip file with the date of the backup:
root@debian-nginx2:~# apt install zip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
unzip
The following NEW packages will be installed:
unzip zip
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 406 kB of archives.
After this operation, 1202 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ftp.debian.org/debian buster/main amd64 unzip amd64 6.0-23+deb10u2 [172 kB]
Get:2 http://ftp.debian.org/debian buster/main amd64 zip amd64 3.0-11+b1 [234 kB]
Fetched 406 kB in 0s (1745 kB/s)
Selecting previously unselected package unzip.
(Reading database ... 26618 files and directories currently installed.)
Preparing to unpack .../unzip_6.0-23+deb10u2_amd64.deb ...
Unpacking unzip (6.0-23+deb10u2) ...
Selecting previously unselected package zip.
Preparing to unpack .../zip_3.0-11+b1_amd64.deb ...
Unpacking zip (3.0-11+b1) ...
Setting up unzip (6.0-23+deb10u2) ...
Setting up zip (3.0-11+b1) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for mime-support (3.62) ...
root@debian-nginx2:~# date --iso-8601
2021-04-02
root@debian-nginx2:~# zip -r backup-$(date --iso-8601) /root/myveryimportantdirectory/
adding: root/myveryimportantdirectory/ (stored 0%)
adding: root/myveryimportantdirectory/myveryimportanttext.txt (stored 0%)
root@debian-nginx2:~# ls -lash | grep backup
4.0K -rw-r--r-- 1 root root 465 Apr 2 07:56 backup-2021-04-02.zip
And now we send the backup:
root@debian-nginx2:~# rsync backup-$(date --iso-8601).zip root@192.168.0.150:/root/backups/
and we check if the backup got sent to our debian1 host:
root@debian-nginx1:~# ls -lash backups/
total 12K
4.0K drwxr-xr-x 2 root root 4.0K Apr 2 07:58 .
4.0K drwx------ 5 root root 4.0K Apr 2 07:52 ..
4.0K -rw-r--r-- 1 root root 465 Apr 2 07:58 backup-2021-04-02.zip
And there it is! Now since you don't want to run that script manually everyday you can use crontab to run it for you:
crontab -e
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
0 3 * * * /bin/bash /root/backup.sh
save the cron changes and then create the backup script:
root@debian-nginx2:~# ls
backup-2021-04-02.zip masterdump.sql myveryimportantdirectory
root@debian-nginx2:~# vim backup.sh
#!/bin/bash
#this must run as root !
if [ "$EUID" -ne 0 ]
then
echo 'MUST RUN AS ROOT!'
exit
fi
cd /tmp/
cooldate=$(date --iso-8601)
echo $cooldate
rm backup*.zip
rm backup-$cooldate.zip
zip -r backup-$cooldate.zip /root/myveryimportantdirectory/
rsync backup-$cooldate.zip root@192.168.0.150:/root/backups/
rm backup*.zip
cd -
then try to run it to see if it works:
And that's it! we have been able to create a script that creates zipped backups. The cronjob we setup earlier will run the script as the root user, which is intended and doing so everyday at 3 AM, that way, the backup process happens when there is least amount of work being done on the server.
Donate XMR: 8AUYjhQeG3D5aodJDtqG499N5jXXM71gYKD8LgSsFB9BUV1o7muLv3DXHoydRTK4SZaaUBq4EAUqpZHLrX2VZLH71Jrd9k8
Contact: nihilist@contact.nowhere.moe (PGP)
