Backups
We here at atomicorp love our customers, and we are always looking for ways to help them with their risk management needs. This wiki article was written to capture one method for backing up their servers thats free, relatively easy to setup and performs full and incremental backups while allowing easy access to the backups.
Requirements:
- A Linux system to back your system up to (we recommend Centos 5.x)
- Enough storage space to back them up
- rdiff-backup
- ssh
- cron (optional)
How to:
You never know when something might go wrong with your box, like a hardware failure or a disgruntled user deleting things. Using the tool rdiff-backup you can backup your system fully, generate incrementals and you can access the backups on a live version of the backup (without having to restore from tape or some giant blob backup file)
1. Install rdiff-backup on your boxes, you can do that with this command:
yum install rdiff-backup
(And if you have the atomic repo setup you will be able to do this, we put out rpms for this)
2. Setup some REMOTE place to store your backups, dont do this on the box you want to backup.
3. On the backup server, setup a cronjob to do the backup like this:
su - crontab -e
15 21 * * * /usr/local/bin/www_backup.sh
We recommend you run the backup script as root, so you can store all the user and group IDs.
Now create the file "/usr/local/bin/www_backup.sh" with this content:
#!/bin/bash if [ -f /backups/rdiff-backups/prod/www_running ] ; then echo "Backup is already running" exit 1 fi touch /backups/rdiff-backups/prod/www_running rdiff-backup --print-statistics --include-filelist /usr/local/etc/www_exclude backup@YOUR_BOXES_IP_OR_NAME::/ /backups/rdiff-backups/prod/www rdiff-backup --remove-older-than 90B /backups/rdiff-backups/prod/www rm /backups/rdiff-backups/prod/www_running
You also need to create this file:
/usr/local/etc/www_exclude
This tells rdiff-backup what NOT to backup. You dont want to backup somes things, like /proc. The format for the file looks like this (and this is a good file to start with):
- /proc/ - /var/tmp/ - /var/named/chroot/proc - /tmp/ - /var/spool/qmailscan/tmp/ - /var/spool/qmailscan/working/new/ - /var/spool/qmailscan/quarantine/new/ - /sys/ - /var/qmail/greylist/ - /var/qmail/queue/ - /var/spool/qmailscan/qmail-queue.log - /var/spool/qmailscan/qmail-queue.log.1.gz - /var/spool/qmailscan/qmail-queue.log.2.gz - /var/qmail/mailnames/.*/Maildir/.Trash/ - /var/qmail/mailnames/.*/Maildir/.Spam/ - /home/httpd/vhosts/.*/tmp/ - /var/named/run-root/ - /var/ossec/queue/alerts/ar - /var/ossec/queue/alerts/execq - /var/ossec/queue/ossec/queue
Modify this list to your needs. Do not backup /proc.
4. On the backup server create the backup directories
mkdir /backups mkdir /backups/rdiff-backups etc.
5. Create an ssh key on the backup server for root to log into the remote server
ssh-keygen -t rsa -b 4096
Or make the key anyway you like, I prefer RSA 4096 bit keys.
copy the public key from this file:
/root/.ssh/id_rsa.pub
It will look like this:
ssh-rsa ALDSAFKj30hasyuujwheuihqwiueefkjwhefuywoerupo etc.
Copy the entire line, no line breaks, etc. - or copy the file to the server you want to backup.
6. Log into the server you want to backup and create the user backup
useradd backup
Make sure you add them to the list of admin users in ASL if you are using that feature, and make sure you re-run asl to set the policy.
asl -s -f
7. su to the user backup
su - backup
8. Create the ssh directory and add backups public key
mkdir .ssh cd .ssh
copy in the id_rsa.pub file, save it as authorized_keys.
9. Now we're going to edit that file to be extra paranoid about what backup can do - we DONT want backup being able to get a shell or run commands on the box. Don't skip this step if you want to be secure.
Edit authorized_keys and add this in:
command="/bin/nice -n 20 /usr/bin/sudo -u root rdiff-backup --server --restrict-read-only /",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAAB3NzaC blahb blah blah
A normal authorized_keys file looks like this:
ssh-rsa AAAAB3NzaC blahb blah blah
So you'll notice we changed that and added a command="" to the front of the key. SSH can be setup to only allow commands from a specific key - so we can lock the backup account and the key created on the backup server so that it can only run one command. This is a great way to prevent the backup account from being used to do bad things.
Also, doublecheck your permissions on the .ssh directory and authorized_keys files. Make sure they are NOT group or globally writable or readable:
chmod og-rwx -R ~backup/.ssh
10. On the server you want to backup, we need to edit the /etc/sudoers file to allow backup to run the rdiff-backup command as root. We can do that by adding this line to /etc/sudoers:
backup ALL = NOPASSWD: /usr/bin/rdiff-backup
On some boxes you will also need to remove or comment out this line from /etc/sudoers
Defaults requiretty
If you have that line, and dont remove it, ssh command based logins will NOT work.
11. On the backup server lets run the script and save the SSH host key.
/usr/local/bin/www_backup.sh
You should be asked to accept and save the host key for your server. Save it and let the backup script and do its job. If you dont do this, the script will sit there forever waiting for a response. You only have to do this one.
rdiff-backup is like rsync on steroids. It will save full working mirrors of the systems you backup, and it will save incrementals of all the changes so you can roll back. The last line the script will also delete backups older than 90 days, if you want to store less backups change 90 to whatever you want - and if you prefer not to delete anything just remove that line.
