Hi all. I have been using a script that I wrote myself for over 2 years and have had *zero* problems with it. It has never failed once. It has the following nice features:
* Automatically rotates backups each night and keeps a fixed number that you specify (for example, 14 gives you 14 days worth of backups at any time, so you can go back 2 weeks if necessary)
* Uses rsync's options for using hard links so that files that are unchanged are not duplicated on the hard drive but instead are hard linked. This means that each backup looks like a complete backup, but only really has changed files. You can look at any backup directory and see a complete filesystem. This makes restoring the backup easier and less error-prone. This is similar in spirit to the "incremental" backups that rdiff-backup is making, but it makes every backup "look" complete except that there is only 1 copy of each file, not one copy per backup (unless the file has changed, in which case each change is a new file in that backup).
I am attaching the posting that I made about this over 2 years ago in this thread:
http://www.linode.com/forums/viewtopic.php?t=666
mcowger wrote:
Im all setup now....would love to see that rsync script...
Happy to oblige. A few caveats:
* This isn't "productized" in any way. You have to know what you are doing and you may need to modify this to work the way you want.
* This script is run from the system which is to be backed up to and it contacts the system which is to be backed up.
* In order to run this script unattended, you will need to set up keys so that the system which is doing the backup can ssh into the system which is being backed up, and as root no less. This could be a security concern because it means that having root on the backed up to machine is as good as having root on the backed up machine.
* I run this script on both systems, so that they back each other up. It is important to "exclude" the backups directory of one system from being backed up on the other, otherwise you will have "recursive" backups that will grow exponentially.
* Usage is as follows:
rsync_backup.sh [remote_root] [local_root] [excludes_file]
[remote_root] is the directory from the remote system to be backed up. I use '/' to back up everything.
[local_root] is the local directory under which the backups will be written. They are rotated so that if you are keeping 14 backups, after 14 days you will have backup.01, backup.02, ..., backup.14, with backup.01 being the oldest.
[excludes_file] is a file on the local system which lists, with one file or directory per line, the files and directories from the remote system which should not be backed up.
You can set the following environment variable also:
NUM_BACKUPS is the number of backups to keep (I back up once per day and use 14 to have two weeks' worth of backups at all times).
This script uses hard linking to eliminate redundancy of files, so each incremental backup only actually uses disk space for those files which have changed since the last backup, although each backup.XX directory will look like a complete backup.
Here is what my crontab entry for doing the backups looks like:
Code:
# At 5:13 every morning, back up mitya.ischo.com
13 5 * * * /data/backup/rsync_backup.sh mitya.ischo.com / /data/backup /data/backup/excludes.txt
And here is what the /data/backup/excludes.txt file looks like:
Code:
/proc
/data/backup/eva.ischo.com
/data/rsync/modules
/data/share
/data/tmp
Have fun!
NOTE: I'm having some problems with the way that this forums system formats the code lines; it's wrapping some even though it doesn't show them as wrapped in the "preview". It looks like it's only some of the comment lines though that have alot of dashes in them. Be careful when/if you copy the script text into a file to fix that before trying to run it.Code:
#!/bin/bash
echo "rsync_backup.sh started at " `date` "."
# ------------------------------- Constants ----------------------------------
NUM_BACKUPS=3
# -------------------------------- Commands ----------------------------------
MKDIR=/bin/mkdir
MV=/bin/mv
RM=/bin/rm
RSYNC=/usr/bin/rsync
SEQ=/usr/bin/seq
# --------------------------------- Paths ------------------------------------
BACKUP_ROOT="$3"
BACKUP_SYSTEM=$1
EXCLUDES_FILE="$4"
EXCLUDES_ARG=""
REMOTE_ROOT="$2"
TMP_OUT=$BACKUP_ROOT/rsync.out
TMP_TOUCH=$BACKUP_ROOT/rsync.touch
# --------------------------- Check Requirements -----------------------------
if [ -z "$BACKUP_ROOT" ]; then
echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
exit -1;
fi
if [ -z "$BACKUP_SYSTEM" ]; then
echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
exit -2;
fi
if [ -z "$REMOTE_ROOT" ]; then
echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
exit -3;
fi
BACKUP_DIR="$BACKUP_ROOT/$BACKUP_SYSTEM"
echo "Back up local root: $BACKUP_DIR"
if [ \! -d "$BACKUP_DIR" ]; then
echo "Making directory: $BACKUP_DIR"
$MKDIR -p "$BACKUP_DIR"
fi
# Figure out which is the newest backup
NEWEST_EXISTING=0
for i in `$SEQ 1 $NUM_BACKUPS`; do
if [ $i -lt 10 ]; then
if [ -d "$BACKUP_DIR/backup.0$i" ]; then
NEWEST_EXISTING=$i
fi
else
if [ -d "$BACKUP_DIR/backup.$i" ]; then
NEWEST_EXISTING=$i
fi
fi
done
if [ $NEWEST_EXISTING -gt 0 ]; then
if [ $NEWEST_EXISTING -lt 10 ]; then
BACKUP_PREV="$BACKUP_DIR/backup.0$NEWEST_EXISTING"
else
BACKUP_PREV="$BACKUP_DIR/backup.$NEWEST_EXISTING"
fi
echo "Newest backup: $BACKUP_PREV"
else
echo "No previous backups"
fi
NEW_NUM=$[NEWEST_EXISTING + 1]
if [ $NEW_NUM -lt 10 ]; then
BACKUP_DEST="$BACKUP_DIR/backup.0$NEW_NUM"
else
BACKUP_DEST="$BACKUP_DIR/backup.$NEW_NUM"
fi
echo "New backup dir: $BACKUP_DEST"
# ------------ Compose rsync args ------------
# Copy all files recursively including all file attributes, verbosely,
# and use compression over the wire, also delete any deleted files
RSYNC_ARGS="-avz --delete"
# Use ssh to the remote system
RSYNC_ARGS="$RSYNC_ARGS -e ssh"
# Exclude file, if present
if [ -n "$EXCLUDES_FILE" ]; then
RSYNC_ARGS="$RSYNC_ARGS --exclude-from=$EXCLUDES_FILE"
fi
# Link dest, if we already have a previous rsync
if [ $NEWEST_EXISTING -gt 0 ]; then
RSYNC_ARGS="$RSYNC_ARGS --link-dest=$BACKUP_PREV/"
fi
# Source location
RSYNC_ARGS="$RSYNC_ARGS root@$BACKUP_SYSTEM:$REMOTE_ROOT/"
# Destination
RSYNC_ARGS="$RSYNC_ARGS $BACKUP_DEST/"
# Do the rsync
$RM -f $TMP_OUT
echo "rsync command: $RSYNC $RSYNC_ARGS"
$RSYNC $RSYNC_ARGS > $TMP_OUT
# Touch the rsync directory to set the backup time
touch "$BACKUP_DEST"
tail -2 $TMP_OUT
# --------------------------------- Rotate -----------------------------------
# This function is unfortunately necessary because Linux is messing with the
# modification times of the directories on mv
function correct_mv() {
$RM -f $TMP_TOUCH;
touch -r $1 $TMP_TOUCH;
$MV $1 $2;
touch -r $TMP_TOUCH $2;
$RM -f $TMP_TOUCH;
}
if [ $NEW_NUM -gt $NUM_BACKUPS ]; then
# Remove number 1
$RM -rf "$BACKUP_DIR/backup.01"
# Renumber the others
for i in `seq 2 $NEW_NUM`; do
if [ $i -lt 10 ]; then
FROM=0$i
else
FROM=$i
fi
j=$[i - 1]
if [ $j -lt 10 ]; then
TO=0$j
else
TO=$j
fi
correct_mv "$BACKUP_DIR/backup.$FROM" "$BACKUP_DIR/backup.$TO"
done
fi
echo "rsync_backup.sh finished at " `date` "."
[/b][/url]
FAQ
Search
Members
Register
Login [ Anonymous ]
