Rsync incremental backup script

**vwgtiturbo** · 05-24-2006, 10:01 PM

Hello, currently I am using rsync in a shell script to do incremental backups of a couple of systems. I know, it isn't glamorous, and can be done in much better ways, but I am trying to keep it simple, and not have to force myself to learn to program in the next... year, just to keep up. Anyways...
I am using a fairly common script online, and I modified it a bit, just to satisfy my anal-retentiveness, and make it a bit easier to decipher. The script is as follows:

#! /bin/bash
#
################################################## #################
# This is a script to run my backups. Put together from sources all over the 'net, it will, #
# in theory, wake up the remote server, perform an rsync-style incremental backup, then shut #
# the the server down. This script is a modified version of one found in an article of Linux #
# Journal in Jan 06 titled 'Build a Home Terabyte Backup System'. I am not sure who the ## author was, but their script was a modified version of one created by rsync creator Andrew #
# Tridgell. #
################################################## #################

###########
# VARIABLES #

# Login name
LOGIN=ron

# Backup server name
BUSRVR=backups

# Backup server MAC address
BUMAC=xx:xx:xx:xx:xx

# Directory to be backed up
BUSRC=/home/ron/test/

# Directory on server to back up to
BUDEST=/backups/ron/rsync.test

# This is the backup subdirectory, dictated by date and time
BUSUBDIR=`date +%A`

# Log file location
LOGLOCATION=/home/ron/Desktop

# Options to pass to rsync
OPTS="--delete -vv --modify-window=2 --force --ignore-errors --backup --backup-dir=$BUDEST/ -az --stats --partial-dir=RSYNC.PARTIAL --progress -e ssh"

###########
# Commands #
###########

# Dump output to log
date > $LOGLOCATION/backuplog.$BUSUBDIR.txt

# Wake up the remote server
sudo /usr/local/bin/ether-wake $BUMAC

# Wait an amount of time to allow the server to boot and get ready for data transfer
sleep 180

# This section deletes the last week's incremental backups
[ -d /tmp/emptydir ] || mkdir /tmp/emptydir
rsync --delete -a /tmp/emptydir
$LOGIN@$BUSRVR:$BUDEST/$BUSUBDIR/
rmdir /tmp/emptydir

# Actual file transfer
rsync $OPTS $BUSRC $LOGIN@$BUSRVR:$BUDEST/current >> $LOGLOCATION/backuplog.$BUSUBDIR.txt

Can someone please explain to me how the Bold/Underlined part works? When I run the script, it does run and creates a subdirectory for 'current' and copies all source files over as it should. But when it runs the increment, it will create a another directory for the day of the week, then run a full backup to it, instead of an increment. I am just not seeing how '-d /tmp... ' is a command and can actually do something. I am a rookie to scripting though (and Linux in general, having only dealt with it for about a year). Any help would be greatly appreciated.

**vwgtiturbo** · 05-24-2006, 10:33 PM

Wow, now I am really going crazy... Let's say you have the destination directory for your backups, called "Backups". Then inside of that, you have "Current". Well, what this script is doing now, for some odd reason, is doing a full backup to "Current", then if I change a file and run the script again, it puts the updated file into the "Current" directory, then moves the original file from "Current" into "Backups". It isn't creating subdirectories for the increments, it just dumps them into the --backup-dir (observing the folder structure of the source).

**vwgtiturbo** · 06-07-2006, 05:37 PM

Nevermind. Everything works great now. I still don't understand the [-d... part, and I have seen it in other scripts (not backup related). I can't see how a set of [ ] with a "-d" or any other letter does something. Is there some command associated with it, or?

**redhead** · 06-07-2006, 07:34 PM

Code:

[ -d "something" ]

says if "something" evaluates to beeing a directory, (hence d, f for file, x for executable, etc.), then do what comes next
If it is made as:

Code:

[ -d "something"] || command

then the command will only be executed if it's not a directory (|| logical OR) or perhaps something like

Code:

[ ! -d "something" ]

this is "NOT equal to directory", like with many other programming languages.

// edit, hmm.. didn't we use to have a "shell programming" PET ???

**vwgtiturbo** · 06-07-2006, 08:05 PM

Thanks a lot for that. I couldn't figure it out... I have MILLIONS of shell scripting references but could NOT find anything in them. I guess if I looked through each of them start to finish, I may have been able to find it, but since I am lost at about page 20, there wouldn't be much sense in that. So, basically, this part of the script checks to see if /tmp/emptydir exists. If not, it creates /tmp/emptydir? Okie dokie... I am still foggy on how this:

[ -d /tmp/emptydir ] || mkdir /tmp/emptydir
rsync --delete -a /tmp/emptydir
$LOGIN@$BUSRVR:$BUDEST/$BUSUBDIR/
rmdir /tmp/emptydir

removes the last week's incremental backup though. Ok, it checks to see if /tmp/emptydir exists. If not, it creates /tmp/emptydir.

It then syncronizes the emptydir to the backup destination directory with todays date (which sort of makes sense, as, if you have a partial backup from earlier in the day, and it was cut off for some reason, then this would clear out the destination to make way for a new backup to start (and hopefully complete). In a way though, this is odd, because at this point in the script, the destination directory hasn't been created yet...

**vwgtiturbo** · 06-07-2006, 08:12 PM

I got it! Okay, the script I originally got this from was used for a 7-day rotating incremental backup. The 'date' format used in that script was 'date %A' which only displays the day of the week. So after a week of backups, in theory, you would have a folder 'current' for the current state, and one for each day of the week (which takes care of my confusion about the folder not existing yet when the [-d... section takes place). Now that the destination has 8 folders, it WOULD find a folder labeled as *day, and it would be able to delete the contents from it (rsyncing the emptydir and the *day directory that already exists).

My problem is that I was really anal-retentive about naming the directory, in that I used day, month, and time. So, this incremental will not work for me unless I modifiy the script a bit... I was starting to think that this is a bad way to do this, as if your computer date was off for some reason, and you ran a backup, you could possibly delete a lot of data. However, after thinking about it for a bit, I realized that your current stuff is contained in the 'current' directory, so it should be safe if that were to happen...
Thanks a lot for the help. I have been befuddled by the [xxxx] for DAYS.

**redhead** · 06-07-2006, 08:19 PM

rsync --delete -a /tmp/emptydir
$LOGIN@$BUSRVR:$BUDEST/$BUSUBDIR/

--delete deletes files that dont exists on sender ie /tmp/emty dir, which we know should be empty.
Which means every file located in $BUDEST/$BUSUBDIR on $BUSRVR will be deleted.

rsync $OPTS $BUSRC $LOGIN@$BUSRVR:$BUDEST/current

Places $BUSRC in $BUDEST/current on $BUSRVR, which we now know isnt holding some old files, since it was just deleted by the comparison to an empty directory.