heilbIT
Systemanalyse & PC-Hilfe Eberbach

Backup

This is a backup system which allows for full and differential backups using rsync and a rsync server.

The software is provided as is without warranty of any kind.

Current Version is v2.4, released 07.08.2006

Features
Prerequisites
Todo's
Changes
Vital information
Recovery procedure
Filenames containing spaces
Installation
Download
Mailing list
Support forum

Features

full or differential backup may be configured depending on backup history.
differential backup: only those files are backuped which have changed since last backup.
full backup: backup all files.
the tree structure of directories to backup is saved.
interactive backup with progress feedback.
backup may be configured to run completely automatic.
files containing sensitive security information, e.g. /etc/shadow, can be encrypted before they are backuped.
files or directories can be excluded from backup.
backup over stunnel (encrypted tunnel to rsync server)
no restrictions but the available space on the backup/rsync server
restore backuped files to arbitrary states in history, e.g. do not restore the file from the latest backup but from the backup which run 2 weeks ago.

Prerequisites

a debian linux like system providing csh/tcsh
rsync client (http://rsync.samba.org)
gpg (or compatible) (http://www.gnupg.org) only for encryption
a rsync server providing space
perl
perl libraries Net::FTP and Getopt::Long are provided with this package and installed locally in your backup installation directory Note:
The Net::FTP library is not used anymore and may be removed in future versions.

Todo's

(in order of priority)

better compatibility with other linux/unix versions
archive of files: user or root initiated archive of directories and files.

Changes

step v2.3 to v2.4
- bug: wrong parsing of time intervall in month results in always perform complete backup
- bug: rsync output not filtered correctly results in empty directories are backuped without need
step v2.2 to v2.3
- time intervall for complete backups configurable in months
- bug: rsync stdout version 2.6.6 not filtered correctly
- bin/restore.csh can now be called in parallel
- bin/restore.csh option --dir has changed meaning
- bin/restore.csh new option --targetdir
step v2.1 to v2.2
- the usage of the ftp protocol is not necessary anymore - all functions are performed using rsync
- more reliable test for a working rsync server
- skipping files/directories if backup is failing for them without crashing of complete backup

step v2.0 to v2.1
- bug: filenames containing spaces not fully supported - fixed
- bug: message backup successfully if connect to backup server failed - fixed

step v1.5 to v2.0
- the files modes, owners and groups are backuped and restored
- running backup.csh from command line prints progress status
- log info now printed to log file (backup.log), not to stdout
- restore.csh can now be run without any user input (option --yes)
- bug: calculation of backup space may result in negative numbers - fixed.

step v1.4 to v1.5
- critical bug: files which are referenced via symbolic link and are in the encryption list do not reach the backup - fixed

step v1.3 to v1.4
- restore script added

step v1.2 to v1.3
- the format of the encryption and exclusion file list has changed

step v1.1 to v1.2
- check for available space on the backup server and some automatic reactions are implemented if not enough space is available.
- the possibility to backup arbitrary directory trees, e.g. /home/user1, is now implemented.
- fixed error in last complete backup date

step v1.1 to v1.2
step v1.0 to v1.1
- missing new line at the end of backup.csh output added
- a bug in backup.csh results in error "badly formed number" and no backup is done anymore when only full backups exist - fixed

Vital information

Backup is controlled by the skript backup.csh

Each backup results in a list of the backuped data. These lists are stored in the subdirectory of the installation directory named db/ . The very first backup must be a complete backup. For complete backups, which are performed by the backup_all.csh script, a file which looks like backup_20040522_0.list is created in db/ and contains a list of all backuped directories. Each directory in this list is completely backuped with all content. The general format of this file is:
backup_YYYYMMDD_0.list
where YYYYMMDD is the date of the complete backup.

If a complete backup exists, it is possible to perform differential backups, where only those files are backuped which changed state and are not already in the backup in this new state. This is done by backup_inc.csh. For each differential backup a file is created in db/ which looks like backup_20040522_20040523023100.list and which contains a list of each file which was backuped. The general format of this file is:
backup_YYYYMMDD_YYYYMMDDHHMMSS.list
where YYYYMMDD is the date of the newest complete backup and YYYYMMDDHHMMSS ist the date and time of the differential backup.

For each of the files in db/ there is a subfolder on the rsync backup server, which is equally named but without the .list extension, e.g. for the file backup_20040522_0.list there is a directory /backup_20040522_0 on the backup server. This directory contains the complete backup data of this backup with identical tree structure as it was on the backuped client. For example if /usr/local was backuped and is in the list backup_20040522_0.list there is a directory on the backup server called /backup_20040522_0/usr/local which contains all data of /usr/local. The same applies for the differential backup list files.

The complete available space must be specified in /etc/backup.conf. An approximation of the used backup space is calculated for each backup. From all available complete backups an approximation of the needed space for a new complete backup is calculated. This approximation is tried to be always free on the backup server by removing old backups before starting a new differential or complete backup. This feature can be switched off in /etc/backup.conf but if space on the backup server decreases the propability of a failed backup and of uncomplete backup data increases.

You may specify encryption and exclusion file lists. The default location of these files are /backup/encrypt_file.list and /backup/encrypt_file.list.exclude. When editing these files just consider the following points:

encryption file list (default encrypt_file.list):
- all files are excluded automatically and need not to be listed in the exclusion list
- full path name must be specified, e.g. /etc/passwd
exclusion file list (default encrypt_file.list.exclude)
- full path name can be specified
- symbolic links can be used
- filenames without path can be used (e.g. core, .rnd, ...)
- directories can be specified (e.g. /tmp)

Recovery procedure

Use script restore.csh to restore data from the backup rsync server to your local file system.

The documentation of restore.csh is available by calling the script with option --help:

restore.csh --help

The basic concept of restore.csh is, that you first use restore.csh in view mode using option --view and when you are satisfied with the list of files you get listed you change the option --view to option --restore without changing any other option you tried in view mode. With option --restore you will then restore all files you have listed before in view mode.

Examples:

1) File system crash, just bring system back to the latest known state:
    restore.csh --view | less
    restore.csh --restore

2) The directory /usr/local was deleted by accident, restore to the latest known state:
    restore.csh --view --include /usr/local | less
    restore.csh --restore --include /usr/local

Filenames containing spaces

The following syntax should be used to specify filenames including spaces on the command line, e.g. in the restore command line option --include and --exclude:
restore.csh --view --include \'filename with spaces\'
That means put the filename into \'.

To specify filenames including spaces in the list files, e.g. in encrypt_file.list or in directories use the syntax
"filename with spaces"
That means put the filename into ".

Installation

Follow the step-by-step installation procedure described in file INSTALL which is included in the tar archive.

Download

This software is public domain.
You may use it as you wish.
The author provides this software as is without warranty of any kind and accepts no responsibility for any loss or damage incurred due to any inaccuracy or technical problems using this software.
The using of this software is at your own risk.

Download file: backup_v2.4b.tar.gz (64147 Bytes)
Feedback is most appreciated!

Old versions:
backup_v2.3.tar.gz (63729 Bytes)
backup_v2.2.tar.gz (63030 Bytes)
backup_v2.1.tar.gz (58384 Bytes)
backup_v2.0.tar.gz (57181 Bytes)
backup_v1.5.tar.gz (53330 Bytes)
backup_v1.4b.tar.gz (53185 Bytes)
backup_v1.3.tar.gz (47856 Bytes)
backup_v1.2.tar.gz (47193 Bytes)
backup_v1.1.tar.gz (9448 Bytes)
backup_v1.0.tar.gz (9305 Bytes)

Mailing list

A mailing list is available for discussion and announcements:
backup-rsync list
Up to now this is a very low volume list mainly used for notification of new versions.