A modern backup idea for your Unix system

Post date

27 Aug 2011

Filed under

Backups are just like flossing your teeth: you should do it often, but at the end, you only do it during some weeks after you are into deep trouble.

What is the best solution for things you never like to do? The one that is as fast and effortless as possible, ideally fully automated. Because of that I usually suggest people to go for a USB powered disk. Don't get one with a power supply, here the comfort factor is essential. Any minor nuisance can trigger that famous "I will do it later...", so make yourself a favor: make your task as dumb as possible.

Now it is time to consider how to get your data there. Forget about tar, zip and all those compression tools. Compression takes time and these days you get plenty of space with any external disk. Go for a file mirror based tool such as rsync. Only if you go later for a two-way based synchronization go for unison).

Most backup posts I have seen are really classic, with the incremental, rotation and compress fluff, even the ones that mention rsync. What many do not mention is the magic "--link-dest" option. What is so magic about it? Well, it is better seen than shown.

I want to backup my Documents folder (the final "/" matters):

$ rsync -av --link-dest=/full/path/Documents/ Documents/ Documents.1

This will copy the full folder into Documents.1, but if you try to get their sizes you will see something strange:

$ du -h --depth=1  (Linux)
$ du -hd1          (Mac OS X)
110M    ./Documents
  0B    ./Documents.1
110M    .

Did anything went wrong? Well, go and see inside that empty folder

$ du -hd0 Documents.1
    110M    .

Now it is full? What happened? Try something else:

$ dd if=/dev/zero of=Documents/file_5MB bs=1024 count=5120
$ rsync -av --link-dest=/full/path/Documents/ Documents/ Documents.2
$ rm Documents/file_5MB

Here you created a new file with 5MB, synced and deleted it from your Documents. How does it look like now?

$  du -hd1
     110M    ./Documents
       0B    ./Documents.1
       5M    ./Documents.2
     115M    .

OK, now I owe you an explanation. The rsync command with the --link-dest parameter is making a duplicate of the Documents folder with hard links. Every file inside of Documents.1 is just a reference, so if you list the size of every folder, the command du will not count the size of the same file in different locations.

Inside every folder it will count the real size, but outside it will just give you the size of the first folder.

Later we added a 5MB file, synced and deleted it. So when listing everything only this extra difference was counted within Documents.2. Unix file system allocation is based on a reference counting algorithm. Hard links increase the reference count and only when you delete all the references the file is really deleted.

The real magic here is that with every sync you get a full backup at the cost of an incremental one! Every folder looks like a full copy, but only the real differences are what take space.

Now I leave up to you the exercise of finding the right command to save your backups!

The Digital Lumberjack

Hacking Code When Life Goes Digital