Last modified: 2021-09-06 14:39

af Additional Scripts

restore-files

restore-files is a script that restore any volume from an af archive. It doesn't have a manpage but prints usage information if it extract-volumes_is run without parameters. The only mandatory parameter is the .archive.tab file of the volume you want to restore. The simple example

# To get shorter command lines
$ VOL=/var/backups/pi/markdown/0023-20210808
$ /usr/lib/af/restore-files $VOL/.archive.tab

creates the shell commands to copy all files from volume 23 to the current directory but doesn't execute them. This is perhaps the most important point: restore-files does not copy or initiate copying files. To actually copy the files back you have to feed the command's output to sh:

$ /usr/lib/af/restore-files $VOL/.archive.tab | sh

First of all that means that you can safely restore-files to check it's operation, especially that

  1. the files are copied to the right location,
  2. you have selected the files you want, and
  3. the copy operation is what you want.

You can try different options, check the restore operation and copy the files when the combination is right.

restore-files allows you to specify regular expressions to select the files you want to get from the archive. To get only the Markdown documentation files you would run

$ /usr/lib/af/restore-files $VOL/.archive.tab '\.md'

You may add as much regular expressions as you want (or run the script repeatedly). A regular expression excludes files if it starts with a -.

The -d dir option changes the restore location to another directory than the current. If the directory contains already files two options may be helpful:

Both options are implemented using cp command line options. Another borrowed cp functionality is to preserve the UNIX file permissions. restore-files uses normally cp --preserve=timestamps,mode ... to copy files back. That works for the file permissions (--preserve=mode) only if the archive filesystem (1) supports UNIX file permissions and (2) the permissions were preserved when the volume was created (which is what af tries). If the two prerequisites are not met (e.g. on a CIFS file server) you can set the -p option. With that option set restore-files create cp commands first and then chmod_s to set the correct permissions.

The output from restore-files copy multiple files with one command line. This uses less command invocation but makes it more difficult to parse. Use the -s option to get one command per file to e.g. grep the commands for some particular files.

Understanding the combination of all the possible options is perhaps not simple, especially when you use restore-files only on rare occasions. Therefore remember to play around with restore-files' settings until you have the operations you want, then pass them to sh.

extract-volumes

Imagine you have lots of backup volumes on a remote disk. Now you want to have some of the latest versions on your computer. This is where you use extract-volumes.

extract-volumes [options] base selector ...

It expects command line parameters to select the archive versions you want and copies all files that belong to them into the current directory. Before going into the details not that

extract-volumes needs two mandatory parameter on its command line. First the archive's base directory base followed by one or more arguments selecting the volume(s) you want to extract. The selector arguments have some variations.

from[-to]
selects volumes by their number. The number may be useful to you but it's a quick way to pick volumes by the date in their directory name.

numberdhwv
selects volumes by their relative creation date. num is just an integer that which is multiplied by its unit: d for days, h for hours and w for weeks. Use v to select a particular number of volumes.

If you put multiple selectors on the command line, extract-archives merges them. E.g. the combination of 30d 3v 10 selects

As usual, you can check the output from extract-volumes before executing it. Set the -s option to check which volumes would be extracted. Other options are:

-d dir
store volumes in dir, not the current directory.

-o fn
extract the volumes to an archive. Support archive format are cpio, tgz (for gzip'ed tar), zip and shar (for shell archive) and are detected from fn's file extension. cpio archives may be gzipped too by using .cpio.gz or .cgz.

-r
remove all files from the destination which do not belong to the extracted volumes. This affects files that are part of previously extracted volumes and files that are not related the archive at all. Use this option for regular volume extractions, e.g. by cron.

-rr fn
copies files that are deleted to archive fn first.

Example - Part 1

Here is an example to show how to extact volumes and how it looks.

# Create a temporary directory and go there
$ mkdir -p ~/tmp/unzip
$ cd ~/tmp/unzip

# Cut the command lines
$ ARC=/var/backups/pi/markdown
$ BIN=/usr/lib/af

# Check the latest volume number
$ ls -1 $ARC | tail -1
0024-20210830

# Prepare ...
$ $BIN/extract-volumes -s $ARC 30d 3v 10
0010
0022
0023
0024

# ... and extract the volumes (run without `| sh` first).
$ $BIN/extract-volumes $ARC 30d 3v 10 | sh

# Take a look at the directory.
$ ls
0004-20210309  0009-20210311  0015-20210317  0020-20210529  0023-20210808
0005-20210310  0010-20210311  0016-20210318  0021-20210626  0024-20210830
0008-20210310  0012-20210313  0019-20210412  0022-20210806

There's e.g. the directory 0004-20210309 because a later version includes files from that so that shouldn't be a surprise.

check-archive is a script that helps you veryfing that the volumes match your source files. It is described together with other possibilities to verify af's operations. We use it here to check a random volume sample if all files were extracted.

$ $BIN/check-archive ./0023-20210808/.archive.tab

** Verifying './0023-20210808/.archive.tab'
++ version numbers match: 0023
++ Verifying files exist in archive, 27 files.
== 0 errors

Everything looks ok.

Example - Part 2

There is one important truth about backups:

You want the restore, not the backup.

and this is what we are going to do in the second part: restore files from the extracted volumes and verify the result.

# Are we still in the right location?
$ pwd
~/tmp/unzip

# Create a test directory, inspect the commands ...
$ mkdir test
$ $BIN/restore-files -d test 0023-20210808/.archive.tab
...

# ... and restore (`-x` to show the commands)
$ $BIN/restore-files -d test 0023-20210808/.archive.tab | sh -x
...

The contents of test should now be the original volume 23. We can use check-archive for some confirmation.

$ $BIN/check-archive ./0023-20210808/.archive.tab test

** Comparing 'test' and './0023-20210808/.archive.tab'
++ version numbers match: 0023
++ Comparing directory and index.
debian/release.all: file status differs:
  -rw-r--r--	pi	pi	2
  -rw-r--r--	root	root	2
mdc_1.0.0-2_all.deb: file status differs:
  -rw-r--r--	pi	pi	11332
  -rw-r--r--	root	root	11332
mdc_1.0.2-1_all.deb: file status differs:
  -rw-r--r--	pi	pi	11744
  -rw-r--r--	root	root	11744
++ Verifying files exist in archive, 27 files.
== 3 errors

The error messages are quite normal: the original files are owned by root because Debian package creation uses sudo. The other look good but notice that check-archive does not compare the files. (This might be added in a later version.)

Example - Part 3

Now imagine there's a new version from markdown and we decide to always keep only the last two as local copies.

# Still there?
$ pwd
~/tmp/unzip

# What are the last two versions?
$ $BIN/extract-volumes -s $ARC 2v
0024
0025

# Check the commands and notice that not only the old
# now unused versions are going to be deleted but also
# the test directory.
$ $BIN/extract-volumes -r $ARC 2v
...

# Update the local archive.
$ $BIN/extract-volumes -r $ARC 2v | sh -x
...

# List the remaining directories.
$ ls
0015-20210317  0019-20210412  0021-20210626  0024-20210830
0016-20210318  0020-20210529  0023-20210808  0025-20210906

A quick check shows

cat 0024-20210830/.archive.tab | awk '{ print $NF }' | sort

that the archive versions 24 and 25 really don't use anything from version or earlier. Of course you can run check-archive for more tests.