Last modified: 2021-09-05 23:10

Verifying af's operations

Archive creations

af operates on you files and you might want to know what af is doing. (I would like to know.) af doesn't copy files on its own - it uses shell commands for that. While this might sound strange but is an excellent way of verifying what is going on and af show displays this. On one of my an average days the output might look like this:

$ af show -q
test -d /var/backups/pi/markdown  ||
  { echo no such directory: /var/backups/pi/markdown >&2; exit 1; }
mkdir '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './.index.ca' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './HISTORY' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './gopher.ca' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './index.gph' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './mdc' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'
cp --preserve=timestamps './news.md' '/var/backups/pi/markdown/zz-0025-7AF2-20210903'

The show command would usually display the modified files first but the -q options turns this off. The other half of show's output are the commands af would execute to create a new volume with af create. There are only two operations missing:

  1. Renaming the temporary file (zz-0025-7AF2-20210903 in this example), and
  2. writing the updated file status to .archive.tab in the volume's directory.

These things are only done if the command list (which is run with sh -e) exits with a zero exit-code.

Archive contents

Whenever af creates a new volume is also creates a directory for it. The directories accumulate over time in the archive's space:

$ ls -l /var/backups/pi/markdown | tail -5
drwxr-xr-x 2 pi pi 4096 May 29 11:41 0020-20210529
drwxr-xr-x 2 pi pi 4096 Jun 26 12:06 0021-20210626
drwxr-xr-x 2 pi pi 4096 Aug  6 17:04 0022-20210806
drwxr-xr-x 2 pi pi 4096 Aug  8 22:07 0023-20210808
drwxr-xr-x 4 pi pi 4096 Aug 30 21:53 0024-20210830

Each directory receives only the modified files - just as the show command shows. A volume directories might then like e.g.:

$ ls -l /var/backups/pi/markdown/0022-20210806/
total 48
-rw-r--r-- 1 pi pi  4636 Aug  6 17:04 HISTORY
-rwxr-xr-x 1 pi pi 39828 Aug  6 17:00 mdc

with all the unchanged files in other directories.

There are two more files in the volume directories .archive.log and .archive.tab. The first is just a logfile that holds the modification status of all files. The important file is the other. .archive.tab holds the information which file of the volume is stored where. The head of it looks like

$ head -5 0022-20210806/.archive.tab 
.archive-files.conf	f	-rw-r--r--	pi	pi	172	2021-03-13,00:10:09	0012-20210313/.archive-files.conf
HISTORY	f	-rw-r--r--	pi	pi	4636	2021-08-06,17:04:01	0022-20210806/HISTORY
Todo	f	-rw-r--r--	pi	pi	1080	2021-04-11,19:20:25	0019-20210412/Todo
debian	d	drwxr-xr-x	pi	pi	4096	2021-05-29,11:40:19	.
debian/arch	f	-rw-r--r--	pi	pi	4	2021-03-18,19:02:11	0016-20210318/debian/arch

and as you can guess from that the tab-separated fields hold the following information:

  1. the filename,
  2. a flag indicating the line is for a file (f) or directory (d),
  3. the standard file permissions,
  4. file user,
  5. file group,
  6. size in bytes,
  7. last-modified date,
  8. the file's path relative to the archive's base directory.

Restoring files on your own

Equipped with the information about the .archive.tab fields it is easy to draft how to restore all or some of the files from a particular volume. You get the archived files you need to copy back e.g. like this:

$ awk 'BEGIN { FS = "\t"; } $2 == "f" { print  $NF }' 0022-20210806/.archive.tab | head -5
0012-20210313/.archive-files.conf
0022-20210806/HISTORY
0019-20210412/Todo
0016-20210318/debian/arch
0016-20210318/debian/changes.in

If you cut away the volume directory the remainder of the line is the position to where you would need to copy the file. Directories must be created, e.g.

$ awk 'BEGIN { FS = "\t"; } $2 == "d" { print }' 0022-20210806/.archive.tab | xargs mkdir

That's important even if af comes with restore-files, which does exactly this. Just imagine in 20 years from now you find an old backup disk full of af archives ... and the software is gone. That doesn't matter because

check-archive

One of the tools that come with af is check-archive. It is installed in /usr/lib/af so you will not find it in your path. The script takes two arguments:

  1. The path to an .archive.tab file (an index file), and
  2. an optional directory.

check-archive uses the first argument to verify if all files in the archive match the information from the index. This should not report any error. Otherwise the files off the index is broken and both is bad.

With the second parameter, check-archive compares the files in the directory against the files in the archives. Two kind of things are reported:

If you run check-archive immediately after you have created an archive you should get file differences but most probably some messages regarding missing files.