Last modified: 2021-09-10 19:39

Introduction to af

Basics

af is a simple file archiver. Every time it is run it checks which files have changed since the last archive volume was created and copies the file into their own directory. (I call these directories volumes). Individual files can be picked from an archive with a simple cp operation and whenever an previous version of all files is needed restore-files (it is stored in /usr/lib/af) can be run to re-create that state.

Individual files can be directly accessed in the archive because af stores complete files (no diffs, no databases). Archive meta-information (e.g. where is the most recent version of file x?) is stored in the plain-text file .archive.tab in each volume directory (again: no database). This file is also used to compute, which files have change.

I wrote af years ago to keep past versions (backups) of source code files. I'm the only one working on my source code so I didn't need collaboration functions and I also don't wanted a full-blown source code control and revision system for any other reason. On the other hand I like to understand how things work and feel safe that I can access backuped files without problems. That made me start working on af which also resulted in moba.sh (mobile archiver).

af (and the related scripts too) is written in gawk and makes heavy use of shell commands because af doesn't copy files or create directories on its own. Instead it creates and executes shell scripts for that. The main sources of documentation are the manpages af.1 and moba.1.

Note: Sometimes I say that af creates a backup of your files. Of course copies on the same harddisk are not real backups and I know that. But creating "real" backups with af is also possible and usually I use a combination of both: daily backups on the same computer and a backup every few days on a network store.

Configuration

af is some kind of file archive or backup program. Therefore the one and only thing you really need to configure is the place where the files are copied (the archive location). You set that parameter with the dir configuration option in the file .archive-files.conf in the directory you want to backup.

dir     /var/backups/pi/markdown
history HISTORY
editor  +vi

This configuration copies files below /var/backups/pi/markdown - just as you would expect. The other two settings are optional:

history fn
defines a file where you record your changes. Each time you are going to create a new archive volume with af create af inserts a timestamp into it, loads the file is into an editor allowing you to describe the changes to the previous version.

editor editor
sets the text editor to use. My personal preference is vi. However, a better default for everyone is nano because it displays the keyboard shortcuts you need to know. So even if you get into the editor by accident you will not be lost.

The + infront the editor
this is not a typo but an additional option. It might happen that you type af create too early and while you are editing the history you notice that you forgot something to add. Therefore, af asks you if you want to continue when you have finished editing the history. Now the + tells af to check the editor's exit-code: if it is non-zero (leave vi with :cq:) af restores the previous history and exits without copying files.

With this configuration I can now check which files changed since I created the last volume.

$ af
M .index.ca
M HISTORY
M gopher.ca
M index.gph
M mdc
N news.md

If you just created your .archive-files.conf all files in the directory and all sub directories (which are included) should be listed.

There are several reasons to check the list of modified files. Here it is that I want to exclude tar and zip files. That's because the directory is a about a gawk script and the tar is made from the source files I'm already archiving. Files are excluded with the omit option

dir     /var/backups/pi/markdown
history HISTORY
editor  +vi
omit	^(out\.)
omit    \.(html|tar\.gz|zip)$

and works like that:

omit regexp
excludes files matching regexp from the archive.

There's also ignore to exclude only a single file by its name:

ignore fn
excludes fn from the archive.

If you feel not sure about af's configuration you can display the most important parameters with af config:

$ af config
dir      /var/backups/pi/markdown
src      .

omit     ^(out(.[0-9])?|cut)$|^(out\.)|\.(html|tar\.gz|zip)$
history  HISTORY

accept-errors rm,rmdir

.workdir /var/backups/pi/markdown/zz-0025-1C68-20210904
.thisdir 0025-20210904
dir
The destination directory.

src
The source directory.

omit
This is the effective regular expression, which is used to exclude files. First, you can see that all omit options are combined into one regexp and second, that af adds ^(out(.[0-9])?|cut)$ to it. (This is my personal preference and the only way to change that for you is to modify the defaultOmitPattern variable in the script). Furthermore, files matching the regular expression (~|$|.tmp$|.swp$) are also excluded.

history
The history file.

accept-errors
af does not copy files on its own. Instead it creates shell commands and passes them to /bin/sh -e. The -e makes sh terminate whenever a command returns an error exit-status.
.workdir
This is the temporary directory af would have used. The third part is af's process id so the name changes every time and the
.thisdir
This would be the next volume's directory.

After checking the configuration I can now continue to create a backup of the files.

Getting information from the archive

My Markdown project is not new so there are already files in the archive.

$ af version
0024

There are 24 version. And since I edited mdc some days ago in a hurry I'm not sure what I changed. So I check that first.

$ af diff mdc
742c742
<               if (token == "<BR>") {
---
>               if (token == "<BR>"  ||  token == "<BR>\n") {
1693c1693
<       version = "1.0.4";
---
>       version = "1.0.5";

Obviously it was a bug fix to recognize
at the end of an input line. I also incremented the version number. That was only a minor change. But perhaps I'm looking where I introduced a new bug and then I want a diff to an older version.

$ af diff 22 mdc
15c15
<               if (x[i] == "exit") {
---
>               if (x[i] == "exit")
...

does just that. af allows to diff against every older version but the file must exist (was changed) in that volume. Here

$af diff 17 mdc
af: no such file version: 17

returns an error because mdc wasn't changed then. So, what mdc versions are available in the archived?

$ af info mdc
mdc                    0016   2021-03-18 19:00:23        36581
mdc                    0018   2021-03-26 09:16:43        36591
mdc                    0019   2021-04-12 20:21:13        37478
mdc                    0020   2021-05-29 11:26:33        39747
mdc                    0022   2021-08-06 17:00:10        39828
mdc                    0023   2021-08-08 09:29:48        40291
mdc                    0024   2021-08-30 21:33:17        39543

The info command lists the volume number, modification date and time and the file's size. (mdc had a different name before version 16.) From that list I could pick my diff targets.

You can query the full pathname of a file's last version

$ af last mdc
/var/backups/pi/markdown/0024-20210830

or all available version

$ af history mdc
/var/backups/pi/markdown/0024-20210830/mdc
/var/backups/pi/markdown/0023-20210808/mdc
/var/backups/pi/markdown/0022-20210806/mdc
...

e.g. to use them from within a script for something. You could use that information to restore an older version, e.g.

$ cp `af last mdc` mdc

to restore the previous version but there is a better way.

$ af cat mdc

prints the mdc's previous version and you can specify any volume number just as for info.

$ af cat 23 mdc >mdc

is a quick way to restore a file.

Summary

So far we looked at these commands:

cat [n] fn
prints a previous or the latest version from fn.

config
shows important configuration parameters.

create
creates a new backup volume.

diff [n] fn
displays the diff between a previous and current version of fn. fn must exist in the directory of volume n.

history fn
lists the full pathnames of all versions of fn.

info fn | ~regexp
lists the available version of fn or all files matching regexp in the archive.

last fn
prints the full path of fn's latest version.

version
displays the version number of the newest (last created) volume.

There are two more to know:

list
lists all changed files, which is also the default if no command if given.

help
display a brief command summary.

Together these are the commands you are using most often I guess.