Dobrica Pavlinušić's Weblog / Blog

If you are system administrator this will sound familiar: you have to quickly fix something, and you know that you should document it somewhere (or keep backup) but it's so much work. You could install one of existing source control management tools on each box, but they usually come with huge dependencies, and having all files in central location would be so useful to co-relate configuration changes. To add insult to injury, existing SCMs don't do good job in tracking just few files spread across file-system.

So, what would be perfect tool for keeping remote files in central git repository look like?

• no dependency on non-standard tools on clients allowing easy deployment
• track individual files and ignore rest
• central repository, one directory per hostname

I tried to solve this problem several times, writing wrappers around subversion to handle sparse checkouts and installing subversion and ssh authentication all over the place. But, all this should be simpler... Like this:

1. add new client to track:

   dpavlin@klin:~/klin/bak-git$ ./bak-git-server.pl ~/backup/ 10.60.0.92 --install brr
   install on brr
   # lot of output stripped
   

   This will do several steps:
   • create git repository in ~/backup/ if it doesn't exist already
   • install root ssh authentication to brr using ssh-copy-id
   • install bak shell helper which uses netcat to connect back to 10.60.0.92
   • install rsync on client and use it as root over ssh to sync files
2. Now we can login into brr and start tracking our files:

   dpavlin@brr:~$ bak add /etc/cron.d/tun0 
   dpavlin@brr:~$ bak add /etc/network/interfaces
   dpavlin@brr:~$ bak commit
   dpavlin@brr:~$ bak log
   commit df09dc5e19ef1d47311d701b4c63f0859b0b81c1
   Author: Dobrica Pavlinusic 
   Date:   Thu Feb 18 19:04:21 2010 +0100
   
       brr [commit] /home/dpavlin/
   
    create mode 100644 brr/etc/cron.d/tun0
    create mode 100644 brr/etc/network/interfaces
   

3. change some configuration and review changes

   dpavlin@brr:~$ bak diff
   diff --git a/brr/etc/network/interfaces b/brr/etc/network/interfaces
   index 806c08e..c52c646 100644
   --- a/brr/etc/network/interfaces
   +++ b/brr/etc/network/interfaces
   @@ -2,8 +2,6 @@
    # and how to activate them. For more information, see interfaces(5).
    
    # The loopback network interface
   -auto lo
   -iface lo inet loopback
    
    # The primary network interface
    #allow-hotplug eth0
   

4. Uups!! Where did loopback disappeared?

   dpavlin@brr:~$ bak revert /etc/network/interfaces 
   dpavlin@brr:~$ bak diff
   

5. If we are content with changes, we can also commit them:

   dpavlin@brr:~$ bak commit /etc/network/interfaces optional note
   

As you guessed by now, it's very similar to git usage (expect revert which is from subversion) but with easy deployment on clients. It implements reduced subset of git commands:

• bak add /path
• bak commit [/path [message]]
• bak diff
• bak status
• bak log
• bak - push all local changes to server (without commit!)

If you need anything more complex, you can use git directly on ~/backup repository (even to commit changes from multiple hosts in one go).

Whole solution seems like ftp protocol, with data channel using ssh and rsync. File transfer should be encrypted (since we are trying to manage configuration files with sensitive information) and if you want to be really secure, just run server on 127.0.0.1 and tunnel port using RemoteForward 9001 localhost:9001 in .ssh/config.

I wrote initial version of bak-git more than a month ago, and started using it to manage my share of Internet cloud. Currently, it's 16 hosts, some of them real hardware, some OpenVZ or LXC containers.

Since then, I implemented two new features in bak-git:

• lighttpd configuration for gitweb with ssl and auth (so you can easily review your changes)
• ability to do diff and revert on remote hosts

Having all configuration files in central place should allow central management of shared part. Let's see how we can manage central apt proxy configuration which should be shared between machines in your lib cloud (cluster, intranet, whatever...)

On central node, you create proxy configuration under new fake host _lib

dpavlin@klin:~/klin/backup$ cat _lib/etc/apt/apt.conf.d/02proxy 
Acquire::http { Proxy "http://10.60.0.91:3142"; };

Then you can login into any other host and check:

dpavlin@mjesec:~$ bak diff _lib:/etc/apt/apt.conf.d/02proxy

no output, so same as in central _lib configuration, but

dpavlin@opl:~$ bak diff _lib:/etc/apt/apt.conf.d/02proxy
--- opl/etc/apt/apt.conf.d/02proxy      1970-01-01 01:00:00.000000000 +0100
+++ _lib/etc/apt/apt.conf.d/02proxy     2010-03-18 23:22:55.000000000 +0100
@@ -0,0 +1 @@
+Acquire::http { Proxy "http://10.60.0.91:3142"; };

This seem like missing configuration. Let's install it (revert from shared configuration template _lib):

dpavlin@opl:~$ bak revert _lib:/etc/apt/apt.conf.d/02proxy

Have in mind that this didn't commit this configuration change to bak-git, it just created file on local file system.

bak diff hostname:/path is more powerful than that. It can make diff of file which isn't tracked on remote host with local one, allowing easy comparison of any file on file-system to another file with same path on remote host. As a (useful) side-effect, file will be copied to central server, but not committed. You can choose to commit it later, or remove it from backup directory, but handy copy at diff time is nice to because it records your interest in that file.

I also misused git's Author: field to track user which committed change:

Author: root/dpavlin 

This means that I was using sudo to become root on hostname prod. To create more useful log in gitweb, I also prefixed messages with hostname: to create nicer output in gitweb:

I don't like inserting redundant information in message, but it's very useful to see author, hostname and message at a glance and if you want more details, you can always use bak log or bak changes on hosts or git directly in your backup directory:

dpavlin@klin:~/klin/backup$ git log --stat
commit 3ba2f2ebde044232983e0ea9ffdeb2afc0012cf9
Author: root/dpavlin <prod>
Date:   Sat Mar 27 01:12:15 2010 +0100

    prod: snap /mnt/koha

 prod/etc/cron.d/btrfs-snap |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

commit 40779a5c13ee12462b3b50f7ea1ace2363facd58
Author: root/dpavlin <prod>
Date:   Sat Mar 27 00:54:15 2010 +0100

    prod: firewall mysql

 prod/etc/rc.local |   18 ++++++++++++++++++
 1 files changed, 18 insertions(+), 0 deletions(-)

commit 6a433b853c9da2817bc76afa2e35cc1ed360c590
Author: root <koha>
Date:   Fri Mar 26 23:29:00 2010 +0100

    koha: default hredupersonexpiredate

 koha/etc/koha/koha-conf.xml |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)