I'm using Subversion for most of my work as all of you well know by now. All this hype about git persuaded me to give it another try. I don't really have anything to gain doing this, since I'm using svk when I need distributed VCS, but somehow I though that git might be right solution to keep all my system configuration so I can debootstrap system checkout configuration and I'm ready to go.

I could use etckeeper to do some of this stuff, but I really didn't want integration with apt. I just wanted single (network connected and backed up) place. I already tried this with git on single machine with local repository, and it worked pretty well.

This time I tried to use git branches to track different machines. I really want a single repository, so I can merge common changes all around. However, today I got this:

root@syslog:/# git push
Counting objects: 16, done.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (10/10), 1.16 KiB, done.
Total 10 (delta 0), reused 0 (delta 0)
To ssh://backup/srv/backup/
   66a2f9b..8f195f5  syslog -> syslog
 ! [rejected]        koha-dev -> koha-dev (non-fast forward)
 ! [rejected]        master -> master (non-fast forward)
error: failed to push some refs to 'ssh://backup/srv/backup/'

I have no idea why would two branches which have nothing to do with current one would disable distributed part of git. If you can't read git output, message above means that I wasn't able to commit my changes to central repository.

This is a huge show stopper for me. Half a day of googling didn't find answer for this particular git question. This makes my whole setup a big useless overhead.

This is all well and fun, but since this is second time that git ate my data I'm falling back to good old friend Subversion. At least, when in breaks, I have error messages which are somewhat useful, Subversion book which explains most operations with it (so I don't have to google for every little bit, like pushing single branch from one repository to another with git).

Don't get me wrong: git is c00l, we all know that, but it's just immature if you don't want to be git developer. If you think that I'm just old-timer which can't join all this new-age DVCS mumble-mumble, read why Google picked Mercurial instead of git as DVCS. Different story, but helpful to see that git isn't only solution to every problem

Now I just need to convert my existing git branches back into subversion. It seems that git-svn dcommit is answer, but how to really push four different git branches back into subversion I still don't know. I will probably just re-add all tracked files to clean Subversion and start all over again.

So, you want to checkout source code on some server and at the same time have ability to commit or diff against local copy on your laptop? Seems like an easy task, but it does involve some unexpected steps (for me at least), so here is a quick how to...

Create ssh tunnel from target host back to your laptop (called llin in output). Edit .ssh/config and add something like:

Host server-dev.rot13.org server-dev
	Hostname server-dev.rot13.org
	RemoteForward 8022 127.0.0.1:22

You will notice that I added short name so I can type just ssh server-dev because I'm lazy.

When you login to server-dev you might think that something like svn checkout is everything that is left. However, that doesn't really work:

dpavlin@server-dev:~$ svn ls svn+ssh://localhost:8022/home/dpavlin/private/svn/SQL2XLS
ssh: localhost:8022: Name or service not known
svn: Connection closed unexpectedly

It seems that subversion doesn't like port number within hostname! So, let's make .ssh/config there also:

Host llin-svn
	Hostname localhost
	Port 8022

Let's try it out:

dpavlin@server-dev:~$ svn ls svn+ssh://llin-svn/home/dpavlin/private/svn/SQL2XLS
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Password or swipe finger: 

Much better, but is asks us for password every time. We don't really like that, so we'll create ssh keys to get around this:

dpavlin@server-dev:~$ ssh-keygen -f .ssh/llin-svn
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in .ssh/llin-svn.
Your public key has been saved in .ssh/llin-svn.pub.
The key fingerprint is:
aa:bb:cc:dd:ee:ff:00:11:22:33:44:55:66:77:88:99 dpavlin@server-dev.rot13.org

Now we will insert generated .ssh/llin-svn.pub into .ssh/authorized_keys on laptop but allow only svnserver to be started:

command="svnserve -t" ssh-rsa AAA...rest of key...AAA== dpavlin@server-dev.rot13.org

If you want normal ssh login back to your laptop, you might leave out command="svnserve -t", but this makes me feel better. On the other hand, tunnel will be open only when we are logged into server-dev, but I usually prefer more security if possible. If you don't want to commit back to laptop, you might add -R flag to make repository read-only.

But wait, there is more! We need to tell ssh on server-dev that we are using newly generated key, so our final .ssh/config looks like this:

Host llin-svn
	Hostname localhost
	Port 8022
	IdentityFile ~/.ssh/llin-svn

We can test it now to make sure that subversion doesn't ask for password by simply checking out source code:

dpavlin@server-dev:~$ svn co svn+ssh://llin-svn/home/dpavlin/private/svn/SQL2XLS
Checked out revision 0.

Happy hacking!

This might seem like a strange title, but let's look at it for a moment. If subversion commit messages carry magic marker (RT #1234 in my case) I want to create RT comment (micro-blog post) with link back to SVN::Web repository (and thus commit which created comment in first place). Nice and circular :-)

First, install subversion post-commit hook in /srv/svn/repository/hooks/post-commit:

#!/bin/sh

REPOS="$1"
REV="$2"

/srv/svn/svn-rt-comment.pl $REPOS $REV

After that, create following script:

#!/usr/bin/perl

use strict;
use warnings;

use RT::Client::REST;
use RT::Client::REST::Ticket;

# Request Tracker
my ( $server, $username, $password ) = ( 'https://bugs.example.com/rt', 'rtuser', 'rtpasswd' );
# patternt to recognize RT references in commits log or diff
my $rt_re = qr/rt\s*#\s*(\d+)/i;
my $svnweb = 'https://svn.example.com/svnweb/index.cgi/repository/revision/?rev=';

die "usage: $0 repo rev\n" unless @ARGV;

my ( $repo, $rev ) = @ARGV;

sub svnlook {
	my $command = shift;
	`svnlook $command --revision $rev $repo`
}

my $log = svnlook 'log';
my $diff = svnlook 'diff';

if ( $log =~ $rt_re || $diff =~ $rt_re ) {

	my $id = $1 or die "no id";

	my $rt = RT::Client::REST->new( server  => $server );
	$rt->login( username=> $username, password=> $password );
	my $ticket = RT::Client::REST::Ticket->new( rt  => $rt, id  => $id );

	my $message =
		svnlook 'author' . "\t" . svnlook 'date' . "\n" .
		$svnweb . $rev . "\n\n" . 
		svnlook 'changed --copy-info' . "\n" .
		$log
	;
	
	$ticket->comment( message => $message );
}

And apply following patch to your Request Tracker 3.6 to create links in messages:

--- /usr/share/request-tracker3.6/html/Ticket/Elements/ShowMessageStanza	2006-06-20 00:44:04.000000000 +0200
+++ /usr/local/share/request-tracker3.6/html/Ticket/Elements/ShowMessageStanza	2008-09-26 13:23:12.000000000 +0200
@@ -57,8 +57,12 @@
         my $content = $stanza->{raw};
         RT::Interface::Web::EscapeUTF8(\$content);
         $m->comp('/Elements/Callback', content => \$content, %ARGS);
-        $content =~ s{$}{
}mg
-            if defined $content;
+        if ( defined($content) ) {
+		# convert urls to links
+		$content =~ s{(https?://\S+)}{<a href="$1">$1</a>};
+		$content =~ s{$}{<br />}mg;
+		warn "## $content\n";
+	}
 
 
 <%$content |n%>

And you have your micro-blogging environment connected by linking :-)

Update: If you have problems using SVN::Web with Subversion 1.5 take a look at this ticket which includes svn-web-svn1.5.diff which fixes it.

We had a double-disk failure on RAID5 array last week (yes, it does happen). One of things that got affected was our source code repository.

So, we reinstalled etch and installed subversion, did svnadmin verify on repository and all seemed well until we tried to commit using SVN DAV. Apache's error log was full of errors like this:

Could not create activity /svn/repo/!svn/act/9526cf2d-4893-4b35-a843-4c0f2d9f8bc6. [500, #0]

could not open dbm files. [500, #120022]

Can't open activity db: APR does not understand this error code [500, #120022]

But, permissions where correct! In utter desperation, I even tried chmod 777 on repository (kids, don't try this at home!) but it didn't help.

What did help was:

mkdir repo.new

svnadmin create repo.new

svnadmin dump repo | svnadmin load repo.new

mv repo repo.old

mv repo.new repo

Checking created repository with diff showed that it was indeed changed:

Binary files repo.old/dav/activities and repo/dav/activities differ

diff -rw repo.old/db/current repo/db/current

1c1

< 59 r9 1

---

> 62 ra 1

Only in repo/db/revprops: 60

Only in repo/db/revprops: 61

Only in repo/db/revprops: 62

diff -rw repo.old/db/revs/10 repo/db/revs/10

4c4

< id: aq.0.r10/17

---

> id: e1.0.r10/17

6c6

< pred: aq.0.r8/1443323

---

> pred: e1.0.r8/1443490

16c16

< dir b0.0.r8/1443476

---

> dir av.0.r8/1443643

and so on, for pages and pages.

I don't feel very good about changes, but repository was created with newer version from backports, and this time we used version from etch (thus downgrading from 1.4.6 to 1.4.2).

The whole point of this post is to help some random Google user with downgrade of subversion. So, here you have it!

I had a particular problem at work: we have upstream subversion repository which we access over ssh tunnel (using svn protocol) which contains two branches in which we are interested and various other stuff we don't care about (and don't want to mirror).

On other hand, we also wanted to have local copy of all changes (preserving history) and local commit messages and SVN::Web interface.

In original idea, I also wanted to keep revision numbers as-is (so I can just checkout our local version and be done), but this wasn't possible. One solution that we examined is to use Pushmi and make local copy, but we didn't want all the other changes.

Other idea was to use svndumpfilter to sync only two branches we are interested in (it will create dummy commits for revision which are outside our branches), but since branches are result of copy from parts of the tree we don't want to sync, it didn't work either.

Did I mentioned that our svn repository can access upstream only through carefully crafted ssh tunnels? Mess, right?

So, in the end, solution was hybrid:

• make local copy of two upstream branches using svk (loosing original order of commits, even if we are commiting into same svk mirror copy at our side)
• install post-commit hook in upstream repository which will call (over https) svk sync at our side (I would probably use SMTP to trigger that, but our machine with svn repository doesn't accept outside e-mail)
• install local post-commit hook to send e-mail notifications

Rest of this post are instructions on how to do this. Since I learned a thing of two doing this, I hope it might be also useful for others.

First create svn-pull.sh shell script which will run under user which has ssh keys to login to upstream firewall (1.2.3.4 in this example) and setup tunnels to upstream svn server (10.1.1.1):

#!/bin/sh
ssh -L 13690:10.1.1.1:3690 1.2.3.4 sleep 2 &
pid=$!
SVKROOT=/home/user/.svk svk sync -a
kill $pid

Now setup mirrors of branches we care about:

svk mirror svn://127.0.0.1:13690/project/carnet-foo /project/foo
svk mirror svn://127.0.0.1:13690/project/carnet-bar /project/bar

This is all nice, but we need to trigger it from www-user which is done with following in /etc/sudoers:

www-data ALL=(user) NOPASSWD:/home/user/svn-pull.sh

and add simple cgi script which will trigger sync operation:

#!/bin/sh
echo -e "Content-type: text/plain\n\r\n\r"
sudo -u user /home/user/svn-pull.sh

I used ScriptAlias in apache to make it visible at https://svn-ours.example.com/upstream-svn-update. No need on obsucate URL, since it's behind SSL for added points. IP address limit might also be a good idea:

  <Location /upstream-svn-update>                   
        Order allow,deny
        Allow from 1.2.3.4
  </Location>

Now install post-commit hook in upstream repository. We care only for files which have /carnet in path since branches which we are interested have that prefix:

svn log -v -r $REV file://$REPOS | grep ' /carnet' 2>/dev/null \
    && wget -q -O /dev/null https://svn-ours.example.com/upstream-svn-update

You will notice that there are no locking or any other tweaks, since all tools have those capabilities anyway, so we are really just using RPC via cgi over https in fact.

Nice and easy, once you know how to do it! It seems like a few bits of configuration all over the place, but I hope that it employs KISS - keep it simple and stupid at it's best.

Update: OK, now we have local repository (with different revisions), but svn switch --relocate doesn't work because those repositories are not same (makes sense, eh?)

Following steps are quick explanation now to copy .svn directories from new repository:

cd /srv/carnet-foo
# update repository to last upstream version
svn update
# delete old .svn directories
find . -name ".svn" -exec rm -Rf {} \;
# checkout new repository
cd /srv
svn co svn://svn-ours.example.com/carnet-foo carnet-foo.new
# copy new .svn files to old repository
cd carnet-foo.new
find . -wholename "*/.svn/*" | cpio -pvd ../carnet-foo/
# cleanup
cd /srv
rm -Rf carnet-foo.new
# following shouldn't return any differences
cd carnet-foo
svn diff

In an effort to continue my hack-of-the-week series, here is a quick overview of few subversion hacks I have worked on lately:

• svn-ignore.sh is a tiny shell script which will bring all unversioned files in current svn or svk repository in your $EDITOR and add result of your edit to svn:ignore
• svndump-move.pl is more complex perl script which will allow you to reorganize directory layout in your repository while preserving revision history -- it solved problems like: oh, if I only had root of my repository is subdirectory foo...
• svn2cvs is a bit older tool which received attention when Bartek Teodorczyk very patiently started to report problems with it. As a result, it now has test suite, and it's much more robust

Most of documentation for those tools is hidden in subversion commit messages. If you think they are useful, take a peek there...

So, it seems that having all my projects in subversion just isn't enough for me, so here is a little gem: svn-overlay.sh script which will, when placed in root of your repository, overlay all directories in that repository on your root directory (/).

Why would that be useful, you might ask? Well, did you ever wanted to track changes to configuration files? How about keeping in sync config files on two different machines? Does installation of your favourite project overwrite your custom changes (or additions)?

All that can be solved simply by creating svn repository and keeping your changes in it, like this:

svnadmin create /srv/svn-config
svn mkdir -m "track /etc" file:///srv/svn-config/etc/
svn mkdir -m "track /etc/init.d" file:///srv/svn-config/etc/init.d
svn import -m "add overlay script" svn-overlay.sh file:///srv/svn-config/svn-overlay.sh

Now we will create make temporary checkout (which can be deleted later):

cd /srv
svn co file:///srv/svn-config local-config
cd local-config
sudo ./svn-overlay.sh

After this you can rm -Rf /srv/local-config if you don't want to have another copy on your disk (it was needed so that svn-overlay.sh would know repository path and that might get fixed in some future version).

After that, your /etc and /etc/init.d directories are under svn control. You will probably have to use sudo to add files to svn (or fix permissions of .svn directories), but other than that it's simple svn add something, svn commit, svn diff routine.

I don't know why I didn't came up with this earlier. I remember reading about keeping of your home directory in subversion, but it all seemed so complicated. Only thing that I can object is those .svn directories. svk might be solution for that, but in some other post...

I guess, it's easy, once you know how to do it :-)

1. cache your user credentials

$ svn checkout https://webpac2.googlecode.com/svn/trunk/ webpac2 --username dpavlin 

2. install and init SVN::Mirror

$ cpanp i SVN::Mirror
$ export SVMREPOS=~/svm 
$ svm init google/ https://webpac2.googlecode.com/svn/
Committed revision 1.
$ svm sync google/
Syncing https://webpac2.googlecode.com/svn
Retrieving log information from 1 to 1
Committed revision 2 from revision 1.

3. create branch and import history

$ svn cp file://$SVMREPOS/google file://$SVMREPOS/initial_import -m "import of history"
Committed revision 3.
$ svn rm file://$SVMREPOS/initial_import/trunk -m "removed pre-created trunk"
Committed revision 4.
$ svnadmin dump ~/private/svn/webpac2/ --incremental | svnadmin load $SVMREPOS --parent-dir /initial_import

[... a lot of output ...]

4. Check if all revisions are here

$ svn info file://$SVMREPOS/initial_import/      
Path: initial_import
URL: file:///home/dpavlin/svm/initial_import
Repository Root: file:///home/dpavlin/svm/initial_import
Repository UUID: 07558da8-63fa-0310-ba24-9fe276d99e06
Revision: 663
Node Kind: directory
Last Changed Author: dpavlin
Last Changed Rev: 663
Last Changed Date: 2006-09-08 19:49:09 +0200 (Fri, 08 Sep 2006)

5. Find first revision to import (last changed in google branch+1)

$ svn info file://$SVMREPOS/google/
Path: google
URL: file:///home/dpavlin/svm/google
Repository Root: file:///home/dpavlin/svm
Repository UUID: 15ab016f-fa1c-0410-a7d3-f6d91a34b6aa
Revision: 670
Node Kind: directory
Last Changed Author: dpavlin
Last Changed Rev: 5
Last Changed Date: 2006-09-09 02:09:47 +0200 (Sat, 09 Sep 2006)

6. import other revisions (this will take forever)

$ perl -e 'print "$_\n" for ( 6 .. 670 )' | xargs -i svm mergeback google initial_import {}

7. watch progress with

$ svn log -v https://webpac2.googlecode.com/svn/

It sucks to loose timestamps, but at least you have your revisions...

---

Update: Lately I would reccomend to use SVN::Pusher to sync your changes. Downside is that SVN::Pusher won't work with file:/// URIs, but other that that it seems like a much easier approach. Usage goes something like this:

$ svn-pusher push -r 663:782 svn://svn.rot13.org/webpac2/ https://webpac2.googlecode.com/svn/