March 2010 Archives

I wrote initial version of bak-git more than a month ago, and started using it to manage my share of Internet cloud. Currently, it's 16 hosts, some of them real hardware, some OpenVZ or LXC containers.

Since then, I implemented two new features in bak-git:

  • lighttpd configuration for gitweb with ssl and auth (so you can easily review your changes)
  • ability to do diff and revert on remote hosts
Having all configuration files in central place should allow central management of shared part. Let's see how we can manage central apt proxy configuration which should be shared between machines in your lib cloud (cluster, intranet, whatever...)

On central node, you create proxy configuration under new fake host _lib

dpavlin@klin:~/klin/backup$ cat _lib/etc/apt/apt.conf.d/02proxy 
Acquire::http { Proxy "http://10.60.0.91:3142"; };
Then you can login into any other host and check:
dpavlin@mjesec:~$ bak diff _lib:/etc/apt/apt.conf.d/02proxy
no output, so same as in central _lib configuration, but
dpavlin@opl:~$ bak diff _lib:/etc/apt/apt.conf.d/02proxy
--- opl/etc/apt/apt.conf.d/02proxy      1970-01-01 01:00:00.000000000 +0100
+++ _lib/etc/apt/apt.conf.d/02proxy     2010-03-18 23:22:55.000000000 +0100
@@ -0,0 +1 @@
+Acquire::http { Proxy "http://10.60.0.91:3142"; };
This seem like missing configuration. Let's install it (revert from shared configuration template _lib):
dpavlin@opl:~$ bak revert _lib:/etc/apt/apt.conf.d/02proxy
Have in mind that this didn't commit this configuration change to bak-git, it just created file on local file system.

bak diff hostname:/path is more powerful than that. It can make diff of file which isn't tracked on remote host with local one, allowing easy comparison of any file on file-system to another file with same path on remote host. As a (useful) side-effect, file will be copied to central server, but not committed. You can choose to commit it later, or remove it from backup directory, but handy copy at diff time is nice to because it records your interest in that file.

I also misused git's Author: field to track user which committed change:

Author: root/dpavlin 
This means that I was using sudo to become root on hostname prod. To create more useful log in gitweb, I also prefixed messages with hostname: to create nicer output in gitweb:

bak-git-gitweb-messages.png

I don't like inserting redundant information in message, but it's very useful to see author, hostname and message at a glance and if you want more details, you can always use bak log or bak changes on hosts or git directly in your backup directory:

dpavlin@klin:~/klin/backup$ git log --stat
commit 3ba2f2ebde044232983e0ea9ffdeb2afc0012cf9
Author: root/dpavlin <prod>
Date:   Sat Mar 27 01:12:15 2010 +0100

    prod: snap /mnt/koha

 prod/etc/cron.d/btrfs-snap |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

commit 40779a5c13ee12462b3b50f7ea1ace2363facd58
Author: root/dpavlin <prod>
Date:   Sat Mar 27 00:54:15 2010 +0100

    prod: firewall mysql

 prod/etc/rc.local |   18 ++++++++++++++++++
 1 files changed, 18 insertions(+), 0 deletions(-)

commit 6a433b853c9da2817bc76afa2e35cc1ed360c590
Author: root <koha>
Date:   Fri Mar 26 23:29:00 2010 +0100

    koha: default hredupersonexpiredate

 koha/etc/koha/koha-conf.xml |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

Linux containers have network configuration similar to plain Linux. In fact, it is plain Linux network configuration! But, user-space lxc tools hides some of steps from you, but allow all reconfigurability you would need. So, let's have a look how to configure two distinct network interfaces, and some options you really want to set...

Let's take a look at configuration for container which uses dual homed network, one internal (on br0 bridge) and another external one (on br1 bridge):
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.ipv4 = 10.60.0.84/23
lxc.network.mtu = 1500
lxc.network.hwaddr = AC:DE:48:00:00:54
lxc.network.veth.pair = veth84

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br1
lxc.network.name = eth1
lxc.network.mtu = 1500
lxc.network.ipv4 = 193.198.212.253/22
lxc.network.hwaddr = AC:DE:48:00:D4:FD
lxc.network.veth.pair = veth212253
Names of some options are somewhat confusing, so let's take a look what lxc-start does for us:
  • create interface for container named lxc.network.veth.pair with mac lxc.network.hwaddr (both parameters are optional, but if you don't specify them you will get randomly generated values which won't be very useful for debugging or monitoring)
  • join that interface into bridge lxc.network.link
  • inside container
    • interface is named lxc.network.name
    • ip address from lxc.network.ipv4 (this allows external IP address configuration of container, with only default route inside container)
    • interface status to lxc.network.flags

You will notice that I'm using prefix AC:DE:48 prefix for my mac addresses, which, to best of my knowledge range for private use and used all over IEEE docs (something like private IP range, but for mac addresses). I also have habit of naming interfaces with last octet of IP adress for internal ones, and last two for external one and same notation for mac addresses. Our brain is good at spotting patterns, and if you can read hex this seems just natural...

I have been playing with Linux containers for a while, and finally I decided to take a plunge and migrate one of my servers from OpenVZ to lxc. It worked quite well for testing until I noticed lack of support for automatic startup or shutdown. lxc-users mailing list was helpful in providing useful hints and I found 5 OpenSUSE scripts but decided it's too complicated for task at hand.

I wanted single script, for easy deployment on any Debian box, with following features:

  • reboot and halt from inside container should work as expected
  • cleanly shutdown or reboot container from host (as opposed to lxc-stop which is equivalent to turning power off)
  • init script to start and stop containers on host boot and shutdown

Result is lxc-watchdog.sh. It can start containers (and remember to start it on next host reboot), reboot or halt containers from host (using signals to trigger container's init) and automatically configure your container when you start it for the first time. Here is quick overview:

root@prod:/srv# ./lxc-watchdog.sh status
koha-240 RUNNING boot /virtual/koha-240
koha-241 STOPPED boot /virtual/koha-241
koha-242 STOPPED      /virtual/koha-242

root@prod:/srv# ./lxc-watchdog.sh start
# start koha-240
'koha-240' is RUNNING
# start koha-241
2010-03-16T23:44:16 koha-241 start
'koha-241' is RUNNING
# skip start koha-242

root@prod:/srv# ./lxc-watchdog.sh status
koha-240 RUNNING boot /virtual/koha-240
koha-241 RUNNING boot /virtual/koha-241
koha-242 STOPPED      /virtual/koha-242

root@prod:/srv# ls -al /var/lib/lxc/*/on_boot
-rw-r--r-- 1 root root 9 2010-03-16 21:40 /var/lib/lxc/koha-240/on_boot
-rw-r--r-- 1 root root 9 2010-03-16 21:40 /var/lib/lxc/koha-241/on_boot
-rw-r--r-- 1 root root 0 2010-03-16 22:58 /var/lib/lxc/koha-242/on_boot
As you can see, I used file /var/lib/lxc/name/on_boot to record which machines to bring up. When container is started for the first time, it will have boot enabled (just in case this is production application which you will reboot in 6 months and then wonder why it doesn't work). You can change boot status using:
root@prod:/srv# ./lxc-watchdog.sh boot koha-242
# boot koha-242

root@prod:/srv# ./lxc-watchdog.sh status
koha-240 RUNNING boot /virtual/koha-240
koha-241 RUNNING boot /virtual/koha-241
koha-242 STOPPED boot /virtual/koha-242

root@prod:/srv# ./lxc-watchdog.sh disable koha-242
# disable koha-242
Installation as init script /etc/init.d/lxc-watchdog is easy:
root@prod:/srv# ln -s /srv/lxc-watchdog.sh /etc/init.d/lxc-watchdog

root@prod:/srv# update-rc.d lxc-watchdog defaults
update-rc.d: using dependency based boot sequencing
And finally, it can also be used to manually start, halt or reboot containers:
root@prod:/srv# /etc/init.d/lxc-watchdog start koha-242
# start koha-242
2010-03-16T23:47:46 koha-242 start
'koha-242' is RUNNING

root@prod:/srv# /etc/init.d/lxc-watchdog status
koha-240 RUNNING boot /virtual/koha-240
koha-241 RUNNING boot /virtual/koha-241
koha-242 RUNNING      /virtual/koha-242

root@prod:/srv# /etc/init.d/lxc-watchdog restart koha-242
# restart koha-242
2010-03-16T23:48:46 koha-242 kill -SIGINT 24838

root@prod:/srv# /etc/init.d/lxc-watchdog status
koha-240 RUNNING boot /virtual/koha-240
koha-241 RUNNING boot /virtual/koha-241
koha-242 RUNNING      /virtual/koha-242

root@prod:/srv# /etc/init.d/lxc-watchdog stop koha-242
# stop koha-242
2010-03-16T23:49:55 koha-242 stop
2010-03-16T23:49:55 koha-242 kill -SIGPWR 26086
2010-03-16T23:50:11 koha-242 stoped
In fact, you can use halt or reboot if you don't like stop and restart, just to keep one mapping less in your brain when working with it.

Log files are created for each container in /tmp/name.log. They include lxc-start output with boot messages and any output that started scripts might create which is useful for debugging container installations. Output from watchdog monitoring /var/run/utmp in container is also included, and it reports number of tasks (processes) in container, and here is example of stopping container:

root@prod:/srv# tail -5 /tmp/koha-242.log
2010-03-16T23:49:56 koha-242 66 tasks
2010-03-16T23:50:04 koha-242 22 tasks
2010-03-16T23:50:11 koha-242 runlevel 2 0
2010-03-16T23:50:11 koha-242 halt
2010-03-16T23:50:12 koha-242 watchdog exited
Hopefully this will make your switch to Linux Containers and recent kernels easier...

After two years of hibernation, my efforts at implementing simple TR-069 server in perl is alive again. General motivation was to try out few new ADSL modems from ZTE with my server. And it wasn't easy...

General idea of my server is to do implementation which is easy to try and hopefully works with all quirks in existing implementations. And there are A LOT of quirks.

CWMP is basically a sick idea of twisting SOAP into something which will provide persistent connection to client which can be used to introspect device options and set them up. Back in 2007 when I started implementation, current version of TR-069 was 1.0. In meantime, DSL forum (which changed it's name to Broadband forum) decided to push out new TR-069 version 1.1, which I still haven't seen on any devices.

I have anticipated different implementations, so in first round of implementation I decided to parse XML using XML::Rules. This allowed me to use DSL-like language to implement extraction of interesting element from XML responses sent by CPE devices (CPEs are ADSL devices sitting in our homes). And that worked fairly well (at least it was tested with Thompson and Zyxel devices). Until, I got hold of ZTE device that is. It decided to return binary date inside one of XML tags (without CDATA around it) making XML::Parser on which XML::Rules depend bark.

So, I had to change XML parser. I opted for XML::Bare which doesn't have problems with invalid XML, but I had to implement rules part on my own. Which wasn't huge problem because I used only subset of functionality.

But then I tried to setup some data using SetParameterValues. And noticed that it doesn't work. This was strange, and after a two days of trials and errors I found out that ZTE devices require ParameterKey field in XML. Strange thing is that this parameter is defined as optional in TR-069 specification, and it isn't really used: it's enough to have just empty tag in generated XML to make ZTE device happy.

This is not only protocol violation which I found. Specification clearly defined that SOAPAction header is mandatory in all requests, but, ZTE decided not to send it at all. After all this you might wonder why would I like to put so much effort in implementing semi-defined protocol which is really useful only to IPSs. My initial motivation was to provide first free implementation of TR-069, but nowadays there are few TR-069 alternatives: OpenACS is probably best known, but it requires Java, JBOSS and MySQL just to get started.

And I didn't really care about user interface at all. I wanted to be able to write a small snippet of perl code which would allow easy customization of CPE configuration without going through user interface.

To do that, I decided to implement on-disk queue using IPC::DirQueue in which specific commands for clients are inserted and then sent one-by-one until queue for this client is empty. In current version all CPE parameters are first introspected (getting current values and writable status) and stored in simple YAML file on disk, allowing easy review or import into another system.

If you have fixed configuration which you want to push to all client, you can just create vendor.yaml file by copy/paste from file generated by CPE introspection and all CPEs pick up setting from it.

If you wanted to do something more complex (make a lookup into legacy database and deduce configuration from it, for example) you can always extend CWMP::Vendor with new perl function will implement something like this.

To make this even easier, I decided to move project to git (so perl-cwmp is now available on github) and setup publicly visible ACS server which you can use to check weather your CPE implementation of TR-069 will work with my server. This requires you to have administrative privileges on your CPE device, and probably isn't good idea if your provider is already managing your modem using TR-069. But, in that case, you won't be able to change it's setting anyway :-)

However, this would mean that you would have to send full passwords to me also (because TR-069 doesn't encrypts them) so basically you are forced to install version locally to try it out if you have any passwords in your configuration which you don't want to share with everyone.

I'm big fan of xterm with bitmap fonts (usually fixed, Terminus or Neep) for my terminals, mostly because it's small and quick (compare find . in your home directory full of files in some terminal with TrueType fonts and xterm with bitmap fonts to see difference).

But, I really wanted to have alternative way to change font size (other than Ctrl+right-click). I already wrote about xtermcontrol, but in my short usage of Gnome Terminal I noticed that having keyboard shortcuts for changing font size is really handy. And as always, xterm has solution for it. Put something like this in your ~/.Xdefaults:

XTerm.VT100.translations: #override \
Meta <Key> minus: smaller-vt-font() \n\
Meta <Key> plus: larger-vt-font() \n\
Meta <Key> KP_Subtract: smaller-vt-font() \n\
Meta <Key> KP_Add: larger-vt-font() \n\
Super <Key> minus: smaller-vt-font() \n\
Super <Key> plus: larger-vt-font() \n\
Super <Key> KP_Subtract: smaller-vt-font() \n\
Super <Key> KP_Add: larger-vt-font() \n
I decided to bind Meta (usually Alt) or Super (usually Windows key) and + or - on main keyboard or numeric keypad for quick and fail-safe access to terminal font size. As always with X server resource database whitespace is very important. Notice space after #override in first line.

You can invoke this configuration with something like xrdb -merge ~/.Xdefaults which you can put in your .xinitrc. You can make various other adjustments (see man xterm for all details), but usually you just want to change default font and reverse video (to get white on black terminal) using something like:

XTerm.VT100.font: -*-terminus-medium-r-normal-*-16-*-*-*-*-*-iso10646-*
XTerm.VT100.reverseVideo: true

About this Archive

This page is an archive of entries from March 2010 listed from newest to oldest.

February 2010 is the previous archive.

April 2010 is the next archive.

Find recent content on the main index or look in the archives to find all content.

Pages

  • pics
OpenID accepted here Learn more about OpenID
Powered by Movable Type 5.04