Dobrica Pavlinušić's Weblog / Blog

After two years of hibernation, my efforts at implementing simple TR-069 server in perl is alive again. General motivation was to try out few new ADSL modems from ZTE with my server. And it wasn't easy...

General idea of my server is to do implementation which is easy to try and hopefully works with all quirks in existing implementations. And there are A LOT of quirks.

CWMP is basically a sick idea of twisting SOAP into something which will provide persistent connection to client which can be used to introspect device options and set them up. Back in 2007 when I started implementation, current version of TR-069 was 1.0. In meantime, DSL forum (which changed it's name to Broadband forum) decided to push out new TR-069 version 1.1, which I still haven't seen on any devices.

I have anticipated different implementations, so in first round of implementation I decided to parse XML using XML::Rules. This allowed me to use DSL-like language to implement extraction of interesting element from XML responses sent by CPE devices (CPEs are ADSL devices sitting in our homes). And that worked fairly well (at least it was tested with Thompson and Zyxel devices). Until, I got hold of ZTE device that is. It decided to return binary date inside one of XML tags (without CDATA around it) making XML::Parser on which XML::Rules depend bark.

So, I had to change XML parser. I opted for XML::Bare which doesn't have problems with invalid XML, but I had to implement rules part on my own. Which wasn't huge problem because I used only subset of functionality.

But then I tried to setup some data using SetParameterValues. And noticed that it doesn't work. This was strange, and after a two days of trials and errors I found out that ZTE devices require ParameterKey field in XML. Strange thing is that this parameter is defined as optional in TR-069 specification, and it isn't really used: it's enough to have just empty tag in generated XML to make ZTE device happy.

This is not only protocol violation which I found. Specification clearly defined that SOAPAction header is mandatory in all requests, but, ZTE decided not to send it at all. After all this you might wonder why would I like to put so much effort in implementing semi-defined protocol which is really useful only to IPSs. My initial motivation was to provide first free implementation of TR-069, but nowadays there are few TR-069 alternatives: OpenACS is probably best known, but it requires Java, JBOSS and MySQL just to get started.

And I didn't really care about user interface at all. I wanted to be able to write a small snippet of perl code which would allow easy customization of CPE configuration without going through user interface.

To do that, I decided to implement on-disk queue using IPC::DirQueue in which specific commands for clients are inserted and then sent one-by-one until queue for this client is empty. In current version all CPE parameters are first introspected (getting current values and writable status) and stored in simple YAML file on disk, allowing easy review or import into another system.

If you have fixed configuration which you want to push to all client, you can just create vendor.yaml file by copy/paste from file generated by CPE introspection and all CPEs pick up setting from it.

If you wanted to do something more complex (make a lookup into legacy database and deduce configuration from it, for example) you can always extend CWMP::Vendor with new perl function will implement something like this.

To make this even easier, I decided to move project to git (so perl-cwmp is now available on github) and setup publicly visible ACS server which you can use to check weather your CPE implementation of TR-069 will work with my server. This requires you to have administrative privileges on your CPE device, and probably isn't good idea if your provider is already managing your modem using TR-069. But, in that case, you won't be able to change it's setting anyway :-)

However, this would mean that you would have to send full passwords to me also (because TR-069 doesn't encrypts them) so basically you are forced to install version locally to try it out if you have any passwords in your configuration which you don't want to share with everyone.

I'm big fan of xterm with bitmap fonts (usually fixed, Terminus or Neep) for my terminals, mostly because it's small and quick (compare find . in your home directory full of files in some terminal with TrueType fonts and xterm with bitmap fonts to see difference).

But, I really wanted to have alternative way to change font size (other than Ctrl+right-click). I already wrote about xtermcontrol, but in my short usage of Gnome Terminal I noticed that having keyboard shortcuts for changing font size is really handy. And as always, xterm has solution for it. Put something like this in your ~/.Xdefaults:

XTerm.VT100.translations: #override \
Meta <Key> minus: smaller-vt-font() \n\
Meta <Key> plus: larger-vt-font() \n\
Meta <Key> KP_Subtract: smaller-vt-font() \n\
Meta <Key> KP_Add: larger-vt-font() \n\
Super <Key> minus: smaller-vt-font() \n\
Super <Key> plus: larger-vt-font() \n\
Super <Key> KP_Subtract: smaller-vt-font() \n\
Super <Key> KP_Add: larger-vt-font() \n

You can invoke this configuration with something like xrdb -merge ~/.Xdefaults which you can put in your .xinitrc. You can make various other adjustments (see man xterm for all details), but usually you just want to change default font and reverse video (to get white on black terminal) using something like:

XTerm.VT100.font: -*-terminus-medium-r-normal-*-16-*-*-*-*-*-iso10646-*
XTerm.VT100.reverseVideo: true

It started as a battery problem. It reported 100% full status, but wouldn't turn laptop on. I went back to T-Com center and tried with another battery, but it didn't work either. Now it's back for repair.

I was wondering where huge savings for netbook class hardware comes from. Partly it's smaller display, older chipset, but partly it's integration testing. Nobody ever tried to discharge this battery within laptop and than charge it again in this laptop (except me). When you buy server, you know that it has passed burn-in period of at least 48 hours. When I install it, I always leave it on for 24 h before I begin to work on it. As it seems, this is also good idea for laptops.

Update: After a day, Ideapad S10-2 stopped working on batteries and is now back for repair. Last week, I had presentation about Virtual LDAP, and during it I become painfully aware that my current Eee PC 701 is no longer sufficient as primary mobile machine (three seconds to change slide is just too much). To make my troubles worse, I own three Eee PCs, and X200 tablet, so why would I need another ultra-mobile PC?

Well, because it is ultra-mobile. Don't get me wrong, X200 is beautiful machine, but with 9 cell battery (and with that price) it's just not something which you will carry always in your bag. I also really liked 9" netbooks form-factor. I don't mind small keyboard, because I want to be able to type on it holding netboot between my hands vertically!

So, I went in search for laptop which will fullfil following requirements:

• Have 1024 horizontal resolution which is really required for Web
• Have integrated 3G modem, so I don't have to carry dongle with me around and bluetooth (I need to use GPS or transfer pictures from cellphone)
• Have internal flash
• It has to pass my shoulder test: I have to be able to carry it around with me without noticing. 900 g Eee PC 701 passes, 1450 g OLPC doesn't.

So, how did I end up with Ideapad S10-S? I looked at Asus Eee PC 1005HA, but deciding factor was really keyboard layout. Let me introduce best keyboard layout I ever saw on small keyboards:

Display

I had to settle with 1024*600 vertical resolution, because it seems that this is only kind of 10" display available in Croatia within my price range. At least, it's a little better than 1024*540 that Ideapad S10 has... Built-in flash was also hard to find in netbook. There wasn't any.

Display is compromise: it's not as tall as your projected presentation (which will be 1024*768), so you won't see whole slide on your display, but other than that it's quite usable for web surfing and terminal usage. It's shiny, but I couldn't find any netbooks with non-glossy display. Bigger problem is maximum opening angle of display. I would love to be able to open it another 15 degrees or so. I'm not quite sure that there is structural reson for this limit, and 1005HA has exactly same angle, so it's conspiracy to force us into buying tablets which don't have this problem.

It's really shiny. So shiny in fact, that I had to use xgamma to get contrast high enough to be able to read Web pages with grayish text on them.

dpavlin@ipad:~$ xgamma -g 0.75,1
-> Red  1.000, Green  1.000, Blue  1.000
<- Red  0.750, Green  0.750, Blue  0.750

3G modem and bluetooth

Dongle is nice, integrated modem is even nicer. Ericsson F3507g has GPS built-in (think what this means for your privacy for a moment because it doesn't work without SIM card inserted, and GSM operators can inquire your position at any time without your knowledge).
Having said that 3G modem, GPS and bluetooth are available, so let's move on...

SSD storage

I made conscious decision that I want splash top built into my netbook. Not because it's really useful (it isn't really), but because I could re-install critical part of my extended system on it and really have *same* working configuration booting from flash. I don't have any idea about size of flash (I seem to remember reading that it's only 512Mb), and I don't have any access to it for now, so we count this one failure.

Shoulder test

I have only one day expirience, but It seems that 1220 g of Ideapad alone without power adapter passes. Battery lasted long enough for intense half-day usage (you know, new toy, click, click), so it will probably be OK for whole day without dragging power adapter everywhere.

I decided to install Ubuntu Netbook remix on it, to try out if I could recommend this to other netbook users. In it's current incarnation, I ended up with do-release-upgrade -d to get 2.6.32 kernel. I did try backports, and bcmwl-kernel-source before going to alpha version, in fact, I tried harder that most normal first-time users would to get wireless working under Linux.

Current development version of Ubuntu did enable wireless, but GUI launcher freezes sometimes, which makes it unusable at current moment for normal netbook users. But, next version of Ubuntu will work without a glitch on this netbook, and in my opinion is much better choice than Windows 7 on this machine. It will give users much more performance on same hardware. Yes, I tried Windows 7 which came with it, it took 15 seconds to open control panel and display 6 icons. Than I noticed it has a slowly moving progress bar on top, and I got used to seeing it often. I'm not impressed.

All in all, this netbook is great improvement if you need smallest possible machine which you can carry around with you.

I know that title is mouthful. But, I occasionally use this blog as place to dump interesting configuration settings and it helps me remember configuration which helps me to remember it and might be useful to lone surfers who stumbles upon this page.

This weekend our nice network admin upgraded VLAN setup to enables 1Gb/s network across whole private segment. That is clearly visible in green arrows on picture, especially if you compare it with last one. We still have a few slower links (100 Mb/s) which pass through NAT to Internet (red arrows), but we now have private segment on every server in library and system room in main building and each server has IP address in new 1 Gb/s segment (you can notice that by blue arrows which are loopback interface on same machine).

All is not well, however. I had to reconfigure my OpenVZ containers from point-to-point ip configuration using venet devices to ethernet bridge setup because multiple hops on OpenVZ machine produced havoc with connections from our private network to containers. It was very wired, I saw TCP retransmission requests, but first packet somehow managed to pass through. ICMP traffic worked (thanks to small packet sizes), but I can't honestly say why our new VLAN setup and/or OpenVZ had problem with it. Up to that point, I just had private IP addresses assigned to OpenVZ container using vzctl set --ipadd 10.60.0.12 and ping -R did show multiple public and private addresses, but it all somehow worked.

First, I had to make network bridge which will be new VLAN 60

root@koha-hw:~# cat /etc/network/interfaces

auto br60
iface br60 inet static
        bridge_ports eth1 veth212226
        bridge_fd 0
        address 10.60.0.10
        netmask 255.255.254.0
        gateway 10.60.0.1

root@koha-hw:~# vzctl set 212226 --ipdel 10.60.0.12 --save

root@koha-hw:~# vzctl set 212226 --netif_add veth1,00:12:34:56:78:9A,veth212226,00:12:34:56:78:9B,br60 --save

koha:~# ifconfig veth1 10.60.0.12 netmask 255.255.254.0 up

Next, I needed to reconfigure my ssh VPN which was using point-to-point tunneling with tap into real ethernet tunnel:

root@brr:~# tail -10 /etc/network/interfaces 

# vpn lib -> home
auto tap0
iface tap0 inet static
        pre-up ssh -M -S /var/run/tap0 -v -f -i /root/.ssh/koha-hw-tun0 -w 0:0 -o Tunnel=ethernet -o ServerAliveInterval=5 -o ServerAliveCountMax=2 koha-hw.ffzg.hr true
        address 10.60.0.81
        netmask 255.255.0.0
        up iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o tap0 -j MASQUERADE
        post-down ssh -S /var/run/tap0 -O exit koha-hw.ffzg.hr

root@koha-hw:~# head -1 /root/.ssh/authorized_keys 
tunnel="0",command="ifconfig tap0 up ; brctl addif br60 tap0" ssh-rsa AAAAB3Nza...

Finally, little cron job to start (and restart) VPN:

root@brr:~# cat /etc/cron.d/tap0 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/1     * * * * root    fping -q 10.60.0.10 || ( ifdown tap0 ; ifup tap0 )

If you are system administrator this will sound familiar: you have to quickly fix something, and you know that you should document it somewhere (or keep backup) but it's so much work. You could install one of existing source control management tools on each box, but they usually come with huge dependencies, and having all files in central location would be so useful to co-relate configuration changes. To add insult to injury, existing SCMs don't do good job in tracking just few files spread across file-system.

So, what would be perfect tool for keeping remote files in central git repository look like?

• no dependency on non-standard tools on clients allowing easy deployment
• track individual files and ignore rest
• central repository, one directory per hostname

I tried to solve this problem several times, writing wrappers around subversion to handle sparse checkouts and installing subversion and ssh authentication all over the place. But, all this should be simpler... Like this:

1. add new client to track:

   dpavlin@klin:~/klin/bak-git$ ./bak-git-server.pl ~/backup/ 10.60.0.92 --install brr
   install on brr
   # lot of output stripped
   

   This will do several steps:
   • create git repository in ~/backup/ if it doesn't exist already
   • install root ssh authentication to brr using ssh-copy-id
   • install bak shell helper which uses netcat to connect back to 10.60.0.92
   • install rsync on client and use it as root over ssh to sync files
2. Now we can login into brr and start tracking our files:

   dpavlin@brr:~$ bak add /etc/cron.d/tun0 
   dpavlin@brr:~$ bak add /etc/network/interfaces
   dpavlin@brr:~$ bak commit
   dpavlin@brr:~$ bak log
   commit df09dc5e19ef1d47311d701b4c63f0859b0b81c1
   Author: Dobrica Pavlinusic 
   Date:   Thu Feb 18 19:04:21 2010 +0100
   
       brr [commit] /home/dpavlin/
   
    create mode 100644 brr/etc/cron.d/tun0
    create mode 100644 brr/etc/network/interfaces
   

3. change some configuration and review changes

   dpavlin@brr:~$ bak diff
   diff --git a/brr/etc/network/interfaces b/brr/etc/network/interfaces
   index 806c08e..c52c646 100644
   --- a/brr/etc/network/interfaces
   +++ b/brr/etc/network/interfaces
   @@ -2,8 +2,6 @@
    # and how to activate them. For more information, see interfaces(5).
    
    # The loopback network interface
   -auto lo
   -iface lo inet loopback
    
    # The primary network interface
    #allow-hotplug eth0
   

4. Uups!! Where did loopback disappeared?

   dpavlin@brr:~$ bak revert /etc/network/interfaces 
   dpavlin@brr:~$ bak diff
   

5. If we are content with changes, we can also commit them:

   dpavlin@brr:~$ bak commit /etc/network/interfaces optional note
   

• bak add /path
• bak commit [/path [message]]
• bak diff
• bak status
• bak log
• bak - push all local changes to server (without commit!)

Whole solution seems like ftp protocol, with data channel using ssh and rsync. File transfer should be encrypted (since we are trying to manage configuration files with sensitive information) and if you want to be really secure, just run server on 127.0.0.1 and tunnel port using RemoteForward 9001 localhost:9001 in .ssh/config.

I have been following MySQL saga with Sun and Oracle for quite some time, mostly because we decided to go mainstream with our Koha installation and use MySQL instead of PostgreSQL because it's support is experimental.

It seems that MariaDB is the future of MySQL if you trust Monty. There is also Drizzle but it's different enough that it's not considered drop-in replacement for MySQL. So, now that I have some hope that MySQL is here to stay I needed to find a quick way to analyze my performance on it.

And there is a great way to see overview of your slow query log: mk-query-digest. It's part of Maatkit which is very interesting and useful project for MySQL DBAs. So, with something like:

dpavlin@mjesec:~$ wget -q http://maatkit.org/get/mk-query-digest
dpavlin@mjesec:~$ perl mk-query-digest /var/log/mysql/mysql-slow.log

You have heard of Apple Time Capsule, and zfs's snapshots which provide nice way to produce incremental backups which you can use to quickly recover a rm -Rf typo or compare changes... You can't really use LVM snapshots for that because they have fixed size of snapshot. ZFS has copy-on-write snapshots, but zfs-fuse isn't fastest solution on Linux.

I decided to give btrfs another try. On Debian 2.6.32-trunk kernels, it stills have problems with 686 kernel, but amd64 version (even with 32-bit users-pace) seems to work stable so far.

Let's take a look at example usage on /dev/vg/koha-btrfs logical volume.

1. create filesystem

   root@mlin:~# mkfs.btrfs /dev/vg/koha-btrfs
   

2. create subvolume which we will use for data

   root@mlin:~# btrfsctl -S koha-2010-01-25 /virtual.btrfs/
   

3. populate it with some data

   root@mlin:~# time cp -ra /virtual.clone/koha-2010-01-25 /virtual.btrfs/
   
   real    15m32.507s
   user    0m1.288s
   sys     0m54.519s
   

4. create base snapshot

   root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.mlin /virtual.btrfs/koha-2010-01-25
   

   There is convention here: since snapshot directories have to be on same btrfs volume, I decided to use base_name(dot)something as convention for my snapshots.
5. make some changes on base directory

   root@mlin:~# time rsync -ravH --numeric-ids --sparse --delete --exclude 'backup*' 10.60.0.90:/opl/clone/koha-2010-01-25/ /virtual.btrfs/koha-2010-01-25/
   
   sent 1538957 bytes  received 381670534 bytes  964049.03 bytes/sec
   total size is 18912566086  speedup is 49.35
   
   real    6m37.539s
   user    0m59.524s
   sys     1m23.633s
   

6. make another snapshot for this version

   root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.opr /virtual.btrfs/koha-2010-01-25
   

dpavlin@mlin:~$ ls -ald /virtual.btrfs/koha-2010-01-25*
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.mlin
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.opr

dpavlin@mlin:~$ mount -t btrfs
/dev/mapper/vg-koha--btrfs on /virtual.btrfs type btrfs (rw,noatime)


root@mlin:~# lvdisplay /dev/vg/koha-btrfs 
  --- Logical volume ---
  LV Name                /dev/vg/koha-btrfs
  VG Name                vg
  LV UUID                kl4QUc-IyK2-IM8m-0DKD-eI5k-H2T1-HySIwE
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                40.00 GB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

   
root@mlin:~# lvextend -L +10G /dev/vg/koha-btrfs
  Extending logical volume koha-btrfs to 50.00 GB
  Logical volume koha-btrfs successfully resized


root@mlin:~# btrfsctl -r max /virtual.btrfs/

Hopefully this will help you to get started with btrfs snapshots. I still don't consider it super stable (especially since i did saw few kernel oopses using 2.6.32-trunk-686) but for quick experiments with Linux containers (or anything stored on filesystem for that matter) it proved more than adequate.

When you are working as system architect or systems librarian, your job is to design systems. My initial idea was to create small Google out of 12 machines which are dedicated to be web kiosks. I decided to strictly follow loosely coupled principle, mostly to provide horizontal scaling for my data processing needs. I wanted to be able to add machine or two if my query is too slow... This easily translates into "now long will I have to wait for my page to generate results"....

I decided to split my system into three logical parts: network booting, data store, and quick reporting. So, let's take a look at each component separately:

• PXElator
  • supported protocols: bootp, dhcp, tftp, http, amt, wol, syslog
  • boot kiosks using Webconverger (Debian Live based kiosk distribution)
  • provides web user interface for overview of network segment for audit
  • configuration is stored as files on disk, suitable for management with git or other source control management
• MongoDB
  • NoSQL storage component which support ad-hoc queries, indexes and other goodies
  • simple store for perl hashes from PXElator generated every time we see network packet from one of clients using one of supported protocols
• Sack
  • fastest possible way to execute snippet of perl code over multiple machines
  • this involves sharing information to nodes, executing code on all of them and collecting results back, all in sub 3 second mark!
  • web user interface for cloud overview and graph generation using gnuplot

When I started implementing this system last summer, I decided to use CouchDB for storage layer. This wasn't really good choice, since I didn't need transactions, MVCC or replication. Hack, I even implemented forking for document stored in CouchDB to provide faster response to clients in PXElator.

Moving to much faster MongoDB I got ad-hoc queries which are usable (as in I can wait for them to finish) and if that's too slow, I can move data to Sack and query it directly from memory. As a happy side effect, making shards from MongoDB is much faster than using CouchDB bulk HTTP API, and it will allow me to feed shards directly from MongoDB to Sack nodes, without first creating shards on disk.

I'm quite happy how it all turned out. I can configure any host using small snippet of perl code in PXElator, issue ad-hoc queries on audit data on it in MongoDB or move data to Sack if I want to do data munging using perl.

As you noticed by now, I'm using live distribution for kiosks, and machines do have hard drivers in them. Idea was to use those disks as storage with something like Sheepdog. seems like perfect fit. With it in place, I will have real distributed, building size computer :-).

I have been using CouchDB for some time now, mostly as audit storage for PXElator. Audit data stores are most useful for ad-hoc queries (hum, when did I saw that host last time?), and CouchDB map/reduces took half an hour or more. I wrote mall script couchdb2mongodb.pl to migrate my data over to MongoDB (in 26 minutes) and run first query I could write after reading MongoDB documentation about advanced queries. It took only 30 seconds, compared to 30 minutes or more in CouchDB. I was amazed.

This was NoSQL database which I can understand and tune. MongoDB has indexes and profiler so tuning query down to three seconds was a simple matter of adding an index. All my RDBMS knowledge was reusable here, so I decided to take a look why is it so much faster than CouchDB for same data...

To be honest, MongoDB, High-Performance SQL-Free Database by Dwight Merriman, CEO of 10gen won me over to finally try MongoDB. It was technical enough to make me think about MongoDB arhitecture and benefits. It's clearly pragmatic, let's re-think horizontally scalable hash storage with ad-hoc queries model, but with funny twist about close coupling with language types all encoded in BSON format, which is very similar to Google's protocol buffers.

First, let's have a look at raw side of data on disk. At some level, it will translate to number of IO operations involving rotating platters and usage of buffer cache.

root@opr:~# du -hc /var/lib/couchdb/0.9.0/.pxelator* /var/lib/couchdb/0.9.0/pxelator.couch
655M    /var/lib/couchdb/0.9.0/.pxelator_design
23M     /var/lib/couchdb/0.9.0/.pxelator_temp
7.8G    /var/lib/couchdb/0.9.0/pxelator.couch
8.4G    total

root@opr:~# du -hc /var/lib/mongodb/pxelator.*
65M     /var/lib/mongodb/pxelator.0
129M    /var/lib/mongodb/pxelator.1
257M    /var/lib/mongodb/pxelator.2
513M    /var/lib/mongodb/pxelator.3
513M    /var/lib/mongodb/pxelator.4
513M    /var/lib/mongodb/pxelator.5
17M     /var/lib/mongodb/pxelator.ns
2.0G    total

MongoDB has montodump utility which dumps bson for backup and it's even smaller:

root@opr:~# du -hcs dump/pxelator/*
1.1G    dump/pxelator/audit.bson
4.0K    dump/pxelator/system.indexes.bson
76K     dump/pxelator/system.profile.bson
1.1G    total

So I switched PXElator to use MongoDB as storage. I never pushed anything in production after just one day of testing it, but first query speedup from 30 min to 30 sec, and ability to cut it down to 3 sec if I added index (which took about 13 sec to create) is just something which provides me with powerful analytical tool I didn't have before.