February 2010 Archives

It started as a battery problem. It reported 100% full status, but wouldn't turn laptop on. I went back to T-Com center and tried with another battery, but it didn't work either. Now it's back for repair.

I was wondering where huge savings for netbook class hardware comes from. Partly it's smaller display, older chipset, but partly it's integration testing. Nobody ever tried to discharge this battery within laptop and than charge it again in this laptop (except me). When you buy server, you know that it has passed burn-in period of at least 48 hours. When I install it, I always leave it on for 24 h before I begin to work on it. As it seems, this is also good idea for laptops.

Update: After a day, Ideapad S10-2 stopped working on batteries and is now back for repair. Last week, I had presentation about Virtual LDAP, and during it I become painfully aware that my current Eee PC 701 is no longer sufficient as primary mobile machine (three seconds to change slide is just too much). To make my troubles worse, I own three Eee PCs, and X200 tablet, so why would I need another ultra-mobile PC?

Well, because it is ultra-mobile. Don't get me wrong, X200 is beautiful machine, but with 9 cell battery (and with that price) it's just not something which you will carry always in your bag. I also really liked 9" netbooks form-factor. I don't mind small keyboard, because I want to be able to type on it holding netboot between my hands vertically!

So, I went in search for laptop which will fullfil following requirements:

• Have 1024 horizontal resolution which is really required for Web
• Have integrated 3G modem, so I don't have to carry dongle with me around and bluetooth (I need to use GPS or transfer pictures from cellphone)
• Have internal flash
• It has to pass my shoulder test: I have to be able to carry it around with me without noticing. 900 g Eee PC 701 passes, 1450 g OLPC doesn't.

So, how did I end up with Ideapad S10-S? I looked at Asus Eee PC 1005HA, but deciding factor was really keyboard layout. Let me introduce best keyboard layout I ever saw on small keyboards:

Display

I had to settle with 1024*600 vertical resolution, because it seems that this is only kind of 10" display available in Croatia within my price range. At least, it's a little better than 1024*540 that Ideapad S10 has... Built-in flash was also hard to find in netbook. There wasn't any.

Display is compromise: it's not as tall as your projected presentation (which will be 1024*768), so you won't see whole slide on your display, but other than that it's quite usable for web surfing and terminal usage. It's shiny, but I couldn't find any netbooks with non-glossy display. Bigger problem is maximum opening angle of display. I would love to be able to open it another 15 degrees or so. I'm not quite sure that there is structural reson for this limit, and 1005HA has exactly same angle, so it's conspiracy to force us into buying tablets which don't have this problem.

It's really shiny. So shiny in fact, that I had to use xgamma to get contrast high enough to be able to read Web pages with grayish text on them.

dpavlin@ipad:~$ xgamma -g 0.75,1
-> Red  1.000, Green  1.000, Blue  1.000
<- Red  0.750, Green  0.750, Blue  0.750

3G modem and bluetooth

Dongle is nice, integrated modem is even nicer. Ericsson F3507g has GPS built-in (think what this means for your privacy for a moment because it doesn't work without SIM card inserted, and GSM operators can inquire your position at any time without your knowledge).
Having said that 3G modem, GPS and bluetooth are available, so let's move on...

SSD storage

I made conscious decision that I want splash top built into my netbook. Not because it's really useful (it isn't really), but because I could re-install critical part of my extended system on it and really have *same* working configuration booting from flash. I don't have any idea about size of flash (I seem to remember reading that it's only 512Mb), and I don't have any access to it for now, so we count this one failure.

Shoulder test

I have only one day expirience, but It seems that 1220 g of Ideapad alone without power adapter passes. Battery lasted long enough for intense half-day usage (you know, new toy, click, click), so it will probably be OK for whole day without dragging power adapter everywhere.

I decided to install Ubuntu Netbook remix on it, to try out if I could recommend this to other netbook users. In it's current incarnation, I ended up with do-release-upgrade -d to get 2.6.32 kernel. I did try backports, and bcmwl-kernel-source before going to alpha version, in fact, I tried harder that most normal first-time users would to get wireless working under Linux.

Current development version of Ubuntu did enable wireless, but GUI launcher freezes sometimes, which makes it unusable at current moment for normal netbook users. But, next version of Ubuntu will work without a glitch on this netbook, and in my opinion is much better choice than Windows 7 on this machine. It will give users much more performance on same hardware. Yes, I tried Windows 7 which came with it, it took 15 seconds to open control panel and display 6 icons. Than I noticed it has a slowly moving progress bar on top, and I got used to seeing it often. I'm not impressed.

All in all, this netbook is great improvement if you need smallest possible machine which you can carry around with you.

I know that title is mouthful. But, I occasionally use this blog as place to dump interesting configuration settings and it helps me remember configuration which helps me to remember it and might be useful to lone surfers who stumbles upon this page.

This weekend our nice network admin upgraded VLAN setup to enables 1Gb/s network across whole private segment. That is clearly visible in green arrows on picture, especially if you compare it with last one. We still have a few slower links (100 Mb/s) which pass through NAT to Internet (red arrows), but we now have private segment on every server in library and system room in main building and each server has IP address in new 1 Gb/s segment (you can notice that by blue arrows which are loopback interface on same machine).

All is not well, however. I had to reconfigure my OpenVZ containers from point-to-point ip configuration using venet devices to ethernet bridge setup because multiple hops on OpenVZ machine produced havoc with connections from our private network to containers. It was very wired, I saw TCP retransmission requests, but first packet somehow managed to pass through. ICMP traffic worked (thanks to small packet sizes), but I can't honestly say why our new VLAN setup and/or OpenVZ had problem with it. Up to that point, I just had private IP addresses assigned to OpenVZ container using vzctl set --ipadd 10.60.0.12 and ping -R did show multiple public and private addresses, but it all somehow worked.

First, I had to make network bridge which will be new VLAN 60

root@koha-hw:~# cat /etc/network/interfaces

auto br60
iface br60 inet static
        bridge_ports eth1 veth212226
        bridge_fd 0
        address 10.60.0.10
        netmask 255.255.254.0
        gateway 10.60.0.1

root@koha-hw:~# vzctl set 212226 --ipdel 10.60.0.12 --save

root@koha-hw:~# vzctl set 212226 --netif_add veth1,00:12:34:56:78:9A,veth212226,00:12:34:56:78:9B,br60 --save

koha:~# ifconfig veth1 10.60.0.12 netmask 255.255.254.0 up

Next, I needed to reconfigure my ssh VPN which was using point-to-point tunneling with tap into real ethernet tunnel:

root@brr:~# tail -10 /etc/network/interfaces 

# vpn lib -> home
auto tap0
iface tap0 inet static
        pre-up ssh -M -S /var/run/tap0 -v -f -i /root/.ssh/koha-hw-tun0 -w 0:0 -o Tunnel=ethernet -o ServerAliveInterval=5 -o ServerAliveCountMax=2 koha-hw.ffzg.hr true
        address 10.60.0.81
        netmask 255.255.0.0
        up iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o tap0 -j MASQUERADE
        post-down ssh -S /var/run/tap0 -O exit koha-hw.ffzg.hr

root@koha-hw:~# head -1 /root/.ssh/authorized_keys 
tunnel="0",command="ifconfig tap0 up ; brctl addif br60 tap0" ssh-rsa AAAAB3Nza...

Finally, little cron job to start (and restart) VPN:

root@brr:~# cat /etc/cron.d/tap0 
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
*/1     * * * * root    fping -q 10.60.0.10 || ( ifdown tap0 ; ifup tap0 )

If you are system administrator this will sound familiar: you have to quickly fix something, and you know that you should document it somewhere (or keep backup) but it's so much work. You could install one of existing source control management tools on each box, but they usually come with huge dependencies, and having all files in central location would be so useful to co-relate configuration changes. To add insult to injury, existing SCMs don't do good job in tracking just few files spread across file-system.

So, what would be perfect tool for keeping remote files in central git repository look like?

• no dependency on non-standard tools on clients allowing easy deployment
• track individual files and ignore rest
• central repository, one directory per hostname

I tried to solve this problem several times, writing wrappers around subversion to handle sparse checkouts and installing subversion and ssh authentication all over the place. But, all this should be simpler... Like this:

1. add new client to track:

   dpavlin@klin:~/klin/bak-git$ ./bak-git-server.pl ~/backup/ 10.60.0.92 --install brr
   install on brr
   # lot of output stripped
   

   This will do several steps:
   • create git repository in ~/backup/ if it doesn't exist already
   • install root ssh authentication to brr using ssh-copy-id
   • install bak shell helper which uses netcat to connect back to 10.60.0.92
   • install rsync on client and use it as root over ssh to sync files
2. Now we can login into brr and start tracking our files:

   dpavlin@brr:~$ bak add /etc/cron.d/tun0 
   dpavlin@brr:~$ bak add /etc/network/interfaces
   dpavlin@brr:~$ bak commit
   dpavlin@brr:~$ bak log
   commit df09dc5e19ef1d47311d701b4c63f0859b0b81c1
   Author: Dobrica Pavlinusic 
   Date:   Thu Feb 18 19:04:21 2010 +0100
   
       brr [commit] /home/dpavlin/
   
    create mode 100644 brr/etc/cron.d/tun0
    create mode 100644 brr/etc/network/interfaces
   

3. change some configuration and review changes

   dpavlin@brr:~$ bak diff
   diff --git a/brr/etc/network/interfaces b/brr/etc/network/interfaces
   index 806c08e..c52c646 100644
   --- a/brr/etc/network/interfaces
   +++ b/brr/etc/network/interfaces
   @@ -2,8 +2,6 @@
    # and how to activate them. For more information, see interfaces(5).
    
    # The loopback network interface
   -auto lo
   -iface lo inet loopback
    
    # The primary network interface
    #allow-hotplug eth0
   

4. Uups!! Where did loopback disappeared?

   dpavlin@brr:~$ bak revert /etc/network/interfaces 
   dpavlin@brr:~$ bak diff
   

5. If we are content with changes, we can also commit them:

   dpavlin@brr:~$ bak commit /etc/network/interfaces optional note
   

• bak add /path
• bak commit [/path [message]]
• bak diff
• bak status
• bak log
• bak - push all local changes to server (without commit!)

Whole solution seems like ftp protocol, with data channel using ssh and rsync. File transfer should be encrypted (since we are trying to manage configuration files with sensitive information) and if you want to be really secure, just run server on 127.0.0.1 and tunnel port using RemoteForward 9001 localhost:9001 in .ssh/config.

I have been following MySQL saga with Sun and Oracle for quite some time, mostly because we decided to go mainstream with our Koha installation and use MySQL instead of PostgreSQL because it's support is experimental.

It seems that MariaDB is the future of MySQL if you trust Monty. There is also Drizzle but it's different enough that it's not considered drop-in replacement for MySQL. So, now that I have some hope that MySQL is here to stay I needed to find a quick way to analyze my performance on it.

And there is a great way to see overview of your slow query log: mk-query-digest. It's part of Maatkit which is very interesting and useful project for MySQL DBAs. So, with something like:

dpavlin@mjesec:~$ wget -q http://maatkit.org/get/mk-query-digest
dpavlin@mjesec:~$ perl mk-query-digest /var/log/mysql/mysql-slow.log

You have heard of Apple Time Capsule, and zfs's snapshots which provide nice way to produce incremental backups which you can use to quickly recover a rm -Rf typo or compare changes... You can't really use LVM snapshots for that because they have fixed size of snapshot. ZFS has copy-on-write snapshots, but zfs-fuse isn't fastest solution on Linux.

I decided to give btrfs another try. On Debian 2.6.32-trunk kernels, it stills have problems with 686 kernel, but amd64 version (even with 32-bit users-pace) seems to work stable so far.

Let's take a look at example usage on /dev/vg/koha-btrfs logical volume.

1. create filesystem

   root@mlin:~# mkfs.btrfs /dev/vg/koha-btrfs
   

2. create subvolume which we will use for data

   root@mlin:~# btrfsctl -S koha-2010-01-25 /virtual.btrfs/
   

3. populate it with some data

   root@mlin:~# time cp -ra /virtual.clone/koha-2010-01-25 /virtual.btrfs/
   
   real    15m32.507s
   user    0m1.288s
   sys     0m54.519s
   

4. create base snapshot

   root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.mlin /virtual.btrfs/koha-2010-01-25
   

   There is convention here: since snapshot directories have to be on same btrfs volume, I decided to use base_name(dot)something as convention for my snapshots.
5. make some changes on base directory

   root@mlin:~# time rsync -ravH --numeric-ids --sparse --delete --exclude 'backup*' 10.60.0.90:/opl/clone/koha-2010-01-25/ /virtual.btrfs/koha-2010-01-25/
   
   sent 1538957 bytes  received 381670534 bytes  964049.03 bytes/sec
   total size is 18912566086  speedup is 49.35
   
   real    6m37.539s
   user    0m59.524s
   sys     1m23.633s
   

6. make another snapshot for this version

   root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.opr /virtual.btrfs/koha-2010-01-25
   

dpavlin@mlin:~$ ls -ald /virtual.btrfs/koha-2010-01-25*
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.mlin
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.opr

dpavlin@mlin:~$ mount -t btrfs
/dev/mapper/vg-koha--btrfs on /virtual.btrfs type btrfs (rw,noatime)


root@mlin:~# lvdisplay /dev/vg/koha-btrfs 
  --- Logical volume ---
  LV Name                /dev/vg/koha-btrfs
  VG Name                vg
  LV UUID                kl4QUc-IyK2-IM8m-0DKD-eI5k-H2T1-HySIwE
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                40.00 GB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

   
root@mlin:~# lvextend -L +10G /dev/vg/koha-btrfs
  Extending logical volume koha-btrfs to 50.00 GB
  Logical volume koha-btrfs successfully resized


root@mlin:~# btrfsctl -r max /virtual.btrfs/

Hopefully this will help you to get started with btrfs snapshots. I still don't consider it super stable (especially since i did saw few kernel oopses using 2.6.32-trunk-686) but for quick experiments with Linux containers (or anything stored on filesystem for that matter) it proved more than adequate.