For a start you need compatible Realtek RTL2832U DVB-T tuner. Just go over to rtl-sdr osmocomSDR and examine supported tuner list. Choice of DVB-T sticks in Croatia isn't so good. They are about 170 kn and come with FC001[23] tuners: Hama 53176 has FC0013 tuner and TV LV5TDLX has FC0012 tuner. I did some research and ordered this $13 DVB-T from dealextreme which has R820T tuner:

dpavlin@blue:~$ sudo rtl_test -t
Using device 0: ezcap USB 2.0 DVB-T/DAB/FM dongle
Found Rafael Micro R820T tuner

I also have a good fortune that my friend is very skillful in radio communications, so he created variant of Coaxial Collinear Antenna for ADS-B Receiver. This one is more sensitive to static electricity which means it's not suitable for top of skyscraper scenarios, but in works very nicely. Under construction is filter which would cut everything expect interesting part of RF spectrum which would also solve problem of static electricity.

On the picture, you can see part of it together with output in browser created using dump1090. It's important to note that normal DVB-T antena without any modification will work with R820T tuner since it's most sensitive so far, but range will be much smaller.

We also had good luck with getting data about keypresses from power-socket controlling device using rtl_433 from github with just a peace of wire as antenna for short range tests.

First ssh into your host machine (garak in this example, tunneling some VNC ports along the way:

dpavlin@x200:~$ ssh $(seq 5900 5904 | awk '{ print "-L "$1":localhost:"$1 }') garak

Then, using virtsh list all domains, and examine vnc port:

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     stargate                       running
 2     treca                          running
 15    hroug                          running
 16    close                          running

virsh # vncdisplay close
:2

dpavlin@x200:~$ vncviewer :2

First I asked for suggestions which wireless keyboard to buy. I have quite big fingers, so mini models just doesn't do it for me. I got suggestion to take a look at Logitech K400 and sure enough it seemed like good choice. One of first things that I noticed is that it supports 128-bit AES encryption. I started to have a good feeling about it, but I wanted to know more, so I hoped to Logitech Advanced 2.4 GHz Technology pdf and discovered that not all keys are encrypted.To quote documentation:

The encryption applies to all standard keyboard keys (a, s, d, f...) and modifiers (Shift, Ctrl, Alt...). The multimedia keys (Play, Pause, Mute...) that may be implemented in some keyboards are transmitted in clear text.

Sure I can. For older 27Mhz keyboards there is KeyKeriki v1.0 - 27MHz project which implement sniffer for it (video DeepSec 2009: Keykeriki: Universal Wireless Keyboard Sniffing For The Masses). But, Logitech is 2.4Ghz, so it's secure, right? Well, there is KeyKeriki v2.0 - 2.4GHz which does same for 2.4Ghz (video Keykeriki V2 - Practical Exploitation of Modern Wireless Devices [SIGINT10]). OK, Logitech does some kind of AES on top of that, but since it does transfer some keys unencrypted, and it's proprietary technology I can't really check that.

I also got suggestion to use bluetooth keyboard because it's secure. Well, quick search revealed Ubertooth One which basically defeats bluetooth protection with a bit of sniffing and a little brute force.

By this point, I was puzzled. Is there secure wireless keyboard with touchpad which I can buy? Something I can be sure that it encrypts all traffic as opposed to only some keys? Or is usb extension cable only real solution for me?

If you examine original Kickstarter page you will see following description:

A desktop 3D printer you can build in a couple hours. Print plastic parts you design or download - even parts for another printer.

This also introduced significant delay in printer shipment, but when you are funding Kickstarter project, you should be prepared for it, so I'm not complaining. When it finally arrived this summer (10 months after end of Kickstarter campaign), it was significantly bigger than I expected:

To be honest, we did upgrade to bigger Printrbot PLUS so I guess we should expect a lot of parts. As we are mostly software geeks, we did only reasonable thing to do: check if all parts are present comparing it with bill of materials which we got printed out.

This is the point where our problems started. We had missing one bag of parts which included termistor and switches. We contacted Printrbot HQ and they sent us missing parts. We started assembling following Printrbot Building Instructions by Brook Drumm and it took us more than 50 hours to get to our first blob.

Of course, it didn't work perfectly on first run. We where trying to print 5mm Calibration Cube Steps is ABS plastic which we received with our Printrbot (we even got additional 450g of ABS plastic as replacement for power supply which wasn't part of international shipments).

To be honest, we didn't expect smooth ride. However, as I mentioned before we are not really hardware hackers, and my only conclusion is that home-made 3D printers are really for people who already have enough experience to make their own 3D printer, and not for software guys like us. However, we won't give up, and I fully expect to have working printer (after we get replacement barrings from Printrbot HQ because our are sticky). We are collecting useful add-on models and instructions on our Printrbot wiki page but I didn't expect that we will have to contact Printrbot HQ twice for missing and replacement parts. But eventually we will be able to print Raspberry Pi box, I hope :-)

Scrub your array to check data consistency

I will assume that you are already monitoring your disks using smart, but from time to time it's useful to force full re-scan of your array to make sure that all data is still there and consistent. Some filesystems provide this option to scrub data on it's own (zfs and btrfs comes to mind) but if your filesystem is located on md array you can always force it using

echo check > /sys/block/md0/md/sync_action

Adding write-intent bitmap to speed up recovery

If you installed your md array a long time ago, you probably didn't turn on write-intent bitmap. It's very useful when you have to recover because bitmap will track changes and this will prevent long re-sync times when disks have to read and compare every block. To turn it on use:

mdadm --grow --bitmap=internal /dev/md0

Mirror between two devices of same speed

Recently, one of my 500Gb disks in RAID1 (mirror) failed. I decided to replace it with 1Gb drive which was unfortunately green drive (which basically means slow). Adding two drives of different speed in mirror will reduce performance to single slower drive which is a shame. Since I wasn't able to add additional disk and wasn't prepared to give up redundancy of data I started searching around and found that I can specify one disk as write-mostly using:

mdadm --add /dev/md0 --write-mostly /dev/sdb1

To get full-blown Debian-based distribution on your Android take a look at BotBrew Basil. It's Emdebian based distribution which will setup mount points and various other stuff so you don't have to do that manually. Since it's Debian based, you are not limited to Emdebian packages -- you can (and will have to) add normal sid:

(BotBrew)root@localhost:/# cat /etc/apt/sources.list.d/sid.list 
deb [arch=armel] http://ftp.debian.org/debian sid main contrib non-free

With all this prepared, we are ready to shut down Android stack:

adb shell setprop ctl.stop media
adb shell setprop ctl.stop zygote
adb shell setprop ctl.stop surfaceflinger
adb shell setprop ctl.stop drm

dpavlin@t61p:~$ adb shell
root@android:/ # TERM=xterm chroot /data/botbrew-basil/ /bin/bash --login
(BotBrew)root@localhost:/# apt-get install xserver-xorg-video-fbdev xserver-xorg-input-evdev \
   xserver-xorg-input-multitouch x11-xserver-utils xinit \
   matchbox matchbox-keyboard xterm

After installation you will need to setup X symlink and create .xinitrc:

root@android:/ # ln -s /usr/bin/Xorg /usr/bin/X

root@android:/ # cat ~/.xinitrc                                              
( sleep 1 ; matchbox-keyboard -o portrait ) &
xhost 192.168.1.61
matchbox-session

Section "ServerLayout"
    Identifier    "Layout0"
    Screen        "Screen0"
    InputDevice   "cyttsp-i2c" "CorePointer"
    InputDevice   "gpio-keys" "CoreKeyboard"
    InputDevice   "twl4030-keypad" "CoreKeyboard"
EndSection

Section "InputDevice"
    Identifier    "gpio-keys"
    Driver        "evdev"
    Option        "Device" "/dev/input/event0"
    # code 102 (KEY_HOME)
    # code 116 (KEY_POWER)
EndSection

Section "InputDevice"
    Identifier     "twl4030-keypad"
    Driver         "evdev"
    Option         "Device" "/dev/input/event1"
    # code 114 (KEY_VOLUMEDOWN)
    # code 115 (KEY_VOLUMEUP)
EndSection

Section "InputDevice"
    Identifier     "cyttsp-i2c"
    Driver         "multitouch"
    Option         "Device" "/dev/input/event2"
    # mouse should move as fast as finger and not faster
    Option         "AccelerationScheme" "none"
    # evdev has it, multitouch doesn't so it behaves like touchpad
#   Option         "IgnoreRelativeAxes" "True"
EndSection

Section "Device"
    Identifier    "Card0"
    Driver        "fbdev"
    Option        "fbdev" "/dev/graphics/fb0"
    # rotate screen to be in sync with touchpad orientation
    Option        "Rotate" "CCW" # CW=90 UD=180 CCW=270
EndSection

Section "Screen"
    Identifier    "Screen0"
    Device        "Card0"
EndSection

Following video shows X server in action.

This is simple unaccelerated frame buffer. This makes performance less then desirable. There are a few implementations of OMAP xorg server:

• xf86-video-omapfb uses DSS kernel support which seems to be part of CM kernel, so this might be good next thing to try out
• xf86-video-omap is newer implementation, but this requires 3.3 kernel and is not yet stable.

Most interesting part of Open Library (IMHO) is Internet Archive BookReader which is JavaScript application which allows users to browse scanned books on-line. For quite some time, I wanted to install something similar to provide web access to our collection of scanned documents. I have found instructions for serving IA like books from own cluster, but I didn't have a cluster, and converting all documents to IA book format seemed like an overhead which I would like to avoid.

Instead, I decided to write image server for JavaScript front-end using plack. I mean, it's basically a directory with images, right? Oh, how wrong can I be? :-)

It turs out that we have pictures in multiple formats (so sorting them required removing common prefix and using number only to get correct order), and most of are scanned images in pdf documents. Here are all types of documents which can be automatically collected into book for on-line browsing:

• images of scanned pages
• multi-file pdf file with single image per page
• single pdf file with one image for each page
• single pdf file with more than one (usually 4) horizontal bitmap strips for each page
• normal pdf documents which contain text and needs rendering to bitmap

Source code of my plack server for Internet Archive book reader is on github, so if you want to take a look, hop over there...

Sysadmin cookbook

I never proparly introduced this project here, but if you want to know more about my convention based documentation examine the presentation of hop over to http://sysadmin-cookbook.rot13.org/ and take a look at generated documentation.

Basic idea is to document changes in easy to write files on file system (preserving symlinks to files on system which allows you to quickly see if cookbook is deployed or not and diff between template and deployed configuration). I know that my cookbook is mix of various things I did in last three years, but I do find it useful, so hopefully it might be useful to you also.

Kindle - so much more than ebook reader

• Free Libre command-line tools for Kindle development
• A PDF (plus DJVU, ePub, TXT, CHM, FB2, HTML...) viewer made for e-ink framebuffer devices, using muPDF, djvulibre, crengine mobileread forum thread
• A VNC viewer for ebook readers which allows you to use Kindle as second screen mobileread forum thread

Web scale monitoring

This was a workshop which doesn't have much with web (it's about monitoring ADSL CPE devices and provider equipment in-between), but it shows (I hope) nice way to integrate several project to provide nice scalable monitoring infrastructure. It's composed of:

• Gearman message queue together with Gearman::Driver provide on-demand scaling of workers
• redis saves all data from external systems (LDAP, CRM) and all results from statistics collection nicely providing data for web interface
• PostgreSQL stores all collected data, using hstore to provide unstructured key value store for different data from different devices while still allowing us to use SQL to query data (and export it to data warehouse)
• Mojolicious provides web interface which uses data from redis and provides JSONP REST interface for Angular.js

For a start, you are probably running rpcbind instead of portmap on recent Debian installations. So you will need to modify flags which are passed to portmap on startup:

root@rsync1:~# cat /etc/default/rpcbind 
OPTIONS="-w -l -h 172.16.10.2 -h 192.168.0.219"

root@rsync1:~# grep rpcbind /etc/hosts.deny 
rpcbind: ALL

root@rsync1-dev:~# rpcinfo -p 192.168.0.219
rpcinfo: can't contact portmapper: RPC: Authentication error; why = Client credential too weak

root@rsync1:~# tail -1 /var/log/auth.log
May  8 20:31:51 rsync1 rpcbind: connect from 192.168.0.21 to dump(): request from unauthorized host

root@rsync1:~# grep rpcbind /etc/hosts.allow 
rpcbind: 192.168.0.21

root@rsync1:~# tcpdmatch rpcbind 192.168.0.21
client:   address  192.168.0.21
server:   process  rpcbind
access:   granted

I started simply by installing new munin instance in new (virtual) machine. I decided that historic data is important, so I decided to move configuration and graphs over to new instance. Simply coping rrd files over didn't went all that well and resulted in dreaded This RRD was created on another architecture error. This was unfortunate but rrd files where so large, that transfer won't fit into 5 minute munin poll interval anyway, so I had to take different approach.

To keep all historical data and not loose any munin polls while I transfer them I decided to first configure new munin node to poll all clients (so new updates will be preserved) and while this is running copy over rrd files from old server. This involved editing all nodes (9 of them!) and Cluster SSH came as perfect solution to add additional allow ^192\.168\.0\.10$ lines in /etc/munin/munin-node.conf on all nodes.

Coping rrd files had to be done using rrdtool dump/restore and it had to be done over compressed ssh link due to slow DSL line. For that, small shell script came very handy:

#!/bin/sh -x

dir=/var/lib/munin/maxxo


via_ssh="ssh -o ControlPath=/tmp/ssh_sock -o ControlMaster=auto -o Compression=yes root@10.1.3.10"

if [ ! -e /tmp/ssh_sock ] ; then
        $via_ssh read
fi

ls $dir/*.rrd | while read file
do
        echo $file
        rrdtool dump $file | $via_ssh rrdtool restore - $file --force-overwrite
done

You might think that it's all, but unfortunately, it's not. Looking in /var/log/munin/munin-update.log I could see [FATAL] Socket read timed out to node. Terminating process.. Some of nodes required more time than default value provided by munin (30 sec) to respond with all data. It seems that ipmi plugins are notoriously snow to respond for example. To change server-side timeout, you have to pass --timeout 90 to munin-update utility. Unfortunately, in Debian you can't do that by modifying munin-cron invocation in /etc/cron.d/munin because it passes all parameters to munin-limit which doesn't have timeout option and dies on you (moral of the story: check cron e-mail while configuring cron jobs). In the end, I edited /usr/bin/munin-cron directly, changing one line:

/usr/share/munin/munin-update --timeout 90 $@ || exit 1

There where also a few tweaks on munin-node plugins to make them work inside kvm. iostat_ios plugin from munin-plugins-extra didn't like virtio devices which have major number 254, same as LVM2 devices which it ignores. Following patch solved this problem:

diff --git a/cs-munin/etc/munin/plugins/iostat_ios b/cs-munin/etc/munin/plugins/iostat_ios
index 1380eb1..823df63 100755
--- a/cs-munin/etc/munin/plugins/iostat_ios
+++ b/cs-munin/etc/munin/plugins/iostat_ios
@@ -101,7 +101,7 @@ sub filter {
         return 0 if ($major ==   1); # RAM devices
         return 0 if ($major ==   9); # MD devices
         return 0 if ($major ==  58); # LVM devices
-        return 0 if ($major == 254); # LVM2 devices
+        return 1 if ($major == 254); # LVM2 devices and KVM virtio
     }
     if(defined($tmpnam)) {
         return 0 if ($tmpnam =~ /part\d+$/);

I also decided to use ksm which is enabled by following line in /etc/rc.local:

echo 1 > /sys/kernel/mm/ksm/run

To track other kvm parameters, I used munin-libvirt-plugins which comes with it's own helper script munin-libvirt-plugins-detect which you have to run to enable plugin and generate configuration.

For a web server, I opted to use apache2 and libapache2-mod-fastcgi together with graph_strategy cgi and html_strategy cgi in /etc/munin/munin.conf mostly to save some performance on polling machine. To make it work, I had to copy /etc/munin/apache.conf into /etc/apache2/conf.d/munin and uncomment relevant fast-cgi lines inside. After that, dynamically generated html is available at http://munin.example.com/munin-cgi/ and if you ever run munin-html before, you will still get old (obsolete) html pages if you visit page directly.

Next step would probably be to get rrdcached up and running...

Update: If you 404 or 500 errors when generating cgi graphs, and you will have to add cgitmpdir /tmp/munin/ to /etc/munin/munin.conf.

Fortunately, we have alternative pdf reader for Kindle based on muPDF library which is mostly written in lua. This is especially nice since it allowed me to add support for reading two-column pdf layouts which you can see in following video:

Basically, you press F to switch to new layout and then use fiveway buttons to move down the column, or right to move to top of next column, with page change if needed. If you move to left, you will be positioned to bottom of previous column which is useful if you want to read again last thing.

If you like this feature, liberate your kindle and download latest version of pdf reader, drop it in /mnt/us/customupdates and press Shift Shift I to install it from louchpad. Then press Shift P D to start it. Wiki pages of project describe all available shortcuts and there is active thread on mobileread forum.

At first sight, this seems like an easy task. Zebra printers are supported under CUPS on Linux and OSX, so there shouldn't be any problems, right? For a start I found out that CUPS driver doesn't work well, mostly because it's older that Windows version of driver, and doesn't seem to send all ZPL codes required to print label.
To make thing even worse, since printer is connected locally to Windows machine, it presents itself as Windows GDI printer which doesn't want to print ZPL (printer protocol) directly without wrapping ZPL in magic quotes and enabling it in Windows.

On the other side, Koha tries to print labels using normal print dialog in Windows. This won't work well, because we (again) need pixel exact label as opposed to web page randomly scaled to printer label. To make this worse, client Windows machines are behind firewall, so I can't send label to IP address of client because all I can see in request is IP address of our firewall.

To solve all this problem I decided to deploy following setup:

• Label design is done in Inkscape. To rasterize it to bitmap I decided to use rsvg-convert which has nice option to create exact bitmap size from SVG file in which I replace placeholders with variable values (bar-code and call number)
• Printers are installed in Windows using Generic/Text driver which is only one which will just pass data directly to printer
• To share printer to Linux, I decided to install lpd server so I don't have to open local Windows accounts to access printer (and they are behind firewall, so It's safe)
• To get local IP address of client (so I can send ZPL to internal IP address on which lpd is listening) I decided to redirect client to internal web server (behind firewall, same as clients) which knows client IP address, and knows where to send label.
• From user's perspective, Koha redirects clients to internal CGI script, which in turn rasterize label from request parameters, sends it to printer and redirect browser back to Koha page (with additional parameter of IP address of client/printer). This page shows label which was just printerd by pointing directly to internal server's PNG rendering of label. It also inserts data about printed label (including printer IP) in database for audit log.

Since I couldn't use CUPS to produce ZPL for printer, I wrote Printer-Zebra which can convert pbm and pnm formats (easily created from png label using pgntopnm). Even better, it also includes script which can render ZPL printer output back to pbm bitmap which is the only good way to verify that your solution doesn't anti-alias bar-codes or does something similar to reduce print quality on back and white printers. Rasterizer was also very useful when tracking differences between Windows driver output (gathered by printing to File on Windows) and CUPS one.

For a start, original Android distribution on Nook doesn't make it really Android tablet (no market). But there is easy solution since this device is supported by CynanogenMod 7 for Nook Color to get proper Android distribution. So, let's get started with hints for easier life with your new tablet...

adb doesn't see Nook Color

For a start, you won't see Nook in adb devices listing. This is easily fixed with proper device id in ~/.android/adb_usb.ini like this:

dpavlin@t61p:~/.android$ echo 0x2080 > ~/.android/adb_usb.ini && adb kill-server && adb devices
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
List of devices attached 
2011120012345678        device

Nook Color specific application

There are lot of application which will look quite broken on your Nook when running Android 2.3. Some of them require audio input, GPS or camera and they generally won't work. There is also problem with application which just don't know about wired 1024*600 screen size and produce strange small screen centered on big display.

However, this is just a list of few extremely useful apps which you should install on Nook with CM7:

• Nook Color Tweaks - change speed of CPU, turn USB host more and more
• Nook Screen Recalibrate - having problems with screen sensitivity?
• Nook Key Editor allows you to map back and menu keys which are very useful on Android 2.3 to volume up/down which makes tablet much more useful.

Ice Cream Sandwich

To get most out of Android on tablet, you have to hop over to CM9/ICS Nightly Builds and install unstable builds on ICS on Nook Color. After initial install reboot into recovery, flash Google Applications from gapps-ics-4.0.3-sam-noinit.zip, turn off signature verification and install fix-bootanimation.zip and telephony-permission-fix.zip. Unfortunately browser application doesn't work for me well, and usually segfaults with:

F/libc    ( 2878): Fatal signal 11 (SIGSEGV) at 0x0000001f (code=1)
I/DEBUG   ( 1072): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG   ( 1072): Build fingerprint: 'bn/bn_encore/encore:2.3.4/IML74K/228551:user/release-keys'
I/DEBUG   ( 1072): pid: 2878, tid: 2895  >>> com.android.browser <<<

ICS on Nook currently doesn't have hardware acceleration (but older Androids also doesn't have it) and it freezes from time to time (so you will get quite used to holding power button for 8 seconds or more to turn it off), but ICS is really so much better experience on tablet that I would really recommend it.

Finally a note about hardware performance: Nook Color has 512Mb of RAM and Ti OMAP 3621 @ 800 MHz which makes it probably a bit slow for ICS. As o consequence I wouldn't really recommend any Android laptops which are slower than this for ICS.

Update 2012-02-07: There is also CyanogenMod Kang build with OpenGL enabled which works very well. In fact, so well that it fixes problem in Hacker's keyboard :-)

Each year, we get a treat right before end of the year in form of 28C3. This year, I can recommend following lectures as a must-see:

• r0ket++ video
  Very nice ARM Cortex M3 embedded device with Nokia LCD, battery, RF interface and extension headers.
• 802.11 Packets in Packets video
  Drop wifi packet on unsuspecting listeners without access to network segment!
• Defending mobile phones video
  Visit GSMmap.org to see if your GSM networks are protecting you or contribute data!
• Reverse Engineering USB Devices video
  Very similar to my approach to USB examinations.
• How governments have tried to block Tor video
  It's arms race all over again, this time on Internet.
• Reverse-engineering a Qualcomm baseband video
  I can't wait to see release of this tools on code.google.com Available at http://code.google.com/p/qcombbdbg/
• KinectFusion video
  I would love to see something similar in open source code.
• Cellular protocol stacks for Internet video
  Very good overview of cellular stack as it is today
• Implementation of MITM Attack on HDCP-Secured Links video
  NeTv is very interesting ARM/FPGA/Linux device which overlays webkit content over HDMI signal on your TV using chroma key.
• Introducing Osmo-GMR video
  Osmocom got some support for satellite phones
• bup: Git for backups video
  Nice backup system with de-duplication implemented on top of git's pack file

We had another gathering called Nothing will happen in Moravice, and this time we wanted to be prepared for it. So we got 2 WRT and 2 TP-Link devices with good faith that we will have stable Internet there. However, our telecom provider decided to screw up our order to increase bandwidth from 4Mbit/s to 10Mbit/s for event and instead decided to downgrade our access to 512Kbit/s. So we opted to create following simple network architecture which involved multiple hops to alternative 4Mbit/s upstream and batman-adv mesh setup.

batman-adv is quite cool layer-2 mash network which operates in ad-hoc mode and allows adaptive routing over mash and multiple upstream providers (at DHCP request or renew time, so we made our DHCP lease time down to 5 minutes). Joining network is quite simple:

iwconfig wlan0 mode ad-hoc essid nsnd-batman
ifconfig wlan0 mtu 1528
modprobe batman_adv
batctl if add wlan0

ifconfig wlan0 up
ifconfig bat0 up
dhclient bat0

My own experience with this experiment is very positive, mostly because we had around 5 clients in mash at any time, compared with overloaded WRTs which handled rest of 40 wifi enabled devices. But, I hope that during next event more people will upgrade to 2.6.33 or newer kernels, so we can have even better mash connectivity.