Dobrica Pavlinušić's Weblog / Blog

Last week we had another annual conference about Free Software and Open Source DORS/CLUC 2013. For a last year, I was playing with various hardware, so this year, I was part of Internet of things panel (talking about ARM based machines), and I had lecture about GNU/Linux on ARM devices for $50-$100.

I also submitted hardware workshop for program which got accepted, so I quickly realized that I'm really no hardware expert and that I could use some help to make interesting workshop. Fortunately I have a few good friends who know more about hardware that I ever will, so I summoned Lovro and Dalibor to help me cover hardware and antenna design. Few weeks ago I was at NSND Belgrade 2013 where I had good fortune to meet Filip who is working for Dangerous Prototypes. I couldn't really believe my good luck since I wanted to talk about Bus Pirate a great multi-purpose tool which got me into hardware in the first place. So, in the end, I had three very skillful people to back me up in this workshop which was hopefully useful and interesting to people attending it. For future reference, I will include a few links below about topics we convered:

• Filip started with Bus Pirate including it's supported protocols I2C, SPI, 1-wire, UART, JTAG (and others) and example of using it to communicate with I2C EEPROM
• Dalibor make great presentation about antenna design, including practical examples for ADS-B and 443.92 antenna (simple one from coax cable and one with RF filter for PiHAT) while I tried to explain why RTL-SDR DVB-T sticks made it possible to listen to various your IMS sensors using rtl_443.
• Next, Filip again took stage to speak about Open Hardware and possibilities to manufacture and sell such devices based on expirience of Dangerous Prototypes cooperation with Seeed Studio.
• In last part of our workshop we played with blinkm, took questions and discussed other Dangerous Prototypes projects which we had with us which included USB Infrared Toy, Bus Blaster and Open Bench Logic Sniffer but we also talked and showed Arduino and Mojo FPGA board which we had us also to ZPUino which is interesting combination of Papilio FPGA and Arduino IDE. Lovro did a nice overview of FPGAs and how are they different from software programming.

As you can see from notes above, workshop was a mix of different projects but hopefully it managed to convey my excitement about current moment in time where you can hack hardware even without taking soldering iron (and burning your fingers). If you do take soldering iron, please make your project Open Source Hardware...

I'm in the process of setting up my HTPC, so I ordered USB IR Toy to be able to turn TV on without using remote control.

When I got it, I was unable to make it work. Fortunately, there is V command which returns current version of firmware, and on my device it was V212 which means Hardware V2 and firmware V12. Since latest firmware is V22, I decided to upgrade firmware. In the process I bricked it (using wrong fw_update program which doesn't flash whole device, but just first 2K of application). Fortunately, this PIC has Diolan bootloader which is accessible using jumper (small blue peace of wire on picture) even if you flashed just part of application on it (as I did).

Result it fw_update for PIC 18F2550 which IR Toy uses , updated wiki page about IR Toy firmware update on Linux and comment on bug tracker. Hopefully, this will help someone else instead of learning all this stuff, which was fun in itself :-)

Next step was to try SUMP logic analyzer mode which can produce nice graph of IR communication. I used Debian sump-logicanalyzer package to produce following picture:

Now that I was sure that it works and can communicate with my LG TV remote, I decided to give one of sample programs, IRToyRecPlay a try. It doesn't come with Makefile, so I compiled it using:

gcc *.c -o IRToy

Unfortunatly, it doesn't seem to work for me (can you see pattern here?). So, I decided to try fork of IRToyRecPlay on github which works better under Linux (after PIC fw_update problems I didn't really spent any time investigating why - I suspect that upstream Dangerous Prototypes source works well under Windows).

First, I recorded power button from remote using

./IRToy -d /dev/ttyACM0 -f lgtv -r

This created lgtv_000.bin file which I could reply to get TV turn on:

./IRToy -d /dev/ttyACM0 -f lgtv -p

I noticed that I can't just reply same sequence again to turn TV off, so more works needs to be done... But that's only software problem, so I did my first clash with PIC hardware somewhat well :-)

For a long time, doing anything with various devices which use radio waves proved to be almost impossible for software geeks like me. Fortunately with RTL DVB-T sticks all this is about to change, and at our last meeting Ništa se neće dogoditi in Osijek I tried to show some of possibilities using them.

For a start you need compatible Realtek RTL2832U DVB-T tuner. Just go over to rtl-sdr osmocomSDR and examine supported tuner list. Choice of DVB-T sticks in Croatia isn't so good. They are about 170 kn and come with FC001[23] tuners: Hama 53176 has FC0013 tuner and TV LV5TDLX has FC0012 tuner. I did some research and ordered this $13 DVB-T from dealextreme which has R820T tuner:

dpavlin@blue:~$ sudo rtl_test -t
Using device 0: ezcap USB 2.0 DVB-T/DAB/FM dongle
Found Rafael Micro R820T tuner

It's a good choice, shipping is free (but takes few weeks), and it works well with ADS-B which was first interesting application which I wanted to try.
I also ordered E4000+2832U USB DVB-T TV Receiver Stick for $16.50 but it hasn't arrived yet so I don't have any experience with it.

I also have a good fortune that my friend is very skillful in radio communications, so he created variant of Coaxial Collinear Antenna for ADS-B Receiver. This one is more sensitive to static electricity which means it's not suitable for top of skyscraper scenarios, but in works very nicely. Under construction is filter which would cut everything expect interesting part of RF spectrum which would also solve problem of static electricity.

On the picture, you can see part of it together with output in browser created using dump1090. It's important to note that normal DVB-T antena without any modification will work with R820T tuner since it's most sensitive so far, but range will be much smaller.

We also had good luck with getting data about keypresses from power-socket controlling device using rtl_433 from github with just a peace of wire as antenna for short range tests.

We have all been in this situation. We need quick access to hardware console of machine to debug nasty hang (kernel oops in my example). But now-days, we are all in the cloud, and that machine is libvirt's kvm. I just need access to VNC console, and installing whole virt-manager seems like overkill. So, let's see if we can make it somewhat easier and quicker.

First ssh into your host machine (garak in this example, tunneling some VNC ports along the way:

dpavlin@x200:~$ ssh $(seq 5900 5904 | awk '{ print "-L "$1":localhost:"$1 }') garak

If you have more than 4 vms running on this machine, increase 5904 above.

Then, using virtsh list all domains, and examine vnc port:

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     stargate                       running
 2     treca                          running
 15    hroug                          running
 16    close                          running

virsh # vncdisplay close
:2

Now, back on laptop, in another terminal start vnc viewer:

dpavlin@x200:~$ vncviewer :2

Since close didn't respond to reboot, reset or shutdown commands in virsh, I opted to kill kvm process and start it again with start close. Quick alternative to virt-manager installation.

I recently got big screen TV (big for my living room at least). It came with few HDMI ports and VGA, so next logical step was to connect computer to it. And of course, then I noticed that it would be really handy to have wireless keyboard and mouse to complete this nice setup. However, I also wanted to ssh over that networks, so I started examining how secure wireless keyboards are. tl;dr; summary: they are not secure.

First I asked for suggestions which wireless keyboard to buy. I have quite big fingers, so mini models just doesn't do it for me. I got suggestion to take a look at Logitech K400 and sure enough it seemed like good choice. One of first things that I noticed is that it supports 128-bit AES encryption. I started to have a good feeling about it, but I wanted to know more, so I hoped to Logitech Advanced 2.4 GHz Technology pdf and discovered that not all keys are encrypted.To quote documentation:

The encryption applies to all standard keyboard keys (a, s, d, f...) and modifiers (Shift, Ctrl, Alt...). The multimedia keys (Play, Pause, Mute...) that may be implemented in some keyboards are transmitted in clear text.

How can I trust keyboard which doesn't encrypt all traffic? This got me thinking. Can I somehow verify that keys are encrypted? Is this wide-spread problem? Can I make mistake and broadcast my keystrokes to whole world?

Sure I can. For older 27Mhz keyboards there is KeyKeriki v1.0 - 27MHz project which implement sniffer for it (video DeepSec 2009: Keykeriki: Universal Wireless Keyboard Sniffing For The Masses). But, Logitech is 2.4Ghz, so it's secure, right? Well, there is KeyKeriki v2.0 - 2.4GHz which does same for 2.4Ghz (video Keykeriki V2 - Practical Exploitation of Modern Wireless Devices [SIGINT10]). OK, Logitech does some kind of AES on top of that, but since it does transfer some keys unencrypted, and it's proprietary technology I can't really check that.

I also got suggestion to use bluetooth keyboard because it's secure. Well, quick search revealed Ubertooth One which basically defeats bluetooth protection with a bit of sniffing and a little brute force.

By this point, I was puzzled. Is there secure wireless keyboard with touchpad which I can buy? Something I can be sure that it encrypts all traffic as opposed to only some keys? Or is usb extension cable only real solution for me?

Allmost a year ago, me and three other friends decided it's a very good idea to support Printrbot kickstarter project and get our-selfs 3D printer. We didn't have any particular use for it (other than printing Raspberry Pi case when it arrives) but it seemed like the right thing to do. This post will try to explain how far did we manage to get with it and why we where wrong.

If you examine original Kickstarter page you will see following description:

A desktop 3D printer you can build in a couple hours. Print plastic parts you design or download - even parts for another printer.

Our experience can't be further from that statement. For a start, Brook Drumm (to whom I'm ethereally grateful for his efforts to make 3D printers commonplace) got his campaign funded with 1,808 backers who spent $830,827 instead of just $25,000 goal he envisioned. This was both good and bad. Good part was that after funding we knew that we will have 3D printer (I'm carefully not mentioning printing anything), but the bad part was logistics: there was simply no way he would be able to print 1808 versions of original design on 3D printers themselves (idea of RapRap-like printers, which Printrbot was one iteration, was always to make them self-replicating). So, he decided to change design and move toward wooden laster-cut parts for most of construction, and print just parts which where necessary.

This also introduced significant delay in printer shipment, but when you are funding Kickstarter project, you should be prepared for it, so I'm not complaining. When it finally arrived this summer (10 months after end of Kickstarter campaign), it was significantly bigger than I expected:

To be honest, we did upgrade to bigger Printrbot PLUS so I guess we should expect a lot of parts. As we are mostly software geeks, we did only reasonable thing to do: check if all parts are present comparing it with bill of materials which we got printed out.

This is the point where our problems started. We had missing one bag of parts which included termistor and switches. We contacted Printrbot HQ and they sent us missing parts. We started assembling following Printrbot Building Instructions by Brook Drumm and it took us more than 50 hours to get to our first blob.

Of course, it didn't work perfectly on first run. We where trying to print 5mm Calibration Cube Steps is ABS plastic which we received with our Printrbot (we even got additional 450g of ABS plastic as replacement for power supply which wasn't part of international shipments).

Actually, it still doesn't work well as you can see in video below, but we are hopeful. In the meantime we figure out that best source of information is Printrbot Talk forum and wiki. Forum is somewhat depressive since most users have some kind of problems with their built, just as we do.

To be honest, we didn't expect smooth ride. However, as I mentioned before we are not really hardware hackers, and my only conclusion is that home-made 3D printers are really for people who already have enough experience to make their own 3D printer, and not for software guys like us. However, we won't give up, and I fully expect to have working printer (after we get replacement barrings from Printrbot HQ because our are sticky). We are collecting useful add-on models and instructions on our Printrbot wiki page but I didn't expect that we will have to contact Printrbot HQ twice for missing and replacement parts. But eventually we will be able to print Raspberry Pi box, I hope :-)

We had software RAID in form of md devices on Linux for a long time. Here are few useful hints how to make it work better...

Scrub your array to check data consistency

I will assume that you are already monitoring your disks using smart, but from time to time it's useful to force full re-scan of your array to make sure that all data is still there and consistent. Some filesystems provide this option to scrub data on it's own (zfs and btrfs comes to mind) but if your filesystem is located on md array you can always force it using

echo check > /sys/block/md0/md/sync_action

I would suggest to do this from cron, hopefully during weekend or some other time when your load is lower.

Adding write-intent bitmap to speed up recovery

If you installed your md array a long time ago, you probably didn't turn on write-intent bitmap. It's very useful when you have to recover because bitmap will track changes and this will prevent long re-sync times when disks have to read and compare every block. To turn it on use:

mdadm --grow --bitmap=internal /dev/md0

Mirror between two devices of same speed

Recently, one of my 500Gb disks in RAID1 (mirror) failed. I decided to replace it with 1Gb drive which was unfortunately green drive (which basically means slow). Adding two drives of different speed in mirror will reduce performance to single slower drive which is a shame. Since I wasn't able to add additional disk and wasn't prepared to give up redundancy of data I started searching around and found that I can specify one disk as write-mostly using:

mdadm --add /dev/md0 --write-mostly /dev/sdb1

Same trick will work on combination of hard drive and SSD, but in that case, you will slow down writes to speed of your hard drive.

I have been toying around with idea of having real Linux stack (X11 and friends) on Nook Color. While this seems like a silly thing to do, it does allow me to use x2x and transfer keyboard and mouse from my laptop to tablet which is handy. If also allows me to run X11 applications on tablet screen using DISPLAY=nook.lan:0. I was fortunate enough to find blog post how to run Linux framebuffer X server on Android but I wanted to use touchscreen so I decided to write my own xorg.conf (this brings back memories...).

To get full-blown Debian-based distribution on your Android take a look at BotBrew Basil. It's Emdebian based distribution which will setup mount points and various other stuff so you don't have to do that manually. Since it's Debian based, you are not limited to Emdebian packages -- you can (and will have to) add normal sid:

(BotBrew)root@localhost:/# cat /etc/apt/sources.list.d/sid.list 
deb [arch=armel] http://ftp.debian.org/debian sid main contrib non-free

If you want to know more about Emdebian hop over to DebConf 12: Integrating Emdebian into Debian [video].

With all this prepared, we are ready to shut down Android stack:

adb shell setprop ctl.stop media
adb shell setprop ctl.stop zygote
adb shell setprop ctl.stop surfaceflinger
adb shell setprop ctl.stop drm

Next step is installation of required packages:

dpavlin@t61p:~$ adb shell
root@android:/ # TERM=xterm chroot /data/botbrew-basil/ /bin/bash --login
(BotBrew)root@localhost:/# apt-get install xserver-xorg-video-fbdev xserver-xorg-input-evdev \
   xserver-xorg-input-multitouch x11-xserver-utils xinit \
   matchbox matchbox-keyboard xterm

I decided to use matchbox, mostly becuase it's only window manager which comes with on-screen keyboard which is useful on touch screen device.

After installation you will need to setup X symlink and create .xinitrc:

root@android:/ # ln -s /usr/bin/Xorg /usr/bin/X

root@android:/ # cat ~/.xinitrc                                              
( sleep 1 ; matchbox-keyboard -o portrait ) &
xhost 192.168.1.61
matchbox-session

Finally, you need to create xorg.conf:

Section "ServerLayout"
    Identifier    "Layout0"
    Screen        "Screen0"
    InputDevice   "cyttsp-i2c" "CorePointer"
    InputDevice   "gpio-keys" "CoreKeyboard"
    InputDevice   "twl4030-keypad" "CoreKeyboard"
EndSection

Section "InputDevice"
    Identifier    "gpio-keys"
    Driver        "evdev"
    Option        "Device" "/dev/input/event0"
    # code 102 (KEY_HOME)
    # code 116 (KEY_POWER)
EndSection

Section "InputDevice"
    Identifier     "twl4030-keypad"
    Driver         "evdev"
    Option         "Device" "/dev/input/event1"
    # code 114 (KEY_VOLUMEDOWN)
    # code 115 (KEY_VOLUMEUP)
EndSection

Section "InputDevice"
    Identifier     "cyttsp-i2c"
    Driver         "multitouch"
    Option         "Device" "/dev/input/event2"
    # mouse should move as fast as finger and not faster
    Option         "AccelerationScheme" "none"
    # evdev has it, multitouch doesn't so it behaves like touchpad
#   Option         "IgnoreRelativeAxes" "True"
EndSection

Section "Device"
    Identifier    "Card0"
    Driver        "fbdev"
    Option        "fbdev" "/dev/graphics/fb0"
    # rotate screen to be in sync with touchpad orientation
    Option        "Rotate" "CCW" # CW=90 UD=180 CCW=270
EndSection

Section "Screen"
    Identifier    "Screen0"
    Device        "Card0"
EndSection

This will map all hardware keys and use mutitouch driver for screen. To make it work, I used evtest package which allows you to see events from input devices so you will know which device produce keyboard events and which produce multitouch events. To be honest, this solution isn't prefect, because screen behaves like touchpad, so you can't just point to screen and expect your cursor to just to that position.

Following video shows X server in action.

This is simple unaccelerated frame buffer. This makes performance less then desirable. There are a few implementations of OMAP xorg server:

• xf86-video-omapfb uses DSS kernel support which seems to be part of CM kernel, so this might be good next thing to try out
• xf86-video-omap is newer implementation, but this requires 3.3 kernel and is not yet stable.

Having accelerated OMAP X server and fixed touchscreen issues would make Nook somewhat nice Linux tablet, if only it isn't so heavy for day-to-day use :-)

Last year, I had good fortune to get acquiented with great work which Open Library does. It's part of Internet Archive which itself is a library. So, libraries are not (yet) dead it seems. Brewster Kahle's Long Now Talk explains it much better than I can do, so take 90 minutes to listen to it.

Most interesting part of Open Library (IMHO) is Internet Archive BookReader which is JavaScript application which allows users to browse scanned books on-line. For quite some time, I wanted to install something similar to provide web access to our collection of scanned documents. I have found instructions for serving IA like books from own cluster, but I didn't have a cluster, and converting all documents to IA book format seemed like an overhead which I would like to avoid.

Instead, I decided to write image server for JavaScript front-end using plack. I mean, it's basically a directory with images, right? Oh, how wrong can I be? :-)

It turs out that we have pictures in multiple formats (so sorting them required removing common prefix and using number only to get correct order), and most of are scanned images in pdf documents. Here are all types of documents which can be automatically collected into book for on-line browsing:

• images of scanned pages
• multi-file pdf file with single image per page
• single pdf file with one image for each page
• single pdf file with more than one (usually 4) horizontal bitmap strips for each page
• normal pdf documents which contain text and needs rendering to bitmap

Source code of my plack server for Internet Archive book reader is on github, so if you want to take a look, hop over there...

As you all know by now, last week we had another DORS/CLUC conference. This time I had two talks and one workshop.

Sysadmin cookbook

I never proparly introduced this project here, but if you want to know more about my convention based documentation examine the presentation of hop over to http://sysadmin-cookbook.rot13.org/ and take a look at generated documentation.

Basic idea is to document changes in easy to write files on file system (preserving symlinks to files on system which allows you to quickly see if cookbook is deployed or not and diff between template and deployed configuration). I know that my cookbook is mix of various things I did in last three years, but I do find it useful, so hopefully it might be useful to you also.

Kindle - so much more than ebook reader

This was longer talk about my one year experience with Kindle. I must say that I'm still very happy user of Kindle, but in this talk, I tried to cover Kindle Developer's Corner at mobileread forum as well as other related projects:

• Free Libre command-line tools for Kindle development
• A PDF (plus DJVU, ePub, TXT, CHM, FB2, HTML...) viewer made for e-ink framebuffer devices, using muPDF, djvulibre, crengine mobileread forum thread
• A VNC viewer for ebook readers which allows you to use Kindle as second screen mobileread forum thread

So if you are still wondering if it's worth the effort to install third-party software on Kindle, answer is yes, it is!.

Web scale monitoring

This was a workshop which doesn't have much with web (it's about monitoring ADSL CPE devices and provider equipment in-between), but it shows (I hope) nice way to integrate several project to provide nice scalable monitoring infrastructure. It's composed of:

• Gearman message queue together with Gearman::Driver provide on-demand scaling of workers
• redis saves all data from external systems (LDAP, CRM) and all results from statistics collection nicely providing data for web interface
• PostgreSQL stores all collected data, using hstore to provide unstructured key value store for different data from different devices while still allowing us to use SQL to query data (and export it to data warehouse)
• Mojolicious provides web interface which uses data from redis and provides JSONP REST interface for Angular.js

All in all it's a nice combination of tools which served my need quite well, so I hope it was also useful to people who attended workshop.