Personal weblog about technology, Open Source, perl, GNU etc...
Few weeks ago, we had our annual conference DORS/CLUC 2015 on which I had interesting (hopefully) presentation about AVR component tester. Since then, we got video recording of conference, so below you can find embedded presentation and video recording (in Croatian).
We have been running ganeti cluster in our institution for more than a year now. We did two cycles of machine upgrades during that time, and so far we where very pleased with ability of this cloud platform. However, last week we had a problem with our instances -- two of them got owned and started generating DoS service attack to external resources. From our side it seemed at first like our upstream link is over saturated, and we needed to way to figure out why it is.
At first, it seemed like this would be easy to do. Using dstat,i I found that we are generating over 3 Gb/s traffic every few seconds to outside world. We have 1 Gb/s upstream link, but our bonded interfaces on ganeti nodes can handle 3 Gb/s of traffic, so for a start we where saturating our own link.
But which instance did that? I had to run dstat on every node in our cluster until I found two nodes which had instances which where overloading our link. Using iftop I was able to get hostname and IP address of instances which I wanted to shut down. However, this is where problems started. We didn't have DNS entries for them, and although I had IP and mac address of instances I didn't had easy way to figure our which instance has that mac.
Than I figured out that I can get mac from kvm itself, using ps. Once I found instances it was easy to stop then and examine what happened with them.
But, this got me thinking. Every time I have a troubleshooting problem with ganeti, I basically use more or less same command-line tools to figure out what is going on. But I didn't have a tool which would display me some basic stats about instance, but including mac addresses and network traffic (which in our configuration are tap devices added to bridges). So I wrote, gnt-info which presents nice overview of your instances in ganeti cluster which you can grep to drill-down into particular instance or host.
We all read hackaday, and when I read Five Dollar RF Controlled Light Sockets post I decided that I have to buy some. However, if you read comments on original Cheap Arduino Controlled Light Sockets - Reverse Engineering RF post and especially comments, you will soon figure out that ordering same looking product from China might bring you something similar but with different internals.
In my case, all four light sockets turn on or off with any button press on remote which was a shame. When I opened remote and socket, I also had bad surprise. My version didn't have any SPI eeprom, but just two chips, ST F081 FB 445 in remote and ST ED08 AFB422 in light bulb (in picture hidden below receiver board).
But, I already had acquired two sets so I wanted to see what I can do with them. Since I couldn't read eeprom to figure out code, I decided to use rtl-sdr to sniff radio signals and try to command them using cheap 315 MHz Arduino module.
I used gqrx to sniff radio signals and I was not pleased. Remote drifted all over the place mostly around 316 MHz and it was some trial and error to capture signals which are generated when buttons are pressed. However, I have verified that it's sending same signal multiple times no matter which keys I press (which would explain why four pins on remote are soldered together).
After a while I had two traces (since I have two sets of light sockets) and could decode binary data which is sent from following picture:
How I knew that one set is transmitting 1000100110110000000000010 and another one 1011001001011111000000010. From looking into timing in audacity, it seemed that each bit is encoded in short-long or long-short sequence where short one is about third of long one, and one bit is about 1200 ms. I cheated here a little and stuck scope into scope into transmit trace on remote to verify length of pulses just to be sure.
So as next step I wrote simple Arduino sketch to try it out:
#define TX_PIN 7
#define LED_PIN 13
char *code = "1000100110110000000000010";
//char *code = "1011001001011111000000010";
void setup() {
pinMode(LED_PIN, OUTPUT);
pinMode(TX_PIN, OUTPUT);
}
void loop() {
digitalWrite(LED_PIN, HIGH);
for(int i = 0; i < strlen(code); i++) {
int i1 = 300;
int i2 = 900;
if (code[i] == '1' ) {
i1 = 900;
i2 = 300;
}
digitalWrite(TX_PIN, HIGH);
delayMicroseconds(i1);
digitalWrite(TX_PIN, LOW);
delayMicroseconds(i2);
}
digitalWrite(LED_PIN, LOW);
delay(3000);
}
So, I compiled it, uploaded to Arduino and... nothing happens.
Back to the drawing board, I guess.
When I was looking into gqrx I could see that signal is sent as long as I'm holding button up to 10 seconds. From experience before I know that this cheap receivers need some tome to tune into frequency so next logical step was to send same signal multiple times. And guess what, when I sent same singal twice with 2000 ms delay between them everything started to work.
Again somewhat. Light socket in far corner of hall seemed to have problems receiving signal which would put two light socket in hall in opposite state: one would be on and another would be off. This was fun, and could be fixed with simple antenna on Arduino module (since currently I don't have any) but I will conclude that your IoT device should send different codes for on and off state so something like this won't happen to you.
Then I got carried away and added commands to change all parameters to experiment how sensitive receiver is. You can find full code at https://git.rot13.org/?p=Arduino;a=blob;f=light_sockets/light_sockets.ino With this experiments I found out that you don't have to be precise with timings (so my oscilloscope step was really not needed). Receiver works with 500 ms low and 1100 ms high (for total of 1600 ms per bit) on high end, down to 200 ms for low and 800 ms for high (for total of 1000 ms per bit).
I suspect that chips are some kind of 26 bit remote encoders/decoders but I can't find any trace of datasheet on Internet. This is a shame, because I suspect that it's possible to program light sockets to respond to any code and in theory address each of them individually (which was my goal in beginning). However poor construction quality, and same code for on and off state (combined with poor reception) makes me wonder if this project is worth additional time.
I must confess that I'm pack rat. When I see old printer, something inside my head tries to figure out what I can do with all parts inside it instead of passing it to land-fill. However, I'm sysadmin and software guy, so JTAGs and programming is more up my wally than hardware. However, I decided to figure out how to drive one of servos using Arduino and this is my journey through this experience.
So I started with printer disassembly and got one stepper motor and some gears on it. It is Mitsumi M42SP-6TE. It has four wires and I couldn't find any data sheet about it. So what do I do now?
First some educated guesses.I assumed that it's 12V servo. This was somewhat influenced by examining similar Mitsumi MP42SP-6NK motor which have rating of 12V or 24V. Using unimer and taking ohm measurement between wires I confirmed that it has 10 Ω between coils which means it's bipolar, having two different coils which had both to be driven at the same time.
To connect it to Arduino, I acquired some time ago clone of Adafruit motor shield. When you buy cheap clones you expect some problems, and mine was fact that screw terminals on board weren't cut flash with board, so I had to use flat cutters and shorten them to prevent motor power from shorting with ICSP header on Arduino and USB connector on Uno. I also used red electrical tape and put it on USB connector just to be safe(r).
I also needed to add power jumper (white on picture) to provide power from Arduino (which in turn is powered by 12V 1A adapter). However, in this configuration L293D H-bridge becomes very hot to touch, so for testing I modified StepperTest example to provide me with serial control and powered Arduino from USB port (from which it draws 0.42 A and stepper still works with 5V supply which makes my 12 V assumption somewhat questionable). This enabled me to deduce that this stepper is also 7.5° which takes 48 steps to do full turn (small red dot on stepper gear helped to verify this). I also verified that top gear has 13:1 ratio to stepper motor making gear mechanism useful for smaller movements and better tork.
I hope this blog post will motive you to take old printers, scanners, faxes and similar devices apart and take useful parts out if it. Re-using boards for driving steppers is also very interesting, but this particular printer didn't come with power supply (and it has strange connector) and driver chip on it doesn't have any publicly available info, so this will have to wait some other printer which will decide to give up it's parts for my next project...
Last week I had pleasure of attending FSec 2014, annual security conference. Just like last year, I had hardware presentation, this time about reverse engineering NComputing CPLD dongle. You can find it on http://bit.ly/fsec2014-jtag or embedded below.
I had great time at conference, but I'm somewhat wondering did audience got something from my lecture. It was very interesting for me to figure out JTAG pinout on this board, and connect it to various JTAG programmers (all with their's good and bad sides) and I noticed that there are not any introductory text on the web how to approach this problem for the first time. So, I decided to present this topic in hope that this will motivate other people to take a hack at some board which would otherwise end up on e-waste of even worse, land-fill. And, who can resist call of free hardware which you can re-purpose? :-)
I have been using Hantek DSO-2090 USB oscilloscope for more than half a year now. While scope purist will say that usb oscilloscopes are not good enough for serious use for my use it's quite sufficient. However, this weekend, I was reverse engineering CPLD with R2R digital to analog converter, and I needed to figure out which steps are produced by turning pins on CPLD on or off. Sure, I can use multi-meter to do this, but if I already have oscilloscope it's much more powerful tool for task like this.
When choosing USB oscilloscope, I searched a lot, and decided to buy Hantek DSO-2090 because it's supported by free software like OpenHantek and sigrok. There are better oscilloscopes out there, but this one is supported by free software, and there is even a detailed tear-down which explains how to increase it's performance. When scope arrived, I was quite pleased with OpenHantek, but never managed to get sigrok working with it. It didn't matter at the time, since OpenHantek had everything I needed. However, for this task at hand I really needed minimum and maximum voltage. As you can see in video describing oscilloscope usage, and especially Hantek DSO-2090, including it's limits.
OpenHantek shows just amplitude of signal, which is difference between minimal and maximal voltage but doesn't show raw values which I needed. So, I wrote simple patch to OpenHantek to display minimum, amplitude and maximum voltage as you can see in picture. I also wrote a message on mailing list with a patch, so I hope you might expect to see this change in next version of OpenHantek.
Every year, our three day annual DORS/CLUC 2014 conference is happening. This year, the dates shifted a few weeks later, which resulted in less students showing up because of exams, so it was a somewhat different experience than years before. For few years now we are not at the University of Zagreb, FER location so it also changed conference a bit. Having said that, even after move from FER, we still had a bus of students from my own faculty FOI in Varaždin, and they where missing this year.
It was still full conference in new (2nd floor, not ideal for breaks in fresh air which is a must to stay for 11 hours each day, mind you) location at Croatian Chamber of Economy new and nice conference hall with wifi which was stable but didn't allow UDP traffic. Both mosh and n2n didn't work for me.
It was also in very different format. I would love to know did it worked for people or not. Instead of charging for workshops, they where included in conference price, and as every year, it you where interested in topic, nobody will turn you away from workshop because of space :-) This also meant that workshops are three hours slots at the end of the day after 7 hours of lectures. When conference started, we where afraid how will we accommodate all that people at workshops, but sense prevailed and about 20 or so people stayed for workshop each day.
I had 5-minute lightning talk about Parallella, and hopefully managed to explain, that there is now interesting dual-core ARM, with interesting DSP-like capabilities backed by OpenCL and FPGA. This is unique combination of processing power, and it would be interesting to see which part of this machine can run OpenVPN encryption best for example, because it has 1Gbit/s ethernet interface.
ZFS on Linux had a 0.6.3 release just in time, and I presented two and half hour long workshop about ZFS for which 10-20 people stayed, after 7 ours of presentations. I somewhat field to show enough in command-line, I'm afraid, because I was typing too little. I did managed to show what will you get if you re-purpose several year old hardware for ZFS storage. Something along lines of 2004 year hardware with 8 SCSI disks.
I managed to create raid-10 like setup, but with all benefits of ZFS, fill it up and scrub it during workshop.
root@debian:/workshop# zfs list
NAME USED AVAIL REFER MOUNTPOINT
workshop 268G 28K 268G /workshop
workshop/test1 280K 28K 144K /workshop/test1
workshop/test1/sub1 136K 28K 136K /workshop/test1/sub1
root@debian:/workshop# zpool status
pool: workshop
state: ONLINE
scan: scrub repaired 0 in 0h44m with 0 errors on Tue Jun 17 17:30:38 2014
config:
NAME STATE READ WRITE CKSUM
workshop ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02KAT ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02KBB ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02KCK ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02KDD ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02L4S ONLINE 0 0 0
scsi-SFUJITSU_MAS3735NC_A107P4B02L4U ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
scsi-SFUJITSU_MAW3073NC_DAL3P6C04079 ONLINE 0 0 0
scsi-SFUJITSU_MAW3073NC_DAL3P6C040BM ONLINE 0 0 0
errors: No known data errors
I think it might be good idea to pxeboot this machine on demand (for long-term archival storage) and copy snapshots to it on weekly basis for example. Think of it as tape alternative (quite small, 300G) but with rather fast random IO. Idea was to use this setup for ganeti-backup target, but dump format of ext file-system forced us to use zfs volumes to restore backup on other RAIDZ1 4*1.5T SATA pool, and it was very slow.
Another interesting part of infrastructure work last year for me was with Luka Blašković. We migrated all servers from faculty and library to two Ganeti groups. We are running cluster of reasonable size (10+ nodes, 70+ instances). Everything we did is done from legacy hardware which is now much better utilized. Some machines where never backuped and firmware upgraded so it was first time for them to have this kind of maintenance in last 10 years. Now we can move VM instances to another machine, and we are much more confident that services will stay running via live migration for scheduled maintenance or restart in case of hardware failure.
For workshop, we decided to chew a bit more than we can swallow. We spun up KVM images on our ganeti cluster and went through installation of workshop ganeti on them and joining them to new cluster. This went fairly well, but then we started configuring xen to spawn new instances (ganeti kvm with ganeti xen on top of it) we had some problems with memory limits which we managed to fix before end of workshop.
In our defense, we really believe that workshop was more interesting this way, probably because people didn't want to leave (few brave ones which where with us all the way to the end, that is). When you try to deploy something as complex as Ganeti you will run into some problems, so seeing troubleshooting methods used is usually as helpful as solution itself.
All in all, it was interesting and very involved three days. Hope to see you all again next year.
After 18 months Parallella kickstarter project delivered and I got the device in my hands. To be honest, I was prepared to write off $100 for it, but decided to support the project because I believe that we should have alternative architectures developed and Epiphany had such a goal.
As you can see on the picture, I got parallella board and heatsink for FPGA in nice box together with the pack slip. Heatsink is recent addition because the FPGA get very hot. However, it's not enough because you will also need some air flow over it to ensure stable operation. And 5V 2A power supply. So, I decided to do some research before the first power-on because burning board on the first try is not a good option.
Here is where Parallella forums came in very handy. It's full of very supportive community, and to learn how to use your board it's better place than the official documentation (and more up-to-date). On it you can learn that there are jumpers on the board to provide 5V for fan, and various other hints about the platform including ability to power the board over USB connector which proved helpful since I could use a 2A Nexus power supply.
Official image for Parallella is based on Ubuntu (which I don't like much, it even doesn't move devtmpfs by default), so I opted to install the unsupported Debian installation and try to lower power usage by disabling HDMI support since I'm not using it. And thanks to helpful parallella community and the forum post about with alternative parallella bitstreams and device tree I was successful in that task lowering power draw to ~0.75 A in idle mode and ~0.86 A while testing with aobench from parallella-examples. CPU load alone (two arm cores) seem to consume ~0.81 A. For comparison, HDMI bitstream consumes ~1.03 A in idle and ~1.19 A under load. All values are maximal ones which I measured using USB charger doctor, so they might not be the most precise.
To cool the device, I have salvaged small fan from an old disk drawer and attached it to the board using zip ties.
Power is supplied from a USB port on PC (for now), but the next logical step is to connect it to the jumpers on board and print the case for it on 3D printer. This involves mocking up with 3D software to the design case, so it might take some time. However, so far I'm very happy with my new toy.
Few days ago, I noticed odd problem with koha-common package. It depends on mysql-client which on squeeze tries to install version 5.1 which conflicts with my installation which uses Percona MySQL build. How can we fix this?
As it turns out, it rather easy. I will just create fake package which will provide mysql-client and in turn depend on percona-server-client using something like:
koha-dev:/srv# cat mysql-client-fake/DEBIAN/control Package: mysql-client-fake Version: 0.0.1 Section: database Priority: optional Architecture: all Depends: percona-server-client Provides: mysql-client Suggests: Conflicts: Maintainer: Dobrica Pavlinusic <dpavlin@rot13.org> Description: Provides mysql-client for percona build koha-dev:/srv# dpkg-deb -b mysql-client-fake . dpkg-deb: building package `mysql-client-fake' in `./mysql-client-fake_0.0.1_all.deb'. koha-dev:/srv# dpkg -i mysql-client-fake_0.0.1_all.deb (Reading database ... 59348 files and directories currently installed.) Preparing to replace mysql-client-fake 0.0.1 (using mysql-client-fake_0.0.1_all.deb) ... Unpacking replacement mysql-client-fake ... Setting up mysql-client-fake (0.0.1) ...Quick and easy. Before you start bashing Debian for this, have in mind that both Koha and Percona MySQL builds are not official Debian packages, so it's not really Debian developers problem.
Update: This problem occurs because Debian developers decided to use virtual-mysql-server and virtual-mysql-client Provides so Percona changed it's provides to virtual-mysql-server but Koha package requires older mysql-client.
Finally I decided to upgrade my wireless network to 802.11n, and to do so I picked up cheap TP-Link TL-WR740N and decided to install OpenVPN, n2n and munin node on it. This is where the problems started because simple opkg install openvpn filled up whole file-system. Instead of declaring fail on this front, I decided to ask a friend how to make this work...
Reason for this upgrade was change in my router provided by ADSL provider. I didn't have any administration privileges on it, and it was only 802.11g device, so my previous configuration with igel which provided pppoe wasn't possible any more (since I can't turn ADSL router into bridge mode). So I decided to scrap igel and move openvpn and n2n to TP-Link instead (which will also help with head dissipation on my closet which hosts all those devices).
Since router has just 4MiB of flash storage, installing large packages is not solution for this platform. However, all is not lost, and there is alternative way to make this work. Trick is in way how OpenWRT uses flash storage. Image which you download from internet contains squashfs (which is compressed) that enable really efficient usage of storage on router itself. All additional packages are installed into overlay file-system, which doesn't support compression so you will fill root file-system really quick. However, there is solution. OpenWrt project provides Image Builder which enables you to select packages which are included in base installation, and thus ends up in squash file-system nicely reducing need for flash storage. Even better, you can also exclude packages which you are not going to use. However, to make this really useful you also have to provide files directory which contains modifications needed to make your specific router configuration work (like IP addresses, OpenVPN keys, n2n keys and similar modification).
First, I downloaded OpenWrt Barrier Breaker (Bleeding Edge Snapshots) and created files directory in which I will create files which are specific for my setup. For a first build (to make sure that it works I just copied /etc/config/network into it and rebuild image with
make image PROFILE=TLWR740 PACKAGES="-dnsmasq -ip6tables -ppp \ -ppp-mod-pppoe -kmod-ipt-nathelper -odhcp6c \ openvpn-openssl n2n muninlite" FILES=../files/I didn't need dnsmasq (because ADSL modem will provide DHCP service for my network) and along the same lines, I also excluded ppp and nat but added openssl, n2n and muninlite (which is munin node written in C).
scp bin/ar71xx/openwrt-ar71xx-generic-tl-wr740n-v4-squashfs-sysupgrade.bin root@192.168.1.2:/tmp/ ssh root@192.168.1.2 sysupgrade -v /tmp/openwrt-ar71xx-generic-tl-wr740n-v4-squashfs-sysupgrade.binThan I hold my breath and after re-flashing router it rebooted and connected to my network. So far, so good. Now I had all required packages installed, so I started configuring packages to my specific need. In the end, I had following configuration files which I copied back to my files folder
dpavlin@t61p:~/openwrt$ find files/ files/ files/etc files/etc/config files/etc/config/system files/etc/config/network files/etc/config/wireless files/etc/config/openvpn files/etc/config/n2n files/etc/openvpn files/etc/openvpn/tap_home.conf files/etc/openvpn/tap_home.sh files/etc/openvpn/prod.key files/etc/init.d files/etc/init.d/openvpn files/etc/dropbear files/etc/dropbear/authorized_keys
After another rebuild of image to make sure that everything works, I was all set with new router for my home network.