Results tagged “linux”

I must admit that Linux administration is getting better with years. I was configuring IPMI serial console on old machines (but with recent Debian) so I decided to find out which is optimal way to configure serial console using systemd.

First, let's inspect ipmi and check it's onfiguration to figure out boudrate for serial port:

root@lib10:~# ipmitool sol info 1
Info: SOL parameter 'Payload Channel (7)' not supported - defaulting to 0x01
Set in progress                 : set-complete
Enabled                         : true
Force Encryption                : true
Force Authentication            : false
Privilege Level                 : ADMINISTRATOR
Character Accumulate Level (ms) : 50
Character Send Threshold        : 220
Retry Count                     : 7
Retry Interval (ms)             : 1000
Volatile Bit Rate (kbps)        : 57.6
Non-Volatile Bit Rate (kbps)    : 57.6
Payload Channel                 : 1 (0x01)
Payload Port                    : 623
Notice that there is 1 after info. This is serial port which is sol console. If you run ipmitool without this parameter or with zero, you will get error:
root@alfa:~# ipmitool sol info 0
Error requesting SOL parameter 'Set In Progress (0)': Invalid data field in request
Don't panic! There is ipmi sol console, but on ttyS1!

To configure serial console for Linux kernel we need to add someting like console=ttyS1,57600 console=tty0 to kernel command-line in grub, and configuring correct serial port and speed:

GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"
All required changes to default configuration are below:
root@lib10:/etc# git diff
diff --git a/default/grub b/default/grub
index b8a096d..2b855fb 100644
--- a/default/grub
+++ b/default/grub
@@ -6,7 +6,8 @@
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2< /dev/null || echo Debian`
-GRUB_CMDLINE_LINUX_DEFAULT="boot=zfs rpool=lib10 bootfs=lib10/ROOT/debian-1"
+# serial console speed from ipmitool sol info 1
+GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,57600 console=tty0 boot=zfs rpool=lib10 bootfs=lib10/ROOT/debian-1"

 # Uncomment to enable BadRAM filtering, modify to suit your needs
@@ -16,6 +17,8 @@ GRUB_CMDLINE_LINUX=""

 # Uncomment to disable graphical terminal (grub-pc only)
+GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"

 # The resolution used on graphical terminal
 # note that you can use only modes which your graphic card supports via VBE
So in the end, there is noting to configure on systemd side. If you want to know why, read man 8 systemd-getty-generator

For a long time, I was looking for PowerPC machine to run Linux on it. I was planning to buy some PowerBook when I get good offer, but that opportunity never really came.

Playstation 3 hardware

Meantime, Sony has been reducing price of PlayStation 3 up to point where they don't include hyper-visor which is a shame. (also known as Other OS support). If I wanted to run Linux on it, I was running out of time. So, I decided to take a leap of fate and buy one 40Gb (CECHGxx) model for ~240 € which seemed like a good price for PowerPC at 3.2GHz with 256 Mb RAM and slowish disk:

dpavlin@ps3:~$ cat /proc/cpuinfo 
processor       : 0
cpu             : Cell Broadband Engine, altivec supported
clock           : 3192.000000MHz
revision        : 16.0 (pvr 0070 1000)

processor       : 1
cpu             : Cell Broadband Engine, altivec supported
clock           : 3192.000000MHz
revision        : 16.0 (pvr 0070 1000)

timebase        : 79800000
platform        : PS3
model           : SonyPS3
dpavlin@ps3:~$ free
             total       used       free     shared    buffers     cached
Mem:        213728      63372     150356          0       2608      25212
-/+ buffers/cache:      35552     178176
Swap:       586332          0     586332
root@ps3:~# hdparm -tT /dev/ps3da

 Timing cached reads:   1640 MB in  2.00 seconds = 819.90 MB/sec
 Timing buffered disk reads:   82 MB in  3.02 seconds =  27.17 MB/sec
You will not get access to graphic cards, but you can use it's 256 Mb of VRAM (DDR3) as block device, but it's also not very fast:
root@ps3:~# hdparm -tT /dev/ps3vram 

 Timing cached reads:   1682 MB in  2.00 seconds = 841.64 MB/sec
 Timing buffered disk reads:   96 MB in  3.06 seconds =  31.40 MB/sec

root@ps3:~# dd_rescue -m 236M /dev/zero /dev/ps3vram
Summary for /dev/zero -> /dev/ps3vram:
dd_rescue: (info): ipos:    241664.0k, opos:    241664.0k, xferd:    241664.0k
                   errs:      0, errxfer:         0.0k, succxfer:    241664.0k
             +curr.rate:    15911kB/s, avg.rate:    15669kB/s, avg.load:  7.4%

root@ps3:~# dd_rescue -m 236M /dev/ps3vram /dev/zero
Summary for /dev/ps3vram -> /dev/zero:
dd_rescue: (warning): /dev/zero (241664.0k): Invalid argument!    
dd_rescue: (info): ipos:    241664.0k, opos:    241664.0k, xferd:    241664.0k
                   errs:      0, errxfer:         0.0k, succxfer:    241664.0k
             +curr.rate:    32621kB/s, avg.rate:    31898kB/s, avg.load: 14.8%
Normal usage is as swap (with higher priority than disk), but this basically just saves disk from spinning because disk is faster at writing (which you really care about if you are swapping to media) but slower at reading:
root@ps3:~# dd_rescue -m 236M /dev/zero /tmp/disk.speed.test
Summary for /dev/zero -> /tmp/disk.speed.test:
dd_rescue: (info): ipos:    241664.0k, opos:    241664.0k, xferd:    241664.0k
                   errs:      0, errxfer:         0.0k, succxfer:    241664.0k
             +curr.rate:    18400kB/s, avg.rate:    18392kB/s, avg.load: 19.2%

root@ps3:~# dd_rescue -m 236M /dev/zero /tmp/disk.speed.test
Summary for /tmp/disk.speed.test -> /dev/zero:
dd_rescue: (warning): /dev/zero (241664.0k): Invalid argument!    
dd_rescue: (info): ipos:    241664.0k, opos:    241664.0k, xferd:    241664.0k
                   errs:      0, errxfer:         0.0k, succxfer:    241664.0k
             +curr.rate:    24036kB/s, avg.rate:    22819kB/s, avg.load:  4.7%
Good news is that USB disks are almost as fast as internal drive:
root@ps3:~# hdparm -tT /dev/sdb

 Timing cached reads:   1740 MB in  2.00 seconds = 870.84 MB/sec
 Timing buffered disk reads:   72 MB in  3.01 seconds =  23.89 MB/sec
Having said that, it's probably one of best machines I saw after my first Alpha :-). It's quiet, all hardware available is supported under Linux and provides great platform for PowerPC development.

Ethernet, on the other hand is fastest way to get data to or from machine:

root@ps3:~# nc -l -p 8888 < /dev/zero

root@t61p:~# nc 8888 | dd_rescue -m 1G - /dev/null
Summary for - -> /dev/null:
dd_rescue: (warning): /dev/null (1048576.0k): Invalid argument!    
dd_rescue: (info): ipos:   1048576.0k, opos:   1048576.0k, xferd:   1048576.0k
                   errs:      0, errxfer:         0.0k, succxfer:   1048576.0k
             +curr.rate:        0kB/s, avg.rate:    86905kB/s, avg.load: 12.4%
root@t61p:~# nc -l -p 8889 < /dev/zero

root@ps3:~# nc 8889 | dd_rescue -m 1G - /dev/null
Summary for - -> /dev/null:
dd_rescue: (warning): /dev/null (1048576.0k): Invalid argument!    
dd_rescue: (info): ipos:   1048576.0k, opos:   1048576.0k, xferd:   1048576.0k
                   errs:      0, errxfer:         0.0k, succxfer:   1048576.0k
             +curr.rate:        0kB/s, avg.rate:    68914kB/s, avg.load: 21.2%

What about graphics?

Basic support is frame-buffer in various resolutions over HDMI (I can't really test compost output). ps3-video-mode -m 130 is something you will type often if you are sitting away from your 1920x1200 LCD...

There is sample code to run RSX from 2007, but it doesn't seem it had any changes since then. There is also libps3rsx and video of it looks promising. It's also from 2007. And it seems you need firmware older than 2.10 which I don't have. Forum thread PS3 Development seems to be dead, but RSX page on wiki is nice overview.

Different approach is to use 6 available SPU cores for YUV to ARGB conversions with scaling. There is also patch for console mplayer -vo to enable movie decoding usable under Linux using SPUs. There is Debian packages for xserver-xorg-video-spu but for sid and it doesn't work for me (next step: recompile).

What about cells?

I still don't know. But, following projects also look very interesting:

Debian installation

I tried to install Debian following instructions for squeeze but between fighting with Sony menu (which I didn't like when I saw it first time on PSP) and trying to type of wireless keyboard which had incredibly short repeat key I wasn't successful. So I tried Debian Live for PS3 and it worked. apt-get dist-upgrade and few hours later I had from working.

As you might know by now, I was debugging memory related problems on one of my systems recently and concluded that normal output from Linux commands are more or less inaccurate. If you want to know why, take a look at Matt Mackall presentation at ELC2009: Visualizing Process Memory or watch following video:

Convinced? So, hop at smem page, compile user-land part and start really tracking your memory usage, let's compare:

dpavlin@t61p:/rest/cvs/smem$ free
             total       used       free     shared    buffers     cached
Mem:       4081400    3882476     198924          0     142904    2731480
-/+ buffers/cache:    1008092    3073308
Swap:      8209172       7492    8201680

dpavlin@t61p:/rest/cvs/smem$ ./smem -w -t
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory       2927016    2845456      81560 
userspace memory             954900     119368     835532 
free memory                  199484     199484          0 
                            4081400    3164308     917092 
Just a few quick notes if you didn't watched whole video carefully:
  • needs kernel 2.6.27 or newer
  • it can work on archived data (from cron in my example usage)
  • userspace cache is backed by file on disk
  • it's a python script which requires matplotlib to create graphs so it's for local reporting

I'm working on Linux version of Sun storage machines, using commodity hardware, OpenVZ and Fuse-ZFS. I'm do have working system in my Sysadmin Cookbook so I might as well write a little bit of documentation about it.

My basic requirements are:

This makes it self-running system which won't fall over itself, so let's see how does it look:

root@opl:~# zpool status
  pool: opl
 state: ONLINE
 scrub: resilver completed after 1h59m with 0 errors on Wed Jun  3 15:29:50 2009

        NAME        STATE     READ WRITE CKSUM
        opl         ONLINE       0     0     0
          mirror    ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0

errors: No known data errors
root@opl:~# zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
opl                            183G  35.7G    21K  /opl
opl/backup                     180G  35.7G    22K  /opl/backup
opl/backup/212052             76.1G  35.7G  8.12G  /opl/backup/212052
opl/backup/212052@2009-05-01  5.69G      -  7.50G  -
opl/backup/212052@2009-05-10  5.69G      -  7.67G  -
opl/backup/212052@2009-05-15  5.57G      -  7.49G  -
opl/backup/212052@2009-05-22  3.54G      -  7.74G  -
opl/backup/212052@2009-05-25  3.99G      -  8.38G  -
opl/backup/212052@2009-05-26  3.99G      -  8.38G  -
opl/backup/212052@2009-06-05  3.72G      -  8.09G  -
opl/backup/212052@2009-06-06      0      -  8.12G  -
opl/backup/212056             1.42G  35.7G   674M  /opl/backup/212056
opl/backup/212056@2009-05-30  37.1M      -   688M  -
opl/backup/212056@2009-05-31  47.3M      -   747M  -
opl/backup/212056@2009-06-01  40.9M      -   762M  -
opl/backup/212056@2009-06-02  62.4M      -   787M  -
opl/backup/212056@2009-06-05  12.1M      -  1.02G  -
opl/backup/212056@2009-06-06      0      -   674M  -
opl/backup/212226              103G  35.7G  26.8G  /opl/backup/212226
opl/backup/212226@2009-05-05  4.29G      -  26.7G  -
opl/backup/212226@2009-05-10  4.04G      -  26.6G  -
opl/backup/212226@2009-05-15  4.19G      -  26.6G  -
opl/backup/212226@2009-05-22  4.12G      -  26.7G  -
opl/backup/212226@2009-05-23  4.12G      -  26.7G  -
opl/backup/212226@2009-05-24  4.09G      -  26.6G  -
opl/backup/212226@2009-05-25  4.14G      -  26.7G  -
opl/backup/212226@2009-05-26  4.13G      -  26.7G  -
opl/backup/212226@2009-06-05  4.20G      -  26.8G  -
opl/backup/212226@2009-06-06      0      -  26.8G  -
opl/clone                      719M  35.7G    25K  /opl/clone
opl/clone/212056-60018         666M  35.7G  1.39G  /opl/clone/212056-60018
opl/clone/212226-60017        53.0M  35.7G  26.7G  /opl/clone/212226-60017
opl/vz                        1.59G  35.7G  43.5K  /opl/vz
opl/vz/private                1.59G  35.7G    22K  /opl/vz/private
opl/vz/private/60014           869M  35.7G   869M  /opl/vz/private/60014
opl/vz/private/60015           488M  35.7G   488M  /opl/vz/private/60015
opl/vz/private/60016           275M  35.7G   275M  /opl/vz/private/60016
There are several conventions here which are useful:
  • pool is named same as machine (borrowing from Debian way of naming LVM volume groups) which makes it easy to export/import pools on different machines (I did run it with mirror over nbd for a while)
  • snapshots names are dates of snapshot for easy overview
  • clones (writable snapshots) are named using combination of backup and new container ID

There are several things which I wouldn't be able to get without zfs:

  • clones can grows as much as they need
  • data is compressed, which increase disk IO as result
  • zfs and zpool commands are really nice and intuitive way to issue commands to filesystem
  • zpool history is great idea of writing all filesystem operations to internal log
  • ability to re-sliver (read/write all data on platters) together with checksums make it robust to disk errors

As you might guessed by now, I played with file-systems for backup appliance So, against my good judgment, I decided to try btrfs to see how ready is it to replace zfs-fuse configuration with real in-kernel file-system (zfs-fuse is not slow, because disks are much slower than any peace of software).

So far, I found following annoyances in brtrs:

  1. snapshots can't be removed (I'm doing incremental forever backups, so this is not show-stopper)
    You can remove all files in snapshot directory, but not directory itself. I would guess that removing files would just increase disk space, because it's copy-on-write filesystem, but I didn't test that.
  2. there is no indication which directory is snapshot (if you didn't wrote down in log which is snapshot, you are out of luck)
  3. it seeks quite a lot (there is 40-70% wait time in vmstat while running rsync which I guess is seek, because there is no block input/output operations at same time)
  4. it will oops your (Debian 2.6.29-2-686) kernel:
    Message from syslogd@klin at May 16 00:42:31 ...
     kernel:[ 4057.994566]  [<c0119e0f>] kmap_atomic_prot+0xbd/0xdd
    Message from syslogd@klin at May 16 00:42:31 ...
     kernel:[ 4057.994576]  [<c0119d30>] kunmap_atomic+0x58/0x7a
    Message from syslogd@klin at May 16 00:42:31 ...
     kernel:[ 4057.994586]  [<f83a61a2>] btrfs_cow_block+0x134/0x13d [btrfs]
    Message from syslogd@klin at May 16 00:42:31 ...
     kernel:[ 4057.994608]  [<f83a8b4b>] btrfs_search_slot+0x1f0/0x622 [btrfs]
    Messag./ line 8:  4316 Segmentation fault      rsync -ravHC --numeric-ids --delete $from:/mnt/vz-backup/private/$1/ /$pool/$1/

After that I concluded that warning about alpha state of btrfs is there with a reason. I didn't fully appreciate Theodore Ts's warning about development status of btrfs until I got kernel oops.

My point of view

First, let me explain my position. I was working for quite a few years in big corporation, and followed EMC storage systems (one from end of of last century and improvement that Clarion did on our production SAP deployment). I even visited EMC factory in Cork, Ireland, and it was very eye-opening experience. They claim that 95% of customers who visited factory did buy EMC storage, and I believe them (we did upgrade to Clarion btw).

In my Linux based deployments on HP, Compaq and IBM hardware I did various crazy RAID configurations (RAID5 across disks on controller and then stripe across other controller, for example). Those where the easy parts: you got RAID controller with DRAM cache (~256Mb) and some kind of battery backup which greatly improved write performance.

Later on in CARNet we had HP EVA storage which proved quite flaky. I heard from friend in one enterprise deployment that they use them only for testing. And you know, it's just shelf of disks with redundant controllers and fiber interface...

In the mean time, on Linux software RAID front, I used md implementation RAID1 and RAID5 back in the days when Linux distributions couldn't handle that.

Solid state drives

However, solid state drives changed a lot of that. I still haven't had pleasure to use Intel SSD which are supposed to be good, but USB sticks are also flash storage, but with quaky characteristics.

This particular one is ID 0951:1603 Kingston Technology Data Traveler 1GB/2GB Pen Drive as reported by Linux, but in fact 8Gb model which seem to have 128Mb of memory which is writable at about 6Mb/s and after that write speed drops to 45K/s.

On the other hand, there is ZFS on FUSE project which enables some really interesting applications of Sun's (and now Oracle) file-system. I do have to mention Sun at this point. Ever since I heard about Oracle's acquisition of Sun, I have wondered what will happen with ZFS. I might even suspect that ZFS is the main reason why Oracle bought Sun. Let me explain...

Sunshine Oracle

If you look at database market (where Oracle is), the only interesting thing to improve relational databases is to make them extremely fast. And that revolution is already here. Don MacAskill from SmugMug makes compelling case about performance of SSD storage. If you don't believe words, watch this video from 24:50 to see solutions to MySQL storage performance problem: hardware!. Sun's hardware. Do you think that Oracle didn't noticed that?

Enterprise storage cheaply

Did you watched the video? I really don't agree that it's hardware. Common! It's Opteron boxes with custom built SSD disks optimized for write speed. SSD with super-capacitors instead of batteries in old RAID controller.

But, to make it really fun, I will try to re-create at least some of those abilities using commodity hardware in my university environment. I have Dell's OptiPlex boxes which come loaded with a lot of goodies to put together a commodity storage cluster:

  • Intel
  • Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
  • 3Gb RAM
  • 2 SATA disks with ~80Mb/s of read/write performance
  • multi-card reader and 8 USB slots
  • fake software RAID on Intel chipset (supported by dmraid but even it's documentation suggests not to use it)


Why ZFS? Isn't btrfs way to go? For this particular application, I don't think so. Let me list features of ZFS which excite me:

  • ability to store log to separate (mirror) device (SSD, USB sticks if that helps)
  • scrub: read all bytes on disk and rewrite it (beats smartctl -t long because it also re-allocates bad blocks, I've seen 80Mb/s scrub)
  • balancing of IO over devices (I will use this over nbd to split mirror between machines for fail-over)
  • arbitrary number of copies (nice for bigger clusters of storage machines)
  • nice snapshots which display it's size and can be cloned to writable ones
  • snapshot send/receive to make off-site backup copies
  • L2ARC - balance read and write cache over SSD devices with different characteristics (USB sticks have fast read and slow write, so they might be good fit)

You might think of it as git with POSIX file-system semantics.

But, it's in user space, you say, it must be slow! It isn't. Really. Linux user-space is much faster than disk speed and having separate process is nice for monitoring purposes. File-system overhead gets counted into user time, not system, so system time is clear indicator of driver (hardware) activity and not file-system overhead.

I have most parts of this setup ready, and I'm using it to backup OpenVZ containers. So, I'm running OpenVZ kernel and I can even make virtual machines from backup snapshots to recover into some point in time. After I finish this setup, expect a detailed guide (it will probably be part of my upcoming virtualization workshop as alternative to LVM).

I just read about LinuxDNA project to compile kernel with Intel's ICC compiler. For a start it's 2.6.22. So forget about recent hardware. Oh, did I mentioned that binary drivers don't work?

But, it has 40% speed improvement. I have said once or twice that Itanium never really worked with Linux because of poor gcc complier. I really hope to see end of x86 architecture it Itanium history has anything to teach us. :-)

Still reading?

O.K, let me elaborate a bit: 40% speed increase available only with propriatory compiler is just enough for me not to buy another x86 processor if I had any realistic alternative.

With low power AMD solutions like SheevaPlug and AMD netbooks this might be possible in future, but not right now.

What is my problem with ICC? Did you know that you have to change binary of ICC to make it support AMD processors? I did, although this page vanished from Internet.
Do you really want to compile your kernel with compiler like that?
If you are still thinking something like: hell, yes! 40% faster! (imagine dark chorus going geeeentooooo in background while reading this) Please explain to me how do you intend to compile binary for any x86 processor, not just Intel, for example for LiveCD?

I wanted to automatically configure second monitor plugged into my laptop as above internal LCD, and I was prepared to allocate few days for this task. But, in recent servers this has gotten much better, so it's enough just to add following to .xinitrc:

xrandr --output DVI-0 --auto --above LVDS
xrandr --output VGA-0 --above LVDS

There is also grandr (with package in Debian) if you want point-and-click functionality.

Don't work on single monitor, having dual monitor again is such productivity boost :-)

Update: this works only on cards which report more than one output connected with xrandr -q:

Screen 0: minimum 320 x 200, current 1920 x 2250, maximum 1920 x 2250
VGA-0 disconnected (normal left inverted right x axis y axis)
LVDS connected 1400x1050+0+1200 (normal left inverted right x axis y axis) 305mm x 228mm
DVI-0 connected 1920x1200+0+0 (normal left inverted right x axis y axis) 474mm x 296mm
Im my case, it's ATI Technologies Inc M52 [Mobility Radeon X1300] with radeon driver. To make this work, I also needed to create virtual desktop which is large enough to accomodate both screens by adding Virtual to /etc/X11/xorg.conf:
Section "Screen"
        Identifier "MyScreen"
        Device     "MyCard"
        DefaultDepth     24
        SubSection "Display"
                Virtual 1920 2250

So It has been a week from time when borrowed OLPC entered my family of computers. I have Thinkpad T60 with Atheros AR5212 (which works with atk5k driver from 2.6.25, nice work!) and Eee PC with Atheros (which works with special madwifi patch).

Since 802.11s just landed into upstream kernel git, I was eager to take a look at this mash network thing. Oh, how ignorant I was. OLPC uses 802.11s protocol which is different from official implementation of 802.11s and with good reason: they are using embedded processor in wifi card do to mash protocol for them (saving power and enabling mash to work when laptop is suspended). I could have installed olsr on OLPC, but I'm really trying to have bigger mash which is compatible with unmodified OLPCs.

Because my time is limited, I would like to work in user-land if at all possible, and since wpa_supplicant can work on unmodified kernels, it would be nice to have that level of support for OLPC mash also. After a lot of browsing (and reading few really great wifi hacking sites), I concluded that only hope is radiotap which is more-or-less supported on every pcmcia wifi card that I have (prism based 802.11b card and rt2500). I had also found simpliest possible code which uses radiotap to start with.

Now, I would just need another OLPC to save some network traces and start experimenting :-)

Aside from that, I switched totally to OLPC for this week, and amazingly enough, I didn't miss my Eee PC one tiny bit. Although a bit slower than Eee, OLPC screen is bigger (and better in black and write mode on sunlight) which helps a lot with web pages. Browser performance is amazing, so I have little doubt that we will be able to support most of web sites on OLPC without much problem. OOH, I did notice a couple of excessive round-tips on one of my web sites, while surfing on it, but that's for best anyway :-)

Update: According to message on libertas-dev mail list there is effort to use kernel's 802.11s implementation which makes my effort in supporting OLPC variant obsolete.

GNU fdisk broken?

I have been backing up whole disk image from Eee PC, and mounting it using loop file system to access partition in it. However, I have problems with GNU fdisk which reports 4Gb image as:

Disk /backup/eee/hda: 3 GB, 3997486080 bytes
255 heads, 63 sectors/track, 486 cylinders, total 7807590 sectors
Units = sectors of 1 * 512 = 512 bytes

Device Boot Start End Blocks Id System
/backup/eee/hda1 63 4803435 2409718 83 Linux
/backup/eee/hda2 4819563 7759395 1469947 83 Linux
/backup/eee/hda3 7775523 7775460 0 c FAT32 LBA
/backup/eee/hda4 7791588 7791525 0 ef EFI FAT

For a start, disk size is wrong:

$ ls -al hda
-rwxrwxrwx 1 dpavlin root 4001292288 2008-01-20 00:59 hda

And then, even more wrong, offsets of partition seem to be wrong. When same image is examined using fdisk from util-linux, sectors are reported like this:

Disk hda: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x332b332a

Device Boot Start End Blocks Id System
hda1 63 4819499 2409718+ 83 Linux
hda2 4819500 7775459 1477980 83 Linux
hda3 7775460 7791524 8032+ c W95 FAT32 (LBA)
hda4 7791525 7807589 8032+ ef EFI (FAT-12/16/32)

And this is correct (let's ignore size for now). I can verify this by mounting second file system as:

sudo mount hda 1 -o loop,offset=`expr 4819500 \* 512`

This seems to be off-by-one error. There is bug reported against Debian package which seems related, but than again, in my case I'm examining same disk image.

So, I began looking for Eee PC last year (well, more than two weeks ago, anyway) and my original plan was to buy one while I'm in Berlin for new year. However, since there wasn't any Eees available in Germany until second week of 2008, I was in some disspair becuause we where leaving on 6th.

You can understand when I read Avian's blog post about his new Eee PC. He is in Slovenia, and I'm in Germany and he managed to buy it? How? Quick e-mail later, and he said that it's available in Building of Fun, some kind of on-line web shop in Ljubljana. Since Slovenia is so much closer to Croatia than Germany (in which I was at that time) I just postponed my purchase for better times. In meantime, Kost got interested also, and after return to Croatia he contacted them and we bought our units.

That would be happy part of story, if only one of them didn't have one constantly lit LCD pixel. Reviewing ASUS warranty, I found that I found one point:

2. TFT LCD defect policy -- Eee PC does not provide ZBD (Zero Bright Dot) warranty for TFT LCD screens.

Yes, it's in there, one you buy the unit, that is. Since we couldn't replace it today I guess I'm out of luck. I will try some of software solutions over the night in hope that it will go away, but some more drastic measures like rubbing LCD screen gently are just too much to ask from me...

Other than that, it's a great device: designed for wifi communication, quick browsing and occasional terminal session using ctrl+alt+t. It's not designed to be primary PC, but all features are so well integrated and working seamlessly that I will have hard time reinstalling it with Debian. Until I need dwm which I got used to so much. I can probably wait for a few more hours :-)

I can't say anything about battery life, other than fact that it got half-filled in two hours while I was working on it with wifi. I will do some monitoring to see how well is battery holding on this device (there isn't acpi command, but all /proc entries are there, so it shouldn't be problem).

Form-factor is just great. Yeah, keyboard is small, but I can type on it (with my big fingers) without any problems. We did try to boot few distribution from USB stick (including Puppy and some Slackware derivative) without any problems. Well, there shouldn't be any: this is basically a palm-top size Intel box (at last!) with strange screen size of 800*480.

After first day with my Eee PC, I'm very pleased with it. To make things a bit easier now that I'm back on ThnikPad, I wanted to access Eee PC using keyboard and mouse from ThinkPad, so on eeepc I edited /etc/X11/xinit/xserverrc to remove -nolisten tcp, restarted X using ctrl+alt+backspace, typed xhost mylaptop.lan on eeepc and then started x2x -west -to eeepc.lan:0 on my laptop and now I can pass from laptop to eeepc sitting on left with just a mouse move. Sweet.

RAID5 for home

Several years after I wrote about software RAID1 and RAID5 I must say that I'm happily surprised how much software and hardware support for software RAID under Linux improved. My articles are more or less obsolete now, but this is short story of my RAID5 array for home.

I have Compaq brand-name desktop PC (refurbished, good buy) with one 160Gb SATA disk. I wanted to add three more 500Gb disks in RAID5 array to create ~1TB storage with redundancy.

Motherboard has just two SATA channels and form talking with a lot of people, it seems to me that SATA devices can't be chained (please correct me if I'm wrong). I somehow assumed that I can connect more than one disk on one SATA channel, partly from SCSI world, partly from old IDE (err, PATA) master/slave relationship. So, additional controller was in order, and only one which was available (with two SATA ports and in PCI-X variant) was no-name Sil 3132 based one:

20:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

I googled a bit and found some horror stories (including binary driver on Silicon Image's site for RHEL kernels), but latest Debian kernel 2.6.23-1-686 just worked. I wanted to use it just as SATA controller (with RAID part), so I was all set. It doesn't have RAID5 anyway.

Now, let's take a short de-tour and explain why do I want to have RAID5? Isn't RAID1 better/faster/right stuff for system?

I know only one use for RAID1: booting. And that's if you don't have hardware RAID5 which shows you RAID arrays as one disk anyway. If you want to boot Linux from software RAID5 you are out of luck.

But there is simply no reason to install system on hardware RAID1 as opposed to hardware RAID5. If you have enough disks (five for example) worst configuration is 2*RAID1 + 3*RAID5 since you get maximum of 2 disks accumulated performace. Much better performace (accumulated 4 disks) is just 5*RAID5.

Having said this rand for RAID5, I also didn't want to boot from it (since I have system on 160Gb disk), and I just need reliable and fast disk. Since I could fit just 4 disks in case, I will get 2 disks of accumulated performance and reliability (third disk).

Why not simple striping over three disks (hack, I would get 500Gb of storage more)?

I don't really believe in desktop-grade disks, (even SATA). Those disks are designed to have 8 hours/day life cycle and will probably die in two years. And I like my data...

Let's see some performance, first single disk:

root@brr:~# hdparm -tT /dev/sda
 Timing cached reads:   2066 MB in  2.00 seconds = 1033.49 MB/sec
 Timing buffered disk reads:  232 MB in  3.00 seconds =  77.22 MB/sec

and RAID5 array:

root@brr:~# hdparm -tT /dev/md0
 Timing cached reads:   2030 MB in  2.00 seconds = 1014.76 MB/sec
 Timing buffered disk reads:  444 MB in  3.01 seconds = 147.49 MB/sec

After several hours of uptime, they are not very hot (order is same as in case):

root@brr:~# uptime
 21:53:24 up  5:57,  1 user,  load average: 0.00, 0.00, 0.00
root@brr:~# echo -e 'c\nd\na\nb'  | xargs -i hddtemp /dev/sd{}
/dev/sdc: WDC WD1600JS-60MHB1: 46°C
/dev/sdd: WDC WD5000AAKS-00YGA0: 45°C
/dev/sda: WDC WD5000AAKS-00YGA0: 48°C
/dev/sdb: WDC WD5000AAKS-00YGA0: 44°C

But they do go up to 55°C under heavy load (specially unhappy sda which is always a bit warmer than other disks).

As you can also see, it moved my old 160Gb disk to sdc. Hack, I'm quite sure that I could force bios into re-ordering this (or turn off RAID BIOS on controller), but I just don't have time.

And lastly how does it look?

root@brr:~# vgdisplay 
  --- Volume group ---
  VG Name               raid5
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               931.52 GB
  PE Size               4.00 MB
  Total PE              238468
  Alloc PE / Size       76800 / 300.00 GB
  Free  PE / Size       161668 / 631.52 GB
  VG UUID               onzKEw-TaRF-JBe1-9YWN-D6aJ-FIuY-1VRmLo

I didn't wrote exact commands to perform creation of array, but madm --create --help was basically enough. And that I waited for two hours for RAID array to sync...

There is one more useful tit-bit: if you have ext3 filesystem formated with resize_inode option you can do on-line resize of ext3 filesystem with resize2fs. Debian already does this by default (since etch, I beleve), but you can check with:

root@brr:~# tune2fs -l /dev/raid5/rest | grep resize_inode
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file

So, if you installed LVM, you can on-line resize your logical volume while coping data to it. I did that while coping from old PATA and USB disks at same time and it works well... It does take much longer (and does more disk I/O) than resize_reiserfs, however.

And don't forget noatime for ext3!

Video editing under Linux

I wanted to edit some video for my local LUG. However, this doesn't seem to be simple task under Linux.

I have some background since I worked on local TV station from 1991-1995 among other things in video editing. So I do know semi-professional and professional video equipment from those days (hack, I was earning good buck with it!) and my judgment is somewhat biased towards ease of use and practical problems which I encountered during editing of professionally shot video material (I don't nearly have so steady hand, it seems).

I have video materials recorded with JVC Everio GZ-MG575E (There doesn't seem to be any good official JVC page about it. Go figure that!) Reposted by lsusb as:

Bus 005 Device 004: ID 04f1:0008 Victor Company of Japan, Ltd GZ-MG30AA/MC500E Digital Video Camera

Files from internal read-only vfat 40Gb disk are available when camera is mounted as USB device on Linux as sd_video/prg001/mov*.mod. File on one of files report MPEG sequence, v2, program multiplex.

Let's look at available options:

  1. Kino - Safe choice, right? Debian package, version 1.1.1.
    It can't import avi in original format. When I tried to convert it some other .avi format using ffmpeg it segfaulted on loading of that avi.
    It has somewhat home video look which I don't like.
  2. cinelera - GUI hurtes my eyes and distracts too much from video footage. I tried. I really tried. It didn't work for me.
  3. The Open Movie Editor can't be compiled from upstream repository. Looked somewhat promising up to that point...
  4. Avidemux - to be honest, I found this one while Googling for video editing on FreeBSD software section video. Good catch, I had Debian package version 2.3.0 available from
    Very nice tool, it fits my brain rather well. Well, I would have to remap keys because it's just half-usable on notebook keyboard (no numeric keymap, allthough I might buy USB one just for this application :-)
  5. pitivi able to import avi, but blocked after a few clicks and stopped refreshing screen. It's interesting to note that clips which where loaded into avidemux had little picture from movie, while the ones which weren't loaded didn't.
  6. LiVES - last time I tried to find video editing software, LiVES turned out as only software which worked at all. Still, strange intraface, in part Gtk based, and in part just ugly doesn't sit well with me.
  7. Kdenlive is another project which is well worth a look. Clean QT interface, all the options I would like, want or need.
    Just two problems: I don't hear any audio. I do get blink sound from KDE, but not from video sound. Another problem is related to by attempts to solve previous one: upstream Subversion repository source code doesn't pass configure on my machine. Yikes.
    Update: My problems with audio are solved after I insalled sdl-alsa variant (instead of old alsa-oss, my fault). It's quite capable software, my only remaining gripe is that every action has a click or two indirection until you can have it done. Other than that I have problems with audio-video sync when producing any format other than mpeg (which is same as my source format), but I don't care much about that, because I intend to do recoding as final step into multiple formats anyway...

bcm963xx git repository

As I mentioned long time ago, I collected various Broadcom GPL releases and now have a git repository with ones that I was able to found.

Following versions are currently there:

  • 3.00.01V
  • 3.00.03
  • 3.02.02V
  • 3.04.01
  • 3.04.02V
  • 3.06.06
  • 3.10.02

Tags are in flux (and will probably mirror branches anyway at some point). Right now different branches have distributions from vendors, but I have to clean a few of them which where imported before I created automatic procedure for import.

Please if you have any Broadcom bcm963xx development kit which is still not in tree, please send it in!

Version of development tree is included in version.make file at top-level:

dpavlin@host:/opt/bcm963xx$ cat version.make 

However, some manufacturers include own version numbers, but most of the time they leave original ones (if you apply some creative encoding to version number variants). However, to be completely sure that version isn't a duplicate you can checkout existing versions using:

git clone git://

Your pushes are welcomed :-)

Repository is quite big, about 600Mb, but it's on fast connection. However, ideas how to reduce that size (I don't think that there is so much changes!) is welcomed from someone who is better in git than me!

OpenMoko arrived

After four (4) weeks of waiting from the moment credit card has been charged, it finally arrived. It seems that shipping GSM-like devices to Croatia as person (as opposed to company) isn't something envisioned by Hrvatske agencije za telekomunikacije so we had to get additional papers (most of which I don't have) and in the end they settled with hardware specification of Neo1973 from OpenMoko wiki.

I must say that people all over the process where helpful and nice: we had to communicate much more with them then I hoped, but in the end it worked out o.k.

And now several obligatory pictures (hopefully not redundant :-)




After that I got rootfs flashed and I now have new (semi-functional) phone :-)

root@fic-gta01:~$ cat /proc/cpuinfo 
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 132.71
Features        : swp half thumb 
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format A
Cache format    : Harvard
I size          : 16384
I assoc         : 64
I line length   : 32
I sets          : 8
D size          : 16384
D assoc         : 64
D line length   : 32
D sets          : 8

Hardware : GTA01
Revision : 0240
Serial : 0000000000000000
root@fic-gta01:~$ free
total used free shared buffers cached
Mem: 126644 52604 74040 0 84 39100
-/+ buffers/cache: 13420 113224
Swap: 0 0 0
root@fic-gta01:~$ df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/mtdblock4 62576 42572 20004 68% /
tmpfs 40 0 40 0% /mnt/.psplash
/dev/mtdblock4 62576 42572 20004 68% /dev/.static/dev
tmpfs 2048 64 1984 3% /dev
tmpfs 63320 4 63316 0% /tmp
tmpfs 63320 116 63204 0% /var/volatile
tmpfs 63320 16 63304 0% /dev/shm
tmpfs 63320 0 63320 0% /media/ram
/dev/mmcblk0p1 495168 0 495168 0% /media/card

Finally OpenMoko is is available for order. So, I ordered mine yesterday. Still no confirmation e-mail, but let's be hopeful.

On a side note, on-line store yesterday didn't have selection of model (black/gray or white/orange) so I didn't made a pick. Today there is choice, but I somehow hope that mine will be black :-). But, it doesn't really matter...

People who know me shouldn't be surprised that I would buy a phone which real demonstration is answering a phone call from command line (on device which doesn't have a keyboard!) - just look at
this video from 1:15:00 or so and you will know why you really want this device :-)

Here are some of fun stuff which I would like to hack on:

  • record GPS route and on second occasion estimate arrival time (even integrate with calendar with option to send apology sms 5 minutes before time with estimate of your arrival :-)
  • make all kind of confirmations and informations using speech synthesis... I usually listen to podcasts so it would be real nice to have it mute and read you incoming sms
  • sms sorter, initial reason for buying a phone which can run perl
  • blue tooth keyboard - seems like no-brainer, but I don't really know much about blue tooth devices

I have been somewhat busy with other stuff in my life (including writing compressed filesystem using gzip with fuse -- but that is topic for another post), so I just managed to update my svn checkout of openmoko before by main build server started freezing again. Oh, it seems that constant kernel updates won't be enough to create stable development machine...

But, than I looked around and found out that I can have simple (emulated) openmoko environment by just installing qemu-neo1973. I went so far to install it on Toshiba Tecra with touch screen to get a feel for user interface. I can guess that hacking this device will be fun.

tar gotcha when migrating disk

Over the years, I have changed 2.5" hard-drives in laptop more often than I want to remember. Last one lasted only year and a half, which is noticably less than previous ones which managed to survive for two and half years at least.

But, this time, I didn't manage to find two 2.5" HD adapters for desktop PC, so I opted to use netcat and tar like this:

new$ nc -w 5 -l -p 8888 | tar xvfSp -
old$ tar cfvSp - / | nc new 8888

Can you see the touble here? (No, I'm not thinking about redundant S in one of tars -- hack, I'm not even sure it's redundant, man page didn't tell me anything useful).

If you couldn't guess, I didn't have completly same /etc/group and /etc/passwd on both machines. Life sucks. I should have used --numeric-owner for tar and all would be well, thus here is a blog post...

Update: if you are using framebuffer, leaving v flag will slow down transfer considerably because screen scrolling is slow. So, faster and correct alternative is:

new$ nc -w 5 -l -p 8888 | tar --numeric-owner -xSpf -
old$ tar --numeric-owner -cSpf - / | nc new 8888

bcm963xx router hacking

For some time, I'm (not quite happy) user of T-Com ADSL router. My particular one is Elcon 2253EU. But, on closer inspection, this device is more or less plain BCM96345 ADSL router. It does, however, have it share of problems: if you click enough on it's web interface, it will just die. While rest of router will work, I won't be able to access it over web, telnet or ftp any more. It sucks (did thay test is at all?)

So, I stared some googling, and found a lot of Broadcom development kits which where (more-or-less) just dropped by various manufacturers on the web (as a side-note, why doesn't Broadcom put those kits on his site and save us all so much trouble?).

I now have various versions (2.14, 2.18, 2.21, 3.02) and I will publish mirror of those images (because original site just disappeared last week) as soon as I sort through them.

With that out of the way, let me say that Broadcom is still GPL violator because they link non-free binary blobs directly in kernel. That's sad. Even worse, they are actually shipping huge parts of user land in binary only forms.

My idea is following: my ADSL modem has an USB port for which I don't have use (it's connected using ethernet). So, I would like to connect USB disk to it. For that, I would guess that I have to re-compile kernel (and since USB driver is binary junk, it might be hard or impossible) and connect disk to it (via USB 1.1, sic!). This will provide me with always-on disk storage on my network (or so I hope). Does this seem sane to you?

Also, did anybody managed to build ADSL images for 96345R target? Most of SDKs actually miss parts for this platform (although most of binary parts are same for all targets). I could build 96348R (which seems similar enough).

But, all this will have to wait few more days to get with a friend who has MAX232 to RS232 cable (for serial console, which should enable booting of new firmware image without flasing device) and JTAG cable (to extract exsiting firmware which, oh, joy, isn't available for download).

I'll keep you posted about progress (if any).