A few weeks ago my WRT54GL decided to die. This was quite fortunate because I wanted to upgrade my always on machine to something more powerful and install Debian on it. I had ADSL modem with wifi, so I didn't need wifi in my new box. Fortunately, I got a Igel 3/4 thin client with VIA Eden chipset, 256Mb of RAM and 128 Mb of CF storage so I decided to move by main gateway functions to it.
I had following requirements:
- pppoe to connect to by ISP
- dnsmasq to provide DHCP for my LAN
- iptables will provide NAT
- openvpn and n2n to provide VPN
- motion so I can see what my dog is doing when I'm not home
As a first step, I plugged 2GB CF card into USB adapter on my desktop (Igel can't boot from USB), created btrfs filesystem and mounted it using compress=zlib,ssd. After coping files over size of installation dropped from 1.9GB to just 840MB. This card will be adequate choice for my gateway. I was also toying with idea of moving my apt-cache-ng to this machine, so I wanted more space, and decided to also plug additional 8GB USB keychain for swap and cache storage. But, since I have two devices wouldn't it be better to create btrfs raid1 on it so I can survive failure of CF or USB (ignoring cache and swap)?
Next I plugged CF and USB into Igel and booted it. First boot was slow, but what can you expect from 600 MHz VIA Eden? Here is hdparm speed of CF (sda) and USB (sdb):
root@igel:~# hdparm -tT /dev/sd{a,b}
/dev/sda:
Timing cached reads: 310 MB in 2.02 seconds = 153.76 MB/sec
Timing buffered disk reads: 38 MB in 3.11 seconds = 12.22 MB/sec
/dev/sdb:
Timing cached reads: 302 MB in 2.00 seconds = 150.92 MB/sec
Timing buffered disk reads: 56 MB in 3.07 seconds = 18.26 MB/sec
It's interesting that USB is faster than CF. So, let's try to move my single drive btrfs filesystem to RAID1 configuration using:
btrfs device add /dev/sdb1 / btrfs balance start -dconvert=raid1 -mconvert=raid1 /This is where the problems started. I was running Debian wheezy with 3.2 kernel and rebalance filters are introduced in kernel 3.3. So I upgraded to sid with 3.10 kernel to see weather it will fix at least some btrfs issues. As we will see it really didn't. I got reports about corrupt blocks and after a while I decided to unplug CF and USB, move then over to desktop and create mirror filesystem there.
This worked (and didn't report any errors after scrubing) so I was somewhat confident that my data is safe. This time around, I decided to use noatime,compress=lzo,ssd_spread mount options to gain some speed. Filesystem did grow to 1.2GB (50% increase since I was using lzo instead of zlib for compression) but it was still acceptable. Next, I returned storage to Igel and tried to boot it. Another surprise: it couldn't mount root file system. When I got initrd shell, I could indeed see that I don't have /dev/disk/by-uuid/ nodes for root mostly because it wasn't mounted. However, manual mount from /dev/sda1 didn't work either. However, manual mount from /dev/sdb1 did work. After searching a web for reason it seems that Debian's initrd doesn't issue btrfs device scan on boot so multi-device btrfs filesystems don't get correctly recognized. Why does it work from second disk in mirror is still a mystery to me. I enabled GRUB_DISABLE_LINUX_UUID=true in /etc/default/grub and moved on...
However, my problems weren't over just yet. After running update-grub on booted filesystem Igel was again unbootable. It was reporting strange errors in part because grub-probe got both devices:
root@igel:~# grub-probe --target=device / /dev/sda1 /dev/sdb1This was easy to fix. I modified /usr/sbin/grub-mkconfig to report just /dev/sdb1 so it will be bootable:
GRUB_DEVICE="`${grub_probe} --target=device / | tail -1`"
The story would end here if I didn't try again to run btrfs balance start /. This time around, it didn't report any corrupt blocks, but it did oops kernel with out of memory backtrace. Lesson learned, 256MB RAM is too small for 2GB btrfs filesystem balance. Live and learn. Problem now was that balance restarted when I booted system. So as solution, I unplugged USB which dropped me into initrd shell where I could issue btrfs balance pause / followed by btrfs balance cancel / (cancel alone wouldn't do the trick).
So, what can I conclude about current state of btrfs? It does work (with some hand-holding) but it also has rough edges. But, if you really have flash based storage and need compression it is a valid choice.