Dobrica Pavlinušić's Weblog / Blog

Using btrfs snapshots for incremental backup

2010-02-01T17:06:16Z

You have heard of Apple Time Capsule, and zfs's snapshots which provide nice way to produce incremental backups which you can use to quickly recover a rm -Rf typo or compare changes... You can't really use LVM snapshots for that because they have fixed size of snapshot. ZFS has copy-on-write snapshots, but zfs-fuse isn't fastest solution on Linux.

I decided to give btrfs another try. On Debian 2.6.32-trunk kernels, it stills have problems with 686 kernel, but amd64 version (even with 32-bit users-pace) seems to work stable so far.

Let's take a look at example usage on /dev/vg/koha-btrfs logical volume.

create filesystem

root@mlin:~# mkfs.btrfs /dev/vg/koha-btrfs

create subvolume which we will use for data

root@mlin:~# btrfsctl -S koha-2010-01-25 /virtual.btrfs/

populate it with some data

root@mlin:~# time cp -ra /virtual.clone/koha-2010-01-25 /virtual.btrfs/

real    15m32.507s
user    0m1.288s
sys     0m54.519s

create base snapshot
```
root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.mlin /virtual.btrfs/koha-2010-01-25
```
There is convention here: since snapshot directories have to be on same btrfs volume, I decided to use base_name(dot)something as convention for my snapshots.

make some changes on base directory

root@mlin:~# time rsync -ravH --numeric-ids --sparse --delete --exclude 'backup*' 10.60.0.90:/opl/clone/koha-2010-01-25/ /virtual.btrfs/koha-2010-01-25/

sent 1538957 bytes  received 381670534 bytes  964049.03 bytes/sec
total size is 18912566086  speedup is 49.35

real    6m37.539s
user    0m59.524s
sys     1m23.633s

make another snapshot for this version

root@mlin:~# btrfsctl -s /virtual.btrfs/koha-2010-01-25.opr /virtual.btrfs/koha-2010-01-25

Following this, we now have three directories which share same data (tanks to copy-on-write feature in btrfs) and look like ordinary directories:

dpavlin@mlin:~$ ls -ald /virtual.btrfs/koha-2010-01-25*
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.mlin
drwxr-xr-x 1 root root 256 Dec 29 19:38 /virtual.btrfs/koha-2010-01-25.opr

After a while, you will run out of disk space, but since we are on LVM, extending filesystem is really easy:

dpavlin@mlin:~$ mount -t btrfs
/dev/mapper/vg-koha--btrfs on /virtual.btrfs type btrfs (rw,noatime)


root@mlin:~# lvdisplay /dev/vg/koha-btrfs 
  --- Logical volume ---
  LV Name                /dev/vg/koha-btrfs
  VG Name                vg
  LV UUID                kl4QUc-IyK2-IM8m-0DKD-eI5k-H2T1-HySIwE
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                40.00 GB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

   
root@mlin:~# lvextend -L +10G /dev/vg/koha-btrfs
  Extending logical volume koha-btrfs to 50.00 GB
  Logical volume koha-btrfs successfully resized


root@mlin:~# btrfsctl -r max /virtual.btrfs/

Hopefully this will help you to get started with btrfs snapshots. I still don't consider it super stable (especially since i did saw few kernel oopses using 2.6.32-trunk-686) but for quick experiments with Linux containers (or anything stored on filesystem for that matter) it proved more than adequate.

It's about system, stupid!

2010-01-28T13:30:00Z

When you are working as system architect or systems librarian, your job is to design systems. My initial idea was to create small Google out of 12 machines which are dedicated to be web kiosks. I decided to strictly follow loosely coupled principle, mostly to provide horizontal scaling for my data processing needs. I wanted to be able to add machine or two if my query is too slow... This easily translates into "now long will I have to wait for my page to generate results"....

I decided to split my system into three logical parts: network booting, data store, and quick reporting. So, let's take a look at each component separately:

PXElator
- supported protocols: bootp, dhcp, tftp, http, amt, wol, syslog
- boot kiosks using Webconverger (Debian Live based kiosk distribution)
- provides web user interface for overview of network segment for audit
- configuration is stored as files on disk, suitable for management with git or other source control management
MongoDB
- NoSQL storage component which support ad-hoc queries, indexes and other goodies
- simple store for perl hashes from PXElator generated every time we see network packet from one of clients using one of supported protocols
Sack
- fastest possible way to execute snippet of perl code over multiple machines
- this involves sharing information to nodes, executing code on all of them and collecting results back, all in sub 3 second mark!
- web user interface for cloud overview and graph generation using gnuplot

When I started implementing this system last summer, I decided to use CouchDB for storage layer. This wasn't really good choice, since I didn't need transactions, MVCC or replication. Hack, I even implemented forking for document stored in CouchDB to provide faster response to clients in PXElator.

Moving to much faster MongoDB I got ad-hoc queries which are usable (as in I can wait for them to finish) and if that's too slow, I can move data to Sack and query it directly from memory. As a happy side effect, making shards from MongoDB is much faster than using CouchDB bulk HTTP API, and it will allow me to feed shards directly from MongoDB to Sack nodes, without first creating shards on disk.

I'm quite happy how it all turned out. I can configure any host using small snippet of perl code in PXElator, issue ad-hoc queries on audit data on it in MongoDB or move data to Sack if I want to do data munging using perl.

As you noticed by now, I'm using live distribution for kiosks, and machines do have hard drivers in them. Idea was to use those disks as storage with something like Sheepdog. seems like perfect fit. With it in place, I will have real distributed, building size computer :-).

MongoDB - so you want fast NoSQL database which you can grok

2010-01-24T11:56:57Z

I have been using CouchDB for some time now, mostly as audit storage for PXElator. Audit data stores are most useful for ad-hoc queries (hum, when did I saw that host last time?), and CouchDB map/reduces took half an hour or more. I wrote mall script couchdb2mongodb.pl to migrate my data over to MongoDB (in 26 minutes) and run first query I could write after reading MongoDB documentation about advanced queries. It took only 30 seconds, compared to 30 minutes or more in CouchDB. I was amazed.

This was NoSQL database which I can understand and tune. MongoDB has indexes and profiler so tuning query down to three seconds was a simple matter of adding an index. All my RDBMS knowledge was reusable here, so I decided to take a look why is it so much faster than CouchDB for same data...

To be honest, MongoDB, High-Performance SQL-Free Database by Dwight Merriman, CEO of 10gen won me over to finally try MongoDB. It was technical enough to make me think about MongoDB arhitecture and benefits. It's clearly pragmatic, let's re-think horizontally scalable hash storage with ad-hoc queries model, but with funny twist about close coupling with language types all encoded in BSON format, which is very similar to Google's protocol buffers.

First, let's have a look at raw side of data on disk. At some level, it will translate to number of IO operations involving rotating platters and usage of buffer cache.

root@opr:~# du -hc /var/lib/couchdb/0.9.0/.pxelator* /var/lib/couchdb/0.9.0/pxelator.couch
655M    /var/lib/couchdb/0.9.0/.pxelator_design
23M     /var/lib/couchdb/0.9.0/.pxelator_temp
7.8G    /var/lib/couchdb/0.9.0/pxelator.couch
8.4G    total

root@opr:~# du -hc /var/lib/mongodb/pxelator.*
65M     /var/lib/mongodb/pxelator.0
129M    /var/lib/mongodb/pxelator.1
257M    /var/lib/mongodb/pxelator.2
513M    /var/lib/mongodb/pxelator.3
513M    /var/lib/mongodb/pxelator.4
513M    /var/lib/mongodb/pxelator.5
17M     /var/lib/mongodb/pxelator.ns
2.0G    total

Here is a first hint about performance: MongoDB's 2G of data (which are used as mmap memory directly, leaving flushes and caching to OS layer) are almost a perfect fit into 3G of RAM memory I have in this machine.

MongoDB has montodump utility which dumps bson for backup and it's even smaller:

root@opr:~# du -hcs dump/pxelator/*
1.1G    dump/pxelator/audit.bson
4.0K    dump/pxelator/system.indexes.bson
76K     dump/pxelator/system.profile.bson
1.1G    total

So I switched PXElator to use MongoDB as storage. I never pushed anything in production after just one day of testing it, but first query speedup from 30 min to 30 sec, and ability to cut it down to 3 sec if I added index (which took about 13 sec to create) is just something which provides me with powerful analytical tool I didn't have before.

OmniKey CardMan 5321 supported by librfid

2010-01-19T15:38:31Z

I had OmniKey CardMan 5321 reader sitting on my desk for quite some time. First time I tried it, I had problem with propitiatory binary driver which expected pcscd to be compiled without hal support to make it work.

Fortunately, we now have pcsc-omnikey package in Debian which should make usage of this reader much easier. But, I really wanted more low-level implementation, allowing me to muck with cards without need to pass through whole smart card stack (since I'm really only interested in RFID part of this reader).

So, I did some searching and found out that librfid - A Free Software RFID stack implements support for this reader, so here is a quick overview of how to get started:

# build dependency
dpavlin@klin:/rest/cvs/librfid$ sudo apt-get install libusb-dev

# checkout source
dpavlin@klin:/rest/cvs$ svn co https://svn.gnumonks.org/trunk/librfid/
dpavlin@klin:/rest/cvs$ cd librfid/
dpavlin@klin:/rest/cvs/librfid$ ./autogen.sh

# build
dpavlin@klin:/rest/cvs/librfid$ ./configure --enable-ccid
dpavlin@klin:/rest/cvs/librfid$ make

Now we can test if our reader is working:

dpavlin@klin:/rest/cvs/librfid$ sudo ./utils/librfid-tool -s
lt-librfid-tool - (C) 2005-2008 by Harald Welte
This program is Free Software and has ABSOLUTELY NO WARRANTY

initializing librfid
opening reader handle OpenPCD, CM5x21
No OpenPCD found
scanning for RFID token...
Layer 2 success (ISO 15693):  eb 6e 77 1f 00 01 04 e0

And, that's not all. We can also read content of our tag:

dpavlin@klin:/rest/cvs/librfid$ sudo ./utils/librfid-tool -r -1
lt-librfid-tool - (C) 2005-2008 by Harald Welte
This program is Free Software and has ABSOLUTELY NO WARRANTY

initializing librfid
opening reader handle OpenPCD, CM5x21
No OpenPCD found
Layer2 init ok
Layer 2 success (ISO 15693)[8]: ' eb 6e 77 1f 00 01 04 e0'
block[  0:00]sec:0x8 data(4):  04 11 00 01
block[  1:01]sec:0x8 data(4):  31 33 30 32
block[  2:02]sec:0x8 data(4):  30 32 39 37
block[  3:03]sec:0x8 data(4):  31 30 00 00
block[  4:04]sec:0x8 data(4):  00 00 00 00
...
block[ 26:1a]sec:0x8 data(4):  00 00 00 00
block[ 27:1b]sec:0x8 data(4):  57 5f 4f 4b
no data(read_block(28)>> -1)

It's exactly what I was looking for: ability to do low-level block transfer with RFID card.

This is great news since I don't have to carry bulky 3M reader and antenna with me to conferences to demonstrate RFID. Since I didn't find librfid first time I searched for software to drive this reader, I hope that this post will be helpful to someone.If you intend to buy RFID reader, take a look at OpenPCD instead of this one :-)

Agile web development in 2010

2010-01-14T22:10:22Z

I have been developing web application since 1995, and things have changed a lot since than. Back in the old days, we tried to render good in text-only browsers and machines where much slower back than. Why it that important? I had interesting problem today: write csv file filtering application in an hour.

Basically, I had one hour to make semi-formated csv file somehow searchable. It had sub-totals, and you wanted to type some words and search for then within each section (they all have to be in same section to make section visible as result). Since data is really textual report from SQL database, it seemed somehow logical to put it back into database and write complex SQL query to return just parts of it. Or store it in full-text search. In 1990 or so...

But, let's take a look in directory: I have 600K csv file. I decided to write simple cgi wrapper which understands sections (and ignores rest), basically streaming csv converted to html table back to client (which provided nice incremental rendering for bigger result sets). I had simple input-box, submit button, html table application in one hour. And it was used 522 times today, speeding up manual search and retrieve work with papers.

Let's think about this situation again. There is only one simple rule for agile: scale your solution to your problem, not to your technological preferences. Rest of the day, I spent fighting utf-8 encoding (in 2010 still, sigh!), aligning columns and formatting floating numbers (with few style="color:gray" here and there) and added highlight to lines within section which match search words.

At the end of the day, I added a simple summaries on the bottom. Not bad for 1 hour/1 day application.

Protocol analysts of DRAC remote console

2010-01-10T19:19:38Z

As you know by now, I have been playing with Dell's remote consoles in hope that I will be able to connect from my Linux to Dell's RAC reliably. Currently, I have to run Windows XP with Internet Explorer and Java in kvm to have access to my servers, and that's clearly not reliable combination.

DRAC is PCI card which is presented to system as VGA which then transfers screen updates over the network to client. It also allows virtual media, but in a sense, it's mix-up of http over ssl and few propitiatory protocols:

443 - https interface
3368 - virtual media (proprietary)
5900 - keyboard and mouse (ssl encrypted)
5901 - video redirection (optionally ssl encrypted)

It's very strange that all documentation calls 5900 video redirection port and 5901 keyboard/mouse redirection when all traces of traffic between client and server clearly show that ports are swapped in implementation.

Did you notice ssl encrypted keyboard/mouse channel? I first decided to tackle this problem with well known SSL man in the middle approach. I decided to use simpliest possible approach first using something like:

apt-get install stunnel

openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem

# https mitm
stunnel -p cert.pem -d 443 -r 5443
stunnel -c -d 5443 -r 10.60.0.100:443

# 5900 mitm
stunnel -p cert.pem -d 5900 -r 5999
stunnel -c -d 5999 -r 10.60.0.100:5900

and than recoding all output using wireshark:

sudo tshark -w /tmp/drac.pcap -i any 'port 5999 or port 5901 or port 5443'

This allowed me to capture all unencrypted traffic into single pcap file which proved very useful for initial protocol analysis using wireshark. In short, you have to do following:

make https connection to https://drac/cgi-bin/webcgi/winvkvm?state=1 and acquire vKvmSessionId console redirection authentication key
connect to keyboard/mouse port 5900 forcing SSL_cipher_list to supported RC4-MD5 cipher and send vKvmSessionId
connect to video port 5901

Finding supported cipher for communication between us and server was a real problem. They are using openssl-0.9.7f and I had to downgrade all the way to Debian woody to make stunnel work. Same problem is visible with latest firmware update for DRAC where Active X plugin doesn't have old enough configuration in SSL handshake and doesn't work any more. Java plugin, on the other hand, provides much more cipher options, so one of them still works. ssldump was very useful for finding such problems.

Fortunately, kost was much more persistent than me, and he found out that adding 'SSL_cipher_list' => 'RC4-MD5' will force supported cipher. Armed with that new finding, I was able to modify kost's ssl mitm script up to the point where I can see decrypted key-presses, mouse movements and video settings. Hack, I even wrote drac-vkvm.pl async client which does steps outlined above.

All is not well, unfortunately. When sending authentication request, we need vKvmSessionId which we get from web server, but packet which is sent contains also two bytes which change with session. I haven't been able to figure this part out, and since same two byte sequence is needed to open video channel (to see VGA output) so I'm stuck.
Bytes don't look like crc16, and source code doesn't provide any hints about secondary 16-bit auth info. It seems that client calculates it somehow, since both connections close when I try to send different values for it.

I could write session recorder, but that isn't terribly useful, because it still forces me to use Windows+Java setup to access my console. I will collect usefull snippets about Dell's RAC protocol on wiki.

Dell's RAC support under Linux == drek

2009-12-30T17:21:12Z

Back in 2007 I wrote draft of blog post about semi-working solution for remote console on Dell's servers. I never finished it, but now I got a few more Dell servers and I'm not impressed. Linux support is even worse. Drek is shit in Croatian, btw.

First, update your RAC to latest firmware. It's a challenging task on Dell's web (I did it three times allready and still I can't remember procedure, so I'm documenting it here): search for drac 4 firmware (DRAC 4/I) and select newest version available (due to ranking sorting this can be hard task, select date sort). If you have DRAC 5, searching over Dell's site isn't much help. But, you can hop directly to ftp site and select file with name f_drac5*.exe.

Now download hard-drive format (it's self-extracting zip) and extract firmware:

dpavlin@t61p:~/Downloads/dell/2800$ unzip f_0603v17en_d4.exe 
Archive:  f_0603v17en_d4.exe
  inflating: firmimg.dm1             

dpavlin@t61p:~/Downloads/dell/2900$ unzip f_drac5v145_A00.exe 
Archive:  f_drac5v145_A00.exe
  inflating: firmimg.d5

Upload firmimg.* via web interface (if you have really old version of RAC, you might need IE to do this, but recent versions from last year or two work also with Firefox. Firmware update doesn't work with Google Chrome). This is just the beginning of Internet Explorer dependencies, so get used to it.

Good idea is to ping your DRAC while uploading firmware. While flash is programming, you will get interruption in ICMP echo reply, but that's not a reason for worry. It will come back.

After all that, you still won't get Linux support. At least not with recent Firefox. Lousy work, Dell! How are you accessing your Dell servers from Linux?

Dump streams using mplayer and at

2009-12-28T17:39:22Z

As you probably know, this part of the year is mostly about Chaos Communication Congress with a lot of great lectures, but you might want to record a stream or two.

When I want to schedule something at specific time, I usually use at but mplayer is picky, so at the end I used following script:

#!/bin/sh -x

mplayer http://streaming-26c3-wmv.fem-net.de/saal$1 -noconsolecontrols \
   -dumpstream -dumpfile saal$1-`date +%Y%m%dT%H%M%S`.wmv &

This will allow mplayer to detach from console, so it will work in backup. Happy hacking!

Update: stopping streams manually is not fun, so here is improved script which has three arguments: room number, from time and to time

#!/bin/sh -x

# usage: record.sh 1 13:45 15:15

room=$1
from=$2
to=$3

url=http://streaming-26c3-wmv.fem-net.de/saal$room
file=`date +%Y%m%d`-$from-$to-saal$room

echo "Record $from - $to $url -> $file"

echo "mplayer $url -noconsolecontrols -dumpstream -dumpfile $file.wmv" | at $from
echo "kill \`ps ax | grep $file\`" | at $to

If you need to see all jobs scheduled, use atq. However, there is no easy way to correlate which task will execute in which time slot in atq output, so I wrote small atq.pl script which displays commands which will be executed (you will need to run it with root privileges).

LDAP server application integration with virtual LDAP

2009-12-13T22:44:38Z

You have your new shiny application, and LDAP server on the other side. Easy as pie. What can go wrong?

you use e-mail as login, and application assumes that logins don't have domain in them and allows you embedding of whole login into DN
application can import various interesting fields using LDAP, but you have data somewhere else, and it doesn't really belong into your LDAP
you need to provide subset of data in your database as LDAP server to application

I had written about my saga with LDAP about augmenting LDAP search responses and exposing RDBMS data as LDAP server. But today, I added rewrite of bind, so now I can use unmodified Koha, and all needed quirks for my AAI@EduHr schema are outside application.

This made my attempt to virtualize LDAP almost complete so I created a project page on Ohloh. I will write small updates about status there, so If any of this is interesting to you, hop over there.

xxd: hexdump something quickly

2009-12-09T18:33:08Z

We have all been in this situation: you have two files which are subtly different, and you need to take a quick look in them. Let me point you towards xxd which is part of vim. Users of other editors can stop reading now! :-)

dpavlin@t61p:~/t61p/utf-8-accents$ xxd good
0000000: 506f 7669 6a65 736e 6f20 6172 6865 6f6c  Povijesno arheol
0000010: 6fc5 a16b 6920 766f 6469 c48d 2070 6f20  o..ki vodi.. po 
0000020: 4f73 6f72 750a                           Osoru.
dpavlin@t61p:~/t61p/utf-8-accents$ xxd bad
0000000: 506f 7669 6a65 736e 6f20 6172 6865 6f6c  Povijesno arheol
0000010: 6f73 cc8c 6b69 2076 6f64 6963 cc8c 2070  os..ki vodic.. p
0000020: 6f20 4f73 6f72 750a                      o Osoru.

Now, if I could only find peace of perl code which puts accents back to characters...

HTML5TV: video archive with slide sync without plugins

2009-12-01T12:22:29Z

Last weekend we had Nothing will happen event and I did small presentation about HTML5TV which is:

video annotation interface based on mplayer with slide overview
web interface using HTML5 tag and some JavaScript

I have collected some of presentations at http://html5tv.rot13.org/, so hop over there and have a look. If you have more interesting presentations, you can easily fork HTML5TV at github, annotate presentation and submit it for inclusion (or make your own archive if you prefer that). Have fun and let me know what you think about it.

x2x: share mouse and keyboard over ssh

2009-11-23T13:05:42Z

For quote some time I have been using x2x to have single keyboard and mouse for multiple machines (EeePC or third monitor at work), but I had problem with selection buffers. x2x was written for scenario where all X servers are listening to network connections, so if you had two machines klin and t42 and you wanted to use keyboard and mouse on klin you would have to do something like this:

dpavlin@t42:~$ xhost klin

dpavlin@klin:~$ x2x -to klin:0 -west

However, recent Debian systems are not listening to TCP connections, so you will have to modify xserverrc for this to work:

dpavlin@t42:~$ cat /etc/X11/xinit/xserverrc 
#!/bin/sh

exec /usr/bin/X11/X #-nolisten tcp

However, it's much nicer to depend on ssh X11 forwarding to provide encryption using:

dpavlin@klin:~$ ssh -X t42 x2x -to :0 -west

To finish this post, I have to include small patch which allows to switch copy/paste direction (which I don't need any more, but it proved useful in at least one case):

diff -r 2ce6789a43da x2x.c
--- a/x2x.c     Tue Mar 18 22:47:34 2008 +0600
+++ b/x2x.c     Tue Nov 24 14:08:16 2009 +0100
@@ -1125,7 +1125,7 @@
       if (doSel) {
         // pDpyInfo->initialClipboardSeen = False;
         pDpyInfo->winSelText = NULL;
-        pDpyInfo->hwndNextViewer = SetClipboardViewer(pDpyInfo->edgewindow);
+        pDpyInfo->hwndNextViewer = SetClipboardViewer(pDpyInfo->bigwindow);
       }
 
       pDpyInfo->onedge = 0;

Change gtk font size without userChrome.css

2009-11-18T16:38:03Z

If you have any objection your X11 setup (especially if you are not running Gnome or KDE) it's probably: Firefox menu font size is too big! However, that's really not right complaint. If you create your own userChrome.css and install it in correct location, you will notice that rest of applications still won't look good. But, Firefox was biggest problem, so you move on...

But, solution is really simple:

dpavlin@t61p:~$ cat .gtkrc-2.0 
gtk-font-name = "Verdana 7"

This will change all GTK application's (including FIrefox, OpenOffice.org and others) to sane font size for huge monitors with high DPI settings. It's a single line fix to all your problems!, take a look:

X11 selection, clipboard and how to use them

2009-11-15T20:31:47Z

One of first differences when you start using X11 is that you can just select any text on screen and press middle mouse button to paste it back. But, after a while you end up in situation where you did ctrl+c in application, and somehow you expect that middle mouse button will paste it back.

X11 implementation of copy/paste is much more complicated, as you might read from excellent X Selections, Cut Buffers, and Kill Rings.

Most of the time, having two different clipboards is very useful. Think about copy/pasting url and title at the same time from one web page to another (like I do often when writing blog posts like this one).

So, how to we manage buffers more efficiently? First, install xclip and try it out:

dpavlin@t61p:~$ xclip -h
Usage: xclip [OPTION] [FILE]...
Access an X server selection for reading or writing.

  -i, -in          read text into X selection from standard input or files
                   (default)
  -o, -out         prints the selection to standard out (generally for
                   piping to a file or program)
  -l, -loops       number of selection requests to wait for before exiting
  -d, -display     X display to connect to (eg localhost:0")
  -h, -help        usage information
      -selection   selection to access ("primary", "secondary", "clipboard" or "buffer-cut")
      -noutf8      don't treat text as utf-8, use old unicode
      -version     version information
      -silent      errors only, run in background (default)
      -quiet       run in foreground, show what's happening
      -verbose     running commentary

Help output is probably a little verbose, but just remember following things:

-in is default
-out displays content
default selection is primary (which is content you marked on screen)
-selection clipboard is only other thing you need to rember

So, armed with that knowledge, here are couple of examples:

# display selected text
xclip -out

# display clipboard (copy/paste in applications)
xclip -out -selection clipbaord

# transfer selection into copy/paste buffer
xclip -out | xclip -selection clipboard

# transfer selection into copy/paste buffer
xclip -out -selection clipboard | xclip

Last example is useful if you want to paste something back in terminal and it somehow ended up in copy/paste buffer, following is Zotero example:

Just let your imagination go wild: what if you have only textarea and you want to do some massive editing for which vi whould be better? You can install excellent It's All Text! Firefox extension but, you could also write something like this:

xclip -out > /tmp/$$.txt && vi /tmp/$$.txt && xclip -in -selection clipboard < /tmp/$$.txt

You could bind that to a key in your window manager or start different editor if you wanted to... or I guess live can be simple again if you just install autocutsel

Few days after I wrote this blog post, I had interesting problem. I was trying to decrease font size in firefox and I found instructions which included filename and content itself. I copied content with Ctrl+C and selected filename and used this little gem:

xclip -sel clipboard -out > `xclip -out`

Edit MARC records in your favourite editor

2009-11-07T12:06:59Z

MARC is binary structured file format used in libraries and occasionally you need a way to quickly edit records in it.

There are quite a few tools available for editing MARC records, but none of them are really easy to use. Few days ago, I overheard conversation in which somebody said you can't just edit in your editor, because you have to fix leader field afterward. That got me thinking...

MARC format is really quite interesting, designed to be easily used on tapes, so it has a bunch of pointers to exact record positions which change if size of field values change.

Editing binary MARC directly in editor isn't really possible because you would have to fix leader (which has size of whole record which allows easy skipping on tape) and all pointers to start of each field manually. I did some of that while writing MARC::Fast, but I wouldn't recommend that to anyone.

MARC::Record already has utility called marcdump which will convert MARC into human readable format:

dpavlin@t61p:/srv/webpac2$ marcdump t/data/marc.iso > dump.txt
t/data/marc.iso
dpavlin@t61p:/srv/webpac2$ head dump.txt 
LDR 01002nam0 2200289   45  
001     800328901
100    _a19800328d1999       y0hrvy0191    ba
101 1  _ahrv
102    _aHR
105    _aa   m   000yy
200 1  _aMjerne nesigurnosti rezultata mjerenja odstupanja od oblika i polo`aja
       _emagistarski rad
       _fZdravko Bar{i}
       _gvoditelj Vedran Mudronja

I can easily edit this in my favorite editor (which is vim if you really wanted to know) to replace CROASCII characters with utf-8 equivalents of something like that.

So, I wrote dump2marc which parse marcdump output and converts it back to MARC file format like this:

dpavlin@t61p:/srv/webpac2$ ./bin/dump2marc.pl dump.txt > dump.marc

It will also dump parsed records to STDERR which you can redirect to file using 2>dump.txt.parsed.

If you dump generated MARC with marcdump you can use diff to view changes:

dpavlin@t61p:/srv/webpac2$ ./bin/dump2marc.pl dump.txt > dump.marc \
  && marcdump dump.marc > dump.marc.txt \
  && diff -u dump.txt dump.marc.txt

If we wanted to track changes, add dump file (and MARC if needed) to git and using it to produce diffs would be one possible solution. I know that by this point, you are already rolling your eyes thinking: this is command line editing for MARC.

Same work-flow could be turned into web application using simple upload button to upload MARC, and then opening textarea with dump which is converted back to MARC ready to download after submit. It would be sane to automatically add every revision to git on server-side and provide some kind of rollback to some last version over git too, maybe over gitweb?