Did you just migrate to SubVersion? I did and I'm not looking back. However, most of on-line collaboration sites for projects (like SourceForge and even Tigris) doesn't support SubVersion repositories yet. Of course, you can host your own (as I do) but wouldn't it be nice that your project has CVS checkout ability also?

I thought about that also (especially since my WebPac project has CVS repository on SourceForge). After searching for such a tool, I found blog from Tollef Fog Heen which has a nice tool, but a little to basic for my need.

So, I started hacking. And created sledgehammer (in Tollef's own words) tool for this task: svn2cvs. It's on the way to FreshMeat as I write this...

Did you ever wanted to search your Mailman archive? Why don't you use SWISH-E?

Well, you couldn't because swish comes just with index_hypermail.pl but after one night of hacking, I present to you: index_mailman.pl. I will test it a bit, and if everything goes well, I will probably submit it for inclusion in swish-e distribution.

Reading How Microsoft Lost the API War by Joel Spolsky had profound influence on me.

I decided to re-implement WOPI, web poll generation software which I was working on since 2001 (first commercial, and later available under GPL). With new, totally web-based user interface. Try it and tell me what you think. This is HTML and Java Script only, so save doesn't work!

I have been looking around for Linux solution to create web page thumbnails. And, I found nice weblog entry which gave me push in right direction. However, I wasn't quite happy about hard coded values (sleep especially), so I started re-writing it.

Result is webthumb, a simple shell script with small number of prerequisites:

• vncserver
• mozilla-firefox
• HEAD (part or LWP perl library, it can be commented out)
• nc (netcat, to simulate web server)
• xwininfo (standard part of X-clients)
• xwit (not so standard command, but very useful)
• import (part of ImageMagick package)

Improvements include signal handling (so you can press CTRL+C to abort), support for dynamic display from vncserver (which enables more than one webthumb to work at the same time), fixed Window ID parsing (and more portable, I hope, please report problems), and special JavaScript+frame set+http server in shell hack to enable wait for page to load.

Aside from that, I must warn you that it won't work well with pages which have meta refresh or 304 redirect as first thing they do. Just point webthumb to redirected page and everything will be well. Here are also some examples: cnn.com, www.ffzg.hr, www.foi.hr, www.rot13.org/~dpavlin, www.linux.com, www.linux.hr, www.monitor.hr, www.pliva.hr, www.perlmonks.org, www.plivamed.net, www.plivazdravlje.hr, www.postgresql.org, prognoza.hr, www.rot13.org.

Possible usage include creating of animation from sites that change often, showing web page in different resolutions easily, or showing pop-up thumbnail image of site before user click on external link (I plan to do this with my home page).

Update: change link to subversion version (always latest one)

I just added nice JavaScript filter to my search page. You can filter titles of pages by just typing and hit search to search whole content.

Also, I fixed jsFind so that B values larger than 10 work (oh, I forgot to base62 jsFind xml files). One of those brown-bag bugs.

Who said that combobox can't be done with html? As a result of another little project and with help from Matko Andjelinic, here is something: real combobox! Have patience while it loads (just index with 10000 words have 260k), but it works! And it works fast. Design sucks (especially icon for drop down), but you will forgive me that.

My blog also got spammed. I hate that. But, I'm still not disabling comments, so feel free to comment...

I just released first version of Fuse::DBI in the wild. Let's see what users of WebGUI think about it.

There isn't anything specific to WebGUI in this module (other than WebGUI example, that is). So, if you use Fuse::DBI for something else, I would like to know how.

This is 42^nd post on this blog. It should mean something...

I'm working currently on two perl modules which will be very useful in next
version of WebPAC:

• IsisDB - perl-only alternative to OpenIsis perl bindings which are depreciated (at it seems)
• Swish::Split - experimental module to enable indexing from perl using swish-e. It uses experimental incremental mode which is available in swish-e for a while, but it still has some problems.

Since my network at home doesn't work (you just have to live wireless!), I
won't blog much for the rest of this year. On the bright side, there isn't
much days left in this year anyway...

Well, it's released. Module which used to be called IsisDB is now Biblio::Isis. It also includes part of CDS/ISIS manual about file format which might be useful even if you don't plan to use this module.

This module was my first journey into decoding file formats (well, not exactly first, but first which had to be released to public). I did a good thing and asked nice people at perl4lib to test it first. And guess what? It didn't work on Mac OS X. I should have known better than to use perl's unpack with l if I wanted cross-platform compatibility. But, over-presence of Linux boxes with x86 processors produced strange problem for me: PowerPC which powers Mac is little-endian while all boxes which I have access to are big-endian. Hum, what to do?

Few weeks ago, I stumbled upon PearPC. Together with OpenDarwin it's a great platform (and little-endian :-) to test your code on BSD-based Unix which runs totally different than your local Linux box. Thanks to all hard working developers from both project for that.
Have in mind that PearPC is really slow. Something like 68k-based Mac. However, for simple perl development it's fast enough. And it solved my problem. Highly recommended.

My pet project is on pgFoundry. pgswish allows integration of search results from swish-e index and structured data from PostgreSQL. It's a function which returns multi-row data in PostgreSQL which is swish-e search result.

However, I also posted e-mail message to swish-e mail list and Bill Moseley had some great comments. If you know how to write PostgreSQL C-functions I would be great-full for another pair of eyes on this project. My C is rusty.

Did you ever needed group chat in ICQ? I did. Thanks to great Net::OSCAR it was a joy to write (but, it took me quite some time to find Net::OSCAR in the first place).

On unrelated note, I started forbidding comments on my blog. Sorry about that, but I don't have time to change blogging software now, and amount of junk that I receive in comments is several times higher than real comments. So, if you have a comment, feel free to send me e-mail at dpavlin(at)rot13(dot)org.

With latest changes going into swish-e CVS repository, swish-e doesn't segfaults any more when used with --enable-incremental option. That is great news, and enables me to continue working on SWISH::Split module, so here is first release (local, not on CPAN yet).

Hyper Estraier, new version of previously mentioned Estraier has a very good API. So, I wrote pgestraier function for it. Now you can query Estraier indexes directly from PostgreSQL.

It's beta, yada, yada, if it breaks you get to keep both parts. However, it's extremely useful if you want to (left/outer) join index results with data in PostgreSQL.

Search::Estraier will be perl API to Hyper Estraier. It's written using excellent Inline::C

Grand plan is to use pgestraier from PostgreSQL to query Estraier index and Search::Estraier to create index. This would allow to combine structured data in RDBMS with semi-structured data in full text index. Additional normalized tables can be created using materialized views in PostgreSQL, and if all goes well, it will be part of WebPAC version 2 which will be universal hybrid (structured/full text) storage.

Update: I had good luck to find already working perl bindings for Hyper Estraier at MATSUNO Tokuhiro blog. Thanks a lot! So, work on Search::Estraier is suspended.

I just added node API to HyperEstraier perl bindings. It's my first real project in C++, so be kind to it :-)
All changes are available at Subversion repository and if I didn't do something stupid it will be included in upstream version, I hope. Next step: add node API to pgestraier...

Well, this was fast. In last post I promised to work some more on pgestraier and well, I did. It now supports node API, and generally I'm quote happy with it. It even has a proper home page.

I needed this for a long time, so finally I wrote it. It's a birday present to... well... me. Geeks are strange, right?

After too long time I finally added svn checkout path to bottom of every page on svn.rot13.org.

So, if you are using Subversion or even better svk you can be in sync with latest development.

All it took was modification of footer SVN::Web template like this:

[% UNLESS nonav %]
    [% svn_url_parts = [ 'svn://svn.rot13.org', repos, navpaths.join('/') ] %]
    [% svn_url = svn_url_parts.join('/') %]
    <div>You can checkout this using <tt>
    <a href="[% svn_url %]">[% svn_url %]</a>
    </tt></div>
[% END %]

I'm somewhat proud to announce that current version of pgestraier now includes consistency triggers which will keep Hyper Estraier index up-to-date with data in your database.

That, coupled with ability to create full-text indexes easily, just by running helper script on database, makes pgestraier powerful solution if you need fast full-text indexing with ability to off-load search to another machine (thanks to Hyper Estraier P2P architecture) or need perfect N-gram search results.

This project might also help people who are porting applications which use MySQL full-text search to PostgreSQL (actually, it's it going to be used just for that).

Real trunk of development is in Subversion, and CVS repository at pgFoundry is just a mirror copy. Enjoy it, while I prepare to leave for seaside.

During last week of my holiday, I had a few days to spare, so I tried jQuery. It's a really cool JavaScript library.

To prove my words, I present to you following:

• jquery-debug - debugging panel for jQuery (on bottom right in demos)
• Tag auto-completer implemented using jQuery (see demo)
• jQuery implementation of cute no-pager idea (demo with CPAN search)

No pager is actually a small project consisting of novel idea to share one hash between perl CGI (or stand-alone server, but it doesn't exist yet) and JavaScript. It does so by implementing small subset of HTML::Mason (well, only single-line <% something %> markup), just to look familiar (or syntax-highlight correctly :-)

Proper explanation of that would have to wait for some serious documentation writing...

Update: new upstream version of jQuery (1.0.1) fixed problems in Safari on Mac. Isn't that cool?

I just added third sub-project my SocialText wiki import tools, so it's fair to write a bit about it.

With those tools (and with some help from strut, socialtext command-line rest tool) you can fairly easily convert your kwiki, phpwiki or pod documents into socialtext markup. If you want to know more, or want to leave comment, please do so on SocialText Open Source Software wiki page about this tools.

I just finished set of shell scripts which test failover and replication using Slony-1 and PostgreSQL.

It's somewhat Debian specific (using pg_ctlcluster to control PostgreSQL instances), but other than that it should work on any platform. This is essentially codification of requirements for HA testing I had to do, and I hope that this automatic demonstration would be valuable to others to test their specific requirements. Without this throwaway model, I wouldn't be able to test production database failover, because there is just too much steps which had to be done in specific order.

So, my fascination with testing expanded from development to deployment.

So, it seems that having all my projects in subversion just isn't enough for me, so here is a little gem: svn-overlay.sh script which will, when placed in root of your repository, overlay all directories in that repository on your root directory (/).

Why would that be useful, you might ask? Well, did you ever wanted to track changes to configuration files? How about keeping in sync config files on two different machines? Does installation of your favourite project overwrite your custom changes (or additions)?

All that can be solved simply by creating svn repository and keeping your changes in it, like this:

svnadmin create /srv/svn-config
svn mkdir -m "track /etc" file:///srv/svn-config/etc/
svn mkdir -m "track /etc/init.d" file:///srv/svn-config/etc/init.d
svn import -m "add overlay script" svn-overlay.sh file:///srv/svn-config/svn-overlay.sh

cd /srv
svn co file:///srv/svn-config local-config
cd local-config
sudo ./svn-overlay.sh

After this you can rm -Rf /srv/local-config if you don't want to have another copy on your disk (it was needed so that svn-overlay.sh would know repository path and that might get fixed in some future version).

After that, your /etc and /etc/init.d directories are under svn control. You will probably have to use sudo to add files to svn (or fix permissions of .svn directories), but other than that it's simple svn add something, svn commit, svn diff routine.

I don't know why I didn't came up with this earlier. I remember reading about keeping of your home directory in subversion, but it all seemed so complicated. Only thing that I can object is those .svn directories. svk might be solution for that, but in some other post...

I will be working on battery for most of next week, so I spent some time tweaking my setup. I have been running 2.6.21 because of my efforts to make gflrx play nice with CONFIG_PARAVIRT, so I had tickless kernel needed for powertop. To my horror, most of interrupts on my laptop was created by PostgreSQL (which I will stop when using battery) and ACPI! And that's because I'm using dwm with a primitive shell grep/sed pipe to produce status.

So, polling ACPI every 3 seconds (which is reasonable refresh time for me, even for clock) is too much. And than I started dreaming: network traffic would be nice. And a disk! Battery status when (dis)charging goes without saying. So, in the end, I also added a temperature, and got something like this (when on power):

2007-05-27 22:15:40 | 0.30 0.13 0.06 |   5M D 1k   |  32b > 54b  | 59 C

perl code it's nice and short but completely broken about estimates for charging time (how does acpi command-line tool calculate that?)

When started as stand-alone utility it can be poor man's replacement for dstat :-)

This post is about perl emulator of 6502-based machine called Orao.

It was my first computer (I attended BASIC introduction course several times so I can work on machine). I never owned one (and almost nobody did back that), since most of machines where installed in schools (like the one near to my place offering basic computer literacy courses).

When I found out that Josip Perušanec wrote Orao Emulator, I was very excited. After all, who can resist temptation of running his old BASIC programs? But, since his version runs only on Windows (to be fair, it runs also under wine), I had to write my own. Credits where credits are due: without Josip's emulator (and especially ROM images) you wouldn't be looking in screenshot above.

I wanted to write it in perl, so at first I used Acme::6502 to do processor emulation and wrote screen interface in SDL. However, that seems slowish, so in the end, I wrote perl bindings for great 6502 emulator M6502 and implemented Orao emulation as embedded perl interpreter inside CPU emulator. I used Extending and Embedding Perl from Tim Jenness and Simon Cozens as my guide, and I can really recommend this book if you want to learn perl-C interaction. If I didn't already owned pdf copy, I would surely buy both pdf and paper copy. It's so good.

Both versions can startup Orao, but pure perl version probably won't receive much love and care. As you can see on screenshot, on the left is display (Orao didn't have text mode, this is graphic display) and on the right is graphic representation of memory map. It doesn't support keyboard at this moment, but it's just a few commits away :-)

Update: Changed link to source code. It's part of the bigger VRač project, but that will have to wait another post...

I'm quite pleased to announce that my efforts to decode .amv format supported in most Chinese mp4 video players is over: I have working decoder :-)

First, let's clean up misconceptions: AMV is very smiliar to mjpeg video (frames are jpeg frames without quantization table, same as mjpeg frames) and IMA ADPCM is used to encode audio (not mp3 frames as often mentioned). Audio format isn't exact IMA ADPCM because it include 8 bytes at beginning of each frame which are used to seed ADPCM decoder.

Let's for a moment consider hardware of those players: there is Z80 and DSP. When playing mp3 files, DSP is used to decode audio, but when playing video DSP is used to decode jpeg frames (this also limits maximum frame rate and picture size). So choice for ADPCM was logical since I suppose that Z80 decodes audio while playing movie.

All this is done using clean-room reverse engining, which means that I only used output from Windows encoder. No disassembling of Windows code was used (hey! why would someone waste his time on that?).

So current decoder decode frames into jpeg images (flipping them using jpegtran) and decode audio to 16-bit linear .au file and use that to produce final .avi movie using ffmpeg.

As a result of this, there is enough knowledge embedded in script (and in this post) to create encoder. I probably won't have time for this in next few weeks, but my idea is to make perl script which will open any movie file supported by ffmpeg, resize it on-the-fly and than mungle avi output stream to produce amv. You can think of it as cp which on-the-fly converts movie to your player.

If somebody knows if this ADPCM variant is supported by ffmpeg (reading source didn't help much) I would be grateful for info.

We have few mp3 players which no longer work, but are still under warranty. So idea was to pick another device (which will hopefully work longer). However, on-line shops leave a lot to be desired if you want to just do quick filtering of data.

As a very fortunate incident, I stumbled upon Exhibit from SMILE project at MIT which brought us such nice tools as Timeline and Potluck.

So, I scraped web, converted it to CSV and tried to do something with it. In the process I again re-visited the problem of semi-structured data: while data is separated in columns, one column has generic description, player name and all characteristics in it.

So, what did I do? Well, I started with CPAN and few hours later I had a script which is rather good in parsing semi-structured CSV files. It supports following:

• guess CSV delimiter on it's own (using Text::CSV::Separator)
• recognize 10 Kb and similar sizes and normalize them (using Number::Bytes::Human)
• splitting of comma (,) separated values within single field
• strip common prefix from all values in one column
• group values and produce additional properties in data
• generate specified number of groups for numeric data, useful for price ranges
• produce JSON output for Exhibit using JSON::Syck

So how does it look?

In the end, it is very similar to the way Dabble DB parses your input. But, I never actually had any luck importing data into Dabble DB, so this one works better for me :-)

This will probably evolve to universal munger from CSV to arbitrary hash structure. What would be good name? Text::CSV::Mungler?

This is a first post in series of posts which will cover one hack a week on my blog. This will (hopefully) force me to write at least one post a week on one side, and provide some historic trace about my work for later.

Really funny name of post, isn't it? It should mean something to people who understand Croatian: it's virtual computer. This fun toy begin it's existence as Orao emulator. In the process, I wrote 6502 and Z80 emulator (actually only perl xs bindings for existing cpu emulators) and implemented Orao, Galaksija and Galeb using it.

Various machines are in different stage of usability. Orao emulation is working (without tape support), Galaksija is too slow to be useful (also doesn't have tape) and Galeb doesn't have keyboard or tape support.

I would be very grateful for hardware information about those machines if you can spare some time writing them. I'm somewhat hope that source code of this emulators might serve as historic reference how this machines where constructed.

Lately, I have also found mess project, but soon after that I also noticed that I need about six times as much to implement changes in it. C is just not my language, so actually only hope for preservation of those computers is good documentation (which is something mess community is striving also).

On the other hand, currently CPU emulation library is not license compatible with GPL so I won't probably push this to CPAN any time soon but I would love to find another implementation (PowerPC anyone?) and basically create thin layer of perl code which can express different architectures.

Having written all that now it's time to show you how to compile dam thing!

svn co svn://svn.rot13.org/VRac/
cd VRac
perl Makefile.PL
make
make orao

And you will have working Orao emulator in a second. You can also try make galaksija or make galeb.

This post was originally written on 2007-09-05 when I wanted to announce this project after playing with it over the summer. Since I never finished emulation of Galaksija or Galeb, I left it in draft mode for too long. Now that first Croatian Perl Workshop is coming along I will talk about this project, so it's just fair to announce it, so you can have preview about what my lecture will be.

Thanks to the amv-codec-tools we now have somewhat working decoder (using ffplay) and encoder (using ffmpeg) of AMV format which cheap s1mp3 players are using.

You can convert video using just Linux box which plays on hardware player (apart from glitz at the end where it sometimes reports format error), but luminance is all wrong: colors are much brighter then they should be. Oh, did I mentioned that frame is flipped so that you can't read text? ;-)

On the bright size, display on players isn't that good, so turning your player under slight angle might make picture quite good.

And player doesn't mind being turned upside-down for video playback...

Anyhow, if you wanted to quickly convert video to your player on Linux box, it has some use, at least for me...

We have part of decoder submitted into ffmpeg upstream, and home to have all part merged when we fix remaining issues. So, if you know something about jpeg, mjpeg or mpeg video, help with croma/luma issue is welcomed. Instructions for download and compile are short and sweet, so dive in!

For quite some time, I have been happy user of OpenVZ virtualization. And for almost same amount of time, I have been writing vz-tools, a set of few utilities to make work with OpenVZ a joy:

• vz-optimize.pl was written first, and it tries to intelligently increase beancounter limits for VE until there is some requires left for virtual machine.
  Best usage would be to run some kind of stress-test on your VE, and then re-run vz-optimize.pl afterwards to allocate enough resources for VE.
• vz-create.pl is second useful tool which will create new VE from upstream Debian mirror and tweak it a bit (the way I like it)
• vz-clone.pl is latest addition to set of tools and it allows to create temporary clone of virtual machine for testing (of upgrades?) or QA

While those tools work well for me, they don't have much error checking, and most of parameters in them are actually hard-coded. Since, I am having presentation about virtualization on Linux on HrOUG this year, I will probably improve this tools a bit more.

However, I'm not quite sure if this tools would be of general use to somebody else. If it is, please drop me a note, and I would love to improve them a bit more...

As I mentioned long time ago, I collected various Broadcom GPL releases and now have a git repository with ones that I was able to found.

Following versions are currently there:

• 3.00.01V
• 3.00.03
• 3.02.02V
• 3.04.01
• 3.04.02V
• 3.06.06
• 3.10.02

Tags are in flux (and will probably mirror branches anyway at some point). Right now different branches have distributions from vendors, but I have to clean a few of them which where imported before I created automatic procedure for import.

Please if you have any Broadcom bcm963xx development kit which is still not in tree, please send it in!

Version of development tree is included in version.make file at top-level:

dpavlin@host:/opt/bcm963xx$ cat version.make 
BRCM_VERSION=3
BRCM_RELEASE=10
BRCM_EXTRAVERSION=02

However, some manufacturers include own version numbers, but most of the time they leave original ones (if you apply some creative encoding to version number variants). However, to be completely sure that version isn't a duplicate you can checkout existing versions using:

git clone git://git.rot13.org/git/bcm963xx.git

Your pushes are welcomed :-)

Repository is quite big, about 600Mb, but it's on fast connection. However, ideas how to reduce that size (I don't think that there is so much changes!) is welcomed from someone who is better in git than me!

After nine months of playing with Thompson ADSL modems I have two projects which I wrote in perl, both of which is, as far as I know, first Open Source GPL implementation of those protocols.

MDAP

MDAP is protocol used by Thompson CPE devices to issue commands to CPEs (called ants) using multicast address 224.0.0.103 and port 3235 registered by IANA.

It's very cool idea, since you can connect as many devices as you have network ports or bandwidth, and allthough they all will boot with same IP address (and this create conflicts on IP network), you can still sand commands to each individual device using multicast.

Originally I developed it to flash multiple modems at once, but since I also added simple rules to change IP addresses and issue commands to devices. This essentially enables you to flash devices to some version of firmware and then change configuration a bit and have you test lab ready in few moments.

This project doesn't have a real project page yet, but you can take a look at source if you are interested...

This project also include (but doesn't use yet) simple perl BOOTP and TFTP servers, so in the end it will probably have perl-only solution for MDAP. If you just use included scripts and documentation for setup, it will use binary bootp and tftp server since this configuration was in use for at least half of the year, and I consider it stable.

CWMP

Perl CWMP server is essentially a low-level support for broken idea to communicate with devices using persistent connection SOAP with invalid XML (name spaces in some responses are just invalid) known as TR-069.

This is work-in-progress, and right now it's stable enough to work with multiple devices at once. In essence, it's protocol-violating SOAP server implementing persistent connection handling as described in TR-069 documentation (empty post, even without headers as first request, ehhh....).

Idea is to enable yo