August 2007 Archives

After four (4) weeks of waiting from the moment credit card has been charged, it finally arrived. It seems that shipping GSM-like devices to Croatia as person (as opposed to company) isn't something envisioned by Hrvatske agencije za telekomunikacije so we had to get additional papers (most of which I don't have) and in the end they settled with hardware specification of Neo1973 from OpenMoko wiki.

I must say that people all over the process where helpful and nice: we had to communicate much more with them then I hoped, but in the end it worked out o.k.

And now several obligatory pictures (hopefully not redundant :-)

After that I got rootfs flashed and I now have new (semi-functional) phone :-)

root@fic-gta01:~$ cat /proc/cpuinfo 
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 132.71
Features        : swp half thumb 
CPU implementer : 0x41
CPU architecture: 4T
CPU variant     : 0x1
CPU part        : 0x920
CPU revision    : 0
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format A
Cache format    : Harvard
I size          : 16384
I assoc         : 64
I line length   : 32
I sets          : 8
D size          : 16384
D assoc         : 64
D line length   : 32
D sets          : 8

Hardware        : GTA01

Revision        : 0240

Serial          : 0000000000000000

root@fic-gta01:~$ free

             total       used       free     shared    buffers     cached

Mem:        126644      52604      74040          0         84      39100

-/+ buffers/cache:      13420     113224

Swap:            0          0          0

root@fic-gta01:~$ df

Filesystem           1k-blocks      Used Available Use% Mounted on

/dev/mtdblock4           62576     42572     20004  68% /

tmpfs                       40         0        40   0% /mnt/.psplash

/dev/mtdblock4           62576     42572     20004  68% /dev/.static/dev

tmpfs                     2048        64      1984   3% /dev

tmpfs                    63320         4     63316   0% /tmp

tmpfs                    63320       116     63204   0% /var/volatile

tmpfs                    63320        16     63304   0% /dev/shm

tmpfs                    63320         0     63320   0% /media/ram

/dev/mmcblk0p1          495168         0    495168   0% /media/card

We have few mp3 players which no longer work, but are still under warranty. So idea was to pick another device (which will hopefully work longer). However, on-line shops leave a lot to be desired if you want to just do quick filtering of data.

As a very fortunate incident, I stumbled upon Exhibit from SMILE project at MIT which brought us such nice tools as Timeline and Potluck.

So, I scraped web, converted it to CSV and tried to do something with it. In the process I again re-visited the problem of semi-structured data: while data is separated in columns, one column has generic description, player name and all characteristics in it.

So, what did I do? Well, I started with CPAN and few hours later I had a script which is rather good in parsing semi-structured CSV files. It supports following:

• guess CSV delimiter on it's own (using Text::CSV::Separator)
• recognize 10 Kb and similar sizes and normalize them (using Number::Bytes::Human)
• splitting of comma (,) separated values within single field
• strip common prefix from all values in one column
• group values and produce additional properties in data
• generate specified number of groups for numeric data, useful for price ranges
• produce JSON output for Exhibit using JSON::Syck

So how does it look?

In the end, it is very similar to the way Dabble DB parses your input. But, I never actually had any luck importing data into Dabble DB, so this one works better for me :-)

This will probably evolve to universal munger from CSV to arbitrary hash structure. What would be good name? Text::CSV::Mungler?

This is a first post in series of posts which will cover one hack a week on my blog. This will (hopefully) force me to write at least one post a week on one side, and provide some historic trace about my work for later.

I'm quite pleased to announce that my efforts to decode .amv format supported in most Chinese mp4 video players is over: I have working decoder :-)

First, let's clean up misconceptions: AMV is very smiliar to mjpeg video (frames are jpeg frames without quantization table, same as mjpeg frames) and IMA ADPCM is used to encode audio (not mp3 frames as often mentioned). Audio format isn't exact IMA ADPCM because it include 8 bytes at beginning of each frame which are used to seed ADPCM decoder.

Let's for a moment consider hardware of those players: there is Z80 and DSP. When playing mp3 files, DSP is used to decode audio, but when playing video DSP is used to decode jpeg frames (this also limits maximum frame rate and picture size). So choice for ADPCM was logical since I suppose that Z80 decodes audio while playing movie.

All this is done using clean-room reverse engining, which means that I only used output from Windows encoder. No disassembling of Windows code was used (hey! why would someone waste his time on that?).

So current decoder decode frames into jpeg images (flipping them using jpegtran) and decode audio to 16-bit linear .au file and use that to produce final .avi movie using ffmpeg.

As a result of this, there is enough knowledge embedded in script (and in this post) to create encoder. I probably won't have time for this in next few weeks, but my idea is to make perl script which will open any movie file supported by ffmpeg, resize it on-the-fly and than mungle avi output stream to produce amv. You can think of it as cp which on-the-fly converts movie to your player.

If somebody knows if this ADPCM variant is supported by ffmpeg (reading source didn't help much) I would be grateful for info.

Great time saver... But when you see something like:

...there is something wrong with you :-)

On the bright side, it's isn't eating too much memory (I have two more windows, one with 30 tabs and nother with only 4):

$ ps axv | grep ice
23661 ?        Ssl    3:18     45 11540 477799 339092 21.8 /usr/lib/iceweasel/firefox-bin -a firefox

And, now... let's close those tabs!

Relevant parts:

This message has been automatically generated with regard to the
progress of your order at the OpenMoko online store (http://direct.openmoko.com/).

Your credit card has now been charged by the following amount:

Subtotal: $300 USD

Shipping: $102.49 USD

Total: $402.49 USD

Please note that this amount might be less than what was originally

mentioned in the webshop, since we meanwhile got better shipping rates!

This means that we will now send out your order ASAP.

You will receive another status update once the order has been sent out.

Shipping is really less than first estimate of $147, and I'm waiting for second mail :-)