Gingerlime » Technology

2nd edition of 1

Yoav Aner — Mon, 06 Sep 2010 23:59:18 +0000

I was really pleased when my good friend chris asked me to help him with his edition of 1 project. I guess it was exciting working on an arts project. I also liked his project because randomness is an interesting concept, particularly when it comes to computers. Put very simply: computers have trouble picking stuff at random.

Chris’s project was to pick a random image from the internet and present it on screen. It sounds fairly simple, and even slightly boring at first, but it creates a rather magical effect (and slightly addictive after a short while). The combination and sequence of images is interesting. The randomness of selection applies to many layers. How do you pick a random number using a deterministic process (like a computer)? How can you select an image at random if the search engine indexed it based on predefined criteria? What about the selection of the search engine you use? Or any other part of the process. I suppose those questions are only a part of what made this project interesting (for me anyway).

Chris initially used the ghettodriveby website, which does just that – picks a random image. So in that respsect, there wasn’t a great deal of technology or artistic innovation. Chris did however use two monitors and stretched the images on screen, to create a really cool effect – contrasting two random images. Also when displayed full screen, without any context or text, the impact is quite different, and even more so in a gallery space (I have to admit I have never seen edition of 1 in a gallery, not yet anyway).

For the impatient – here’s the temporary end result. Wait for the image to change.

So my first task was to mimic or copy ghettodriveby. The second one was to create a 2nd version for the Deptford X festival. On the 2nd version, images will be searched at random from a limited set of key words – having to do with Deptford.

Why not re-use ghettodriveby? There were a couple of reasons to re-invent the wheel here:

ghettodriveby isn’t very clear about the rights to use it, especially the way chris modified it
Chris had no control over ghettodriveby. If it goes down or changes, edition of 1 won’t work
When using a custom keyword search, ghettodriveby seemed to always return the same image – this was a real problem for Deptford X
because it’s fun to DIY

So the first task (which ended up being put aside for the deptford edition) was reasonably easy. Google provides a search API which allows you to search for images. Whilst there’s no option to search for a random image, we could easily use a random word to search for. Picking up a random word was the next challenge. I then figured out the best way to get a long list of words is wiktionary. You can download their whole database in xml format, and then extract the terms. I created a little python proof-of-concept and it seems to work well. I didn’t include the random word selection into it, but that’s quite easy.

# img_search.py

import urllib2
import cjson
import sys

# usage: img_search.py [search term]
url = (‘http://ajax.googleapis.com/ajax/services/search/images?v=1.0&q=%s&start=1′ % sys.argv[1])

request = urllib2.Request(url, None, {‘Referer’: ‘http://www.your-domain.com’})
response = urllib2.urlopen(request)
results = cjson.decode(response.read())

for i in results[‘responseData’][‘results’]:
print i[‘url’]

The problem I faced when narrowing down the search into only a few words (like ‘deptford’) was that the google api seems to limit the maximum results to 60. I have no idea why, but if the start parameter was set to anything higher than 60, google api choked. No results. Bummer. The show was meant to run for 10 days, changing the image every 10 seconds. 60 images per search term isn’t going to cut it.

One workaround was to mix the term with another word, perhaps at random. maybe do a combination of ‘deptford’ and ‘book’, ‘cat’, ‘morning’ or ‘baguette’ (especially ‘baguette’). It wasn’t elegant though, and not random enough.

So we then searched for alternatives. Yahoo BOSS turned out to provide a very similar api, and after going through the motions of getting the API key (which yahoo requires), it looked like yahoo didn’t limit the results. Well, it kinda did, but much higher than 60. I did some experiments and somewhere above 1,000, yahoo seems to always return the same image. 1,000 is manageable though.

Despite having very little experience with php, and not liking it as much as I like python, it seemed like a better choice. You can run php on almost any server. So I then had to work out how to turn my python code into php. This is the end result (more or less). This php only returns a random URL based on a set of keywords, each with its own maximum count (some search expressions might not even return 1,000 results).

function getLink() {
$search_arr = array(
1 => array("term"=>"deptford", "max"=>1000),
2 => array("term"=>"lewisham", "max"=>1000),
3 => array("term"=>"river+ravensbourne", "max"=>1000),
4 => array("term"=>"deptford+market", "max"=>1000)
);

// picking a random term, and then picking a random position for the url
$term_id = rand(1,count($search_arr));
$term = $search_arr[$term_id]["term"];
$pos = rand(1,$search_arr[$term_id]["max"]);

$url = "http://boss.yahooapis.com/ysearch/images/v1/" . $term . "?appid=&format=json&count=1&start=" . $pos;

// sendRequest
// note how referer is set manually
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, "");
$body = curl_exec($ch);
curl_close($ch);

// now, process the JSON string
$json = json_decode($body);
return $json;
}
$json = getLink();

// in case we don’t get any results – we simply try again
while (is_null($json->ysearchresponse->resultset_images)) {
$json = getLink();
}

// Using foreach – but we only get one URL
foreach ($json->ysearchresponse->resultset_images as $val)
{
echo $val->url;
}
?>

The last part took me probably the longest to work out. Primarily because (very much like php), I have no clue when it comes to javascript and all that XMLHTTPRequest stuff. The code below kinda works, but probably not in IE, and it’s probably pretty lame too. What it simply does is pull a new url for an image from the php script above every 10 seconds. It’s supposed to detect an invalid image and then reload another, but I’m not entirely sure it works actually.

// I’m using two images, so if there’s an error on one, I can quickly swap it with another. I’m not even sure it works
var img1=null;
var img2=null;

// this is the function that gets called on error
function ImgErr() {
document.myimage.src = img1;
img1 = img2;
return true;
}

// this function gets called every 10 seconds and loads a new image
function RefreshImage()
{
var xhr=null;
if (img1 != null)
document.myimage.src = img1;
img1 = img2;
try
{
xhr = new XMLHttpRequest();
} catch(e)
{
try { xhr = new ActiveXObject("Msxml2.XMLHTTP"); }
catch (e2)
{
try { xhr = new ActiveXObject("Microsoft.XMLHTTP"); }
catch (e) {}
}
}

xhr.onreadystatechange = function()
{
document.myimage.alt="Loading…";
if(xhr.readyState == 4)
{
if(xhr.status == 200)
{
if (img1 != null)
document.myimage.src=img1;
img2 = xhr.responseText;
}
}
};

xhr.open("GET", "editionof1-img.php", true);
xhr.send(null);
}

window.setInterval(‘RefreshImage()’, 10000);
script>
head>
<body>
<img src="http://editionof1.org/Editionof1.org/image-1_files/droppedImage_1.jpg" name="myimage" width="1497" height="1342" onerror="ImgErr()">
<script>
RefreshImage();
script>
body>
html>

Once upon a time

Yoav Aner — Wed, 14 Jul 2010 22:47:54 +0000

One-Time-Passwords always fascinated me. Long long time ago in a land far far away I suddenly had this idea. The idea was simple and in today’s terms pretty common, perhaps trivial. One-Time-Password without the need for an extra token. After the user keys in their username and password, they get sent a random password via SMS. Ten years ago there wasn’t anything that did that. I created a basic RADIUS implementation with support for different SMS gateways, all in Java. Sadly however, with no funding, no clue how to turn it into a business, and just finishing my computer science degree, it had to be abandoned for an easier day job.

I was recently pulled into looking at two-factor-authentication (2FA) solutions. I used SecurID at a previous job, and know of several solutions in this area. I was quite pleased to discover many soft-token solutions working on mobile phones (iphone, blackberry, HTC, Nokia) and USB-based ideas like Yubikey. I was even more pleased to discover open source initiatives in this area, and OATH HOTP in particular.

HMAC-based One-Time-Password (HOTP) is reasonably simple to implement. It’s open, free and secure. Perhaps not militrary-grade security, but pretty good security nevertheless. I also must have found half a dozen free HOTP apps for my iphone (I’m using oathtoken), and it seems like there are apps for almost any phone. When coming to implement it on the server though, I felt a little disappointed. There are a couple of pam modules for linux (HOTP toolkit and Barada), but neither seem very stable or widely used.

What about python implementations then?? I was genuinely surprised there were virtually none. I found one, but it seemed rather specific to the Hebrew University and slightly complex. Or maybe I was too lazy to work out what it actually did…

So I then decided to write my own. It’s a very basic implementation, and by no means ‘production ready’, but it seems to work. I tested it against the test vectors on the rfc, and with my oathtoken iphone app, and it seems to match. I even implemented TOTP, a time-based variant instead of counter based – and that worked too!

#!/usr/bin/env python

import hmac
import sha
import struct
import array
import time

def HOTP(K, C, digits=6):
C_bytes = _long_to_byte_array(C)
#C_bytes = _hex_to_byte_array(str(hex(C)).lstrip("0x").rstrip("L"))
hmac_sha1 = hmac.new(key=K, msg=C_bytes, digestmod=sha).hexdigest()
return Truncate(hmac_sha1)[-digits:]

def TOTP(K, digits=6):
C = long(time.time()/30)
return HOTP(K, C, digits=digits)

def _long_to_byte_array(long_num):
byte_array = array.array(‘B’)
for i in reversed(range(0,8)):
byte_array.insert(0,long_num & 0xff)
long_num >>= 8
return byte_array

def _hex_to_byte_array(hexStr):
"""
Source: http://code.activestate.com/recipes/510399-byte-to-hex-and-hex-to-byte-string-conversion/

Convert a string hex byte values into a byte string. The Hex Byte values may
or may not be space separated.
"""
# The list comprehension implementation is fractionally slower in this case
#
# hexStr = ”.join( hexStr.split(" ") )
# return ”.join( ["%c" % chr( int ( hexStr[i:i+2],16 ) ) \
# for i in range(0, len( hexStr ), 2) ] )

bytes = []

hexStr = ”.join( hexStr.split(" ") )

for i in range(0, len(hexStr), 2):
bytes.append( chr( int (hexStr[i:i+2], 16 ) ) )

return ”.join( bytes )

def Truncate(hmac_sha1):
offset = int(hmac_sha1[-1],16)
binary = int(hmac_sha1[offset*2 : offset*2+8],16) & 0x7fffffff
return str(binary)

def main():
# some test vectors
for i in range(0,10):
print HOTP(’12345678901234567890′,i)
print HOTP(_hex_to_byte_array(’3132333435363738393031323334353637383930′),i)

print TOTP(_hex_to_byte_array(’3132333435363738393031323334353637383930′))

iphone running late

Yoav Aner — Wed, 23 Jun 2010 20:49:16 +0000

I recently noticed my iphone clock wasn’t accurate. I’m not exactly sure why. It was only a few minutes behind, but it still annoyed me. Why couldn’t my iphone sync its time with an internet time server?? I know it is supposed to sync with my mobile network operator, but I think mine doesn’t sync… Maybe it’s my operator?

For jailbroken iphones, there’s a neat app on cydia called NTPDate. It’s a great app and I recommend installing it. All you need is specify the ntp server, and click ‘set’ and it will sync your clock for you. However, I wanted to go a step further. I wanted my iphone to sync itself automatically for me, using a cron job. Well, not quite using cron, but it can be done automatically.

Here’s a quick step by step:

1. Install NTPDate from cydia.
2. SSH to your iphone and create a shell script in /usr/bin (or SCP it to your iphone), lets call it /usr/bin/time_sync.sh

#!/bin/sh

/Applications/NTPdate.app/bin/ntpdate ntp.exnet.com

Notice I used an ntp server in the UK (ntp.exnet.com). You can use any ntp server.

3. Create a new plist file in /Library/LaunchDaemons (or SCP it to your iphone), lets call it /Library/LaunchDaemons/com.test.synctime.plist

version="1.0" encoding="UTF-8"?>

version="1.0">

>
>Label>
>test.synctime>
>OnDemand>
/>
>Program>
>/usr/bin/time_sync.sh>
>StartCalendarInterval>
>
>Minute>
>7>
>
>

This plist is set to run every hour, at 7 minutes past the hour

4. install the plist file

user1s-iPhone: root# launchctl load /Library/LaunchDaemons/com.test.synctime.plist

Job done.

I don’t know what happens if your phone’s time is actually ahead, the job might run several times and keep moving the time… Try at your own risk.

Update: I realised as soon as I plugged my iphone to my mac, it synchornised its time with it. Maybe that’s why it was drifting… I left this automatic sync there anyway though.

iphone asterisk sync

Yoav Aner — Sun, 18 Apr 2010 22:32:44 +0000

On my last post I described how I get my asterisk box to know the caller name from a csv data file. The thing is, my address book keeps changing on my iphone. People change their phone numbers, I meet new people (can you believe it? I don’t let it happen too often though)… I wanted to be able to sync it automatically to my asterisk. This synchronisation also doubles up as a backup for my address book.

This method works on jailbroken iphone version 3. You’d need to setup and enable ssh access to your iphone. There are many resources on how to do it, like this and this.

The process is fairly simple. You can schedule certain tasks or jobs on the iphone. On older version there used to be a cron daemon, but from version 3 – this is all managed by launchd (the Launch Daemon). First lets create a script that syncs the address book file (AddressBook.sqlitedb) to a remote host. We’ll call it address_sync.sh

#!/bin/sh

rsync -e ssh /private/var/mobile/Library/AddressBook/AddressBook.sqlitedb iphone@remote.server:/home/iphone/

I’m using a dedicated user called iphone on the remote server, and the script uses rsync over ssh – it will only use a fraction of the bandwidth to transfer the differences between the files (if any). You can use scp instead if you wish.

Trying to run the script, it would prompt us for a password. We don’t want to use a password because it must be run automatically, so we should set up ssh-key access.

This is a two-part process. First part, on the iphone itself. ssh to your iphone as root and use these commands:

user1s-iPhone:~ root# cd ~/.ssh/
user1s-iPhone:~/.ssh root# ssh-keygen -b 1024 -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/var/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/root/.ssh/id_rsa.
Your public key has been saved in /var/root/.ssh/id_rsa.pub.

Note I used an empty passphrase. Now you’d need to store the contents of id_rsa.pub file (use scp/winscp or simply copy&paste from the terminal).

Go to the remote server we want to allow our iphone key-based access to and add the public key to the authorized_keys file (replace with the contents of your id_rsa.pub file)

root@server:~# cd /home/iphone/.ssh
root@server:/home/iphone/.ssh# echo "[contents of the id_rsa.pub]" >>authorized_keys

Test the connection from your iphone:

user1s-iPhone:~/.ssh root# ssh remote.server
The authenticity of host ‘remote.server (1.2.3.4)’ can’t be established.
RSA key fingerprint is 0c:aa:24:51:2b:61:0f:3e:f2:55:11:ab:a2:e4:61:21.
Are you sure you want to continue connecting (yes/no)? yes

All set. Try running the address_sync.sh and check that it copied the file. It should not prompt for password. Put this script under /usr/local/bin

The next part is to schedule this copy to run automatically. In my case, I have decided to run it daily at 1pm. I’ve followed the excellent post about iphone 3 scheduling here and created another file called com.test.syncjob.plist:

version="1.0" encoding="UTF-8">

version="1.0">
>
>Label>
>test.syncjob>
>OnDemand>
/>
>Program>
>/usr/local/bin/address_sync.sh>
>StartCalendarInterval>
>
>Hour>
>13>
>Minute>
>0>
>
>
>

Copy this file to /Library/LaunchDaemons on your iphone, and run this command on your iphone:

user1s-iPhone:~ root# launchctl load /Library/LaunchDaemons/com.test.syncjob.plist

All done. Now every day at 1pm, my iphone automatically syncs my address book to my server. That’s great for backups. How do I convert the AddressBook.sqlitedb file to csv for my asterisk box?

I created a cron job on the server that does it automatically. It’s comprised of a shell script and some python code:

export-csv.sh:

#!/bin/sh

# converts the sqlite database to csv
sqlite3 -csv /home/iphone/AddressBook.sqlitedb "SELECT First, Last, ABMultiValue.value FROM ABPerson, ABMultiValue WHERE ROWID=record_id;" >/home/iphone/AddressBook.csv

# converts the csv to a more asterisk-friendly format using a python script (see below)
# also removes any email addresses. I only care about phone numbers
/home/iphone/convert-csv.py /home/iphone/AddressBook.csv |grep -v @ >/home/iphone/myPhoneBook.csv

Note that I could easily pipe data instead of saving it into files, but I wanted to have a csv file anyway.

convert-csv.py:

#!/usr/bin/env python

import csv
from sys import argv

if len(argv) <= 1:
print "usage: %s " % argv[0]
exit()
fd = file(argv[1])
for lines in csv.reader(fd):
full_name = "%s %s" % (lines[0].strip(), lines[1].strip())
print "\"%s\",%s" % (full_name.strip(), lines[2].replace(" ",""))

Guilty Pleasures

Yoav Aner — Wed, 25 Mar 2009 14:23:02 +0000

Perhaps yet another misleading title for this post, but bear with me. When I was a child we used to play outside a lot. I always remember the neighbours complaining if we made too much noise. Such is life. There was one period of time that I knew I would get in trouble though. We would get told off big time!! When?? Every day between 2 and 4 in the afternoon. There was even a sign in big red letters telling us all to keep quiet at ‘rest time’ (loosely translated). There was no sign about making noise after 11pm, but there was one for the afternoon nap. It was THAT important.

These days seem long gone now. Does anybody have time for an afternoon nap?? I certainly don’t recall seeing any such signs around.

I just LOVE an afternoon nap. It’s one of modern life’s guiltiest pleasures. You just know you have things to do, and you’re not supposed to just switch off, but you slip into bed, close the door, pull the blinds down, and even 30 minutes get you feeling like a whole new person. Oh the pure joy.

Alas, even with no kids playing around (do kids play outside anyway??), there is always an odd telephone call to wake you up and make you feel even worse (either from guilt or just for being woken up). This can also happen in the morning. Well, for some people it’s late morning or even early afternoon, but I do like to stay in bed late and have a good lie-in.

So what’s the problem?? I can easily unplug my phones. Well, it’s a no brainer with the mobile. The only time it’s switched off is when I’m asleep anyway. But the landline?? I don’t get many people to call me at home these days, but particularly if my mobile is switched off, people (my mum is the worst offender) will make an effort and check that I’m alive. I know I can unplug it, and on my fancy Asterisk box I can even set it to Do Not Disturb by simply calling *78. But I usually forget about it, or forget to plug it in again, and frankly it’s a hassle.

What I want to do is switch the ringing off as soon as I switch off my mobile phone. Have those two aware of each other somehow. How??

Well, this is not a solution for everybody, but it works for me. The idea was taken from Nerd Vittles but modified for my needs and even improved it. No need to install bluetooth software on my Asterisk box. I already have it on my Ubuntu at home. Most bluetooth dongles are pretty much plug&play these days with most Linux desktop distros.

Who is this solution for? or what do you need?

Asterisk – the best open source VOIP system
ssh access to your asterisk (with ssh keys to avoid password prompts)
Voip Phone
bluetooth enabled on your home Linux box (mine is Ubuntu 8.04)
Blueproximity
A bluetooth enabled mobile phone

So what’s going on and how??

If you’re familiar with Asterisk, ssh and some basic scripting, you’ll have no problems following it. I’m not going to spell out each step, but feel free to ask if you need.

The idea is very simple. Blueproximity is meant to be used to lock/unlock your PC when you’re close or far from it, based on signal from your bluetooth phone. It’s extensible enough to launch any script you choose, so I simply configured it so when I’m at home – my bluetooth phone is within a reasonable proximity – it launches a script to enable my home phone to ring. When I’m out of reach (mobile or bluetooth is off) – it automatically redirects calls to my home phone to my voicemail.

Nerd Vittles‘s set-up was to forward calls to your mobile. You can do that too. However, the biggest downsides of this solution is that you need the bluetooth installed on your Asterisk server. Mine is hosted in Germany, so just a little too far to detect my mobile’s bluetooth signal.

My desktop however, is more than capable to cover the entire flat.

So I created a little script on my remote Asterisk box which changes the dialplan to either forward direct to voicemail, or to my home extension:

#!/bin/sh

# DND On – Do Not Disturb

asterisk -rx "dialplan add extension 1234,1,Voicemail(u1234@default) into inbound replace"

#!/bin/sh

# DND Off – Ring my phone

asterisk -rx "dialplan add extension 1234,1,Dial(SIP/1234,20,trj) into inbound replace"

I then created two matching scripts on my home desktop which simply call those scripts remotely:

#!/bin/sh

ssh my.asterisk.box "dnd-on"

#!/bin/sh

ssh my.asterisk.box "dnd-off"

I obviously use ssh keys to authenticate to my asterisk box without a password.

Now all is left is to set up Blueproximity to launch those scripts instead of the default gnome-screensaver-command

I set the lock proximity in Blueproximity to the max and the unlock to slightly lower.

All I have to do to have some quiet time and enjoy my guilty pleasure is simply switch off my mobile.

GoDaddy taking european domains hostage prior to expiration

Yoav Aner — Mon, 23 Feb 2009 02:17:50 +0000

This is something I never thought would concern me. All those domain ownership issues, I thought, were with people who just aren’t organised enough to renew their domains, forget their passwords, or pick domains that others try to steal.

I’m currently managing a couple of .DE and one .CO.UK domains on Godaddy. The .co.uk domain was bought for a fairly long period, so no problems there, but recently, both the .de domains were up for renewal. I received a couple of email reminders from GoDaddy, but when I logged into my account I noticed they are only due to expire on 26th March. More than a month away. On the last email I received on the 20th, I decided it’s probably time to renew, so clicked on the link and was getting my credit card ready.

To my surprise, the GoDaddy domain portal did not allow me to renew them, and marked them as ‘pending expiration’. hmmm… weird. Oh well, I emailed GoDaddy and asked to renew, not even worrying too much. Maybe a system glitch of some sort.

When an email arrived in response, it said that all .co.uk (yeah, this is not a typo, even though I emailed them about 2 .de domains) have to be renewed on the 20th of the month prior to expiration. Very strange. I pointed out to GoDaddy that these were .de domains and even so, why can’t they allow you to renew over their own website??

It was then, that things were beginning to look suspicious. I tried to access both those domains, and was shocked to find out I’m being redirected to a GoDaddy ‘free parking’ page – i.e. a page full of GoDaddy ads and links. Not to where it should have led me to obviously.

WHAT?? The domains are still mine. They have not expired, yet GoDaddy simply hijacked them. That can’t be right. Surely enough. That’s what GoDaddy did.

Ok. I felt angry, and disappointed and very stupid too. Did I miss an email reminder telling me this is going to happen and I ignored it?? I searched through my emails, and nowhere did it say anything about it.

This is what their initial renewal email said

You have elected that the domain name(s) above renew manually. This means that they WILL EXPIRE if you do not log in to your account and renew them by the expiration date(s) listed.

BY THE EXPIRATION DATE(s) LISTED. The dates listed were 26th March. Not the 20th of February.

Ok, so maybe it’s on the original Terms and Conditions I must have signed. I searched, and looked, and searched again. Nothing.

Right then, so it’s probably something the .DE domain TLD forces GoDaddy to do. I trawled through DENIC. Trying to find a clue. Nope. Nein. Nicht.

I’ve had numerous emails and a long telephone conversation with GoDaddy support, but they simply wouldn’t listen. I eventually got to a supervisor who gave me two options:

1. Do nothing, and these domains will expire, but only on the 26th March. Until then they are not going to be released or re-activated.
2. Pay their renewal fees, agree to their terms (which also mean 60 days lock from transferring)

I obviously had no alternative but to pay. The fees were reasonable. At least no nasty ‘re-activation fee’. I didn’t mind paying the renewal fees anyway and in the first place. But why hijack my domains??

As of now, nearly 24 hours after paying, and after another couple of emails to their support, my domains are still locked and redirected to a landing page.

I then came across this post by Simon Kapenda. Looks like he’s in a similar position to mine. There was another similar post, mostly with .co.uk domains here, but not much more I could find.

If you’ve been affected please let me know. Surely there’s something that can be done to complain against such practice.

Postcode, Barcode and python code

Yoav Aner — Fri, 30 Jan 2009 15:22:14 +0000

I’ve had a strange thing happening a while ago. I sent a CD in a padded envelope to someone, and it was returned to me. Well, it didn’t look like it was returned, more like they actually delivered it to me instead of to the person I sent it to.

Then I noticed something. I was re-using an old envelope. For environmental reasons of course (read: being so tight-assed, saving money on padded envelopes). I did write the destination address on the right side, so my address was nowhere on the envelope. What was left there however was this tiny printed barcode used by the previous sender with my address.

So it occurred to me, the postoffice didn’t even bother reading the address I put there, it scanned the little barcode instead.

Neat. So maybe I could print those barcodes and make sure my letters get to the right destination.

A little googling brought me to this nice web page you can put your postcode and it generates a string for you, which if you use in a word processor with a special barcode font (also available for download there), you can print out yourself.

That’s nice. What I couldn’t work out however, is how the postcode checksum is calculated. The website was nice enough to do it for you, but what happens if it’s unavailable or goes down??

My good friend Dan, after noticing I used those fancy barcodes on a letter I sent him, managed to find more information about the checksum algorithm, and was even kind enough to write some perl code that runs it.

When Dan shares his code I’ll link to it, but meanwhile, and just to practice with some python coding, I wrote my own python code to do the same thing (but obviously in a much nicer way than perl):

#!/usr/bin/env python

import sys
import re

if len(sys.argv)<2:
print """
Usage: Postcode.py []

must appear with no spaces, e.g. SN34RD, case insensitive
is optional and set by default to 9Z (if you don’t know it, don’t use it)
"""
sys.exit()

Suffix = ’9Z’
if len(sys.argv)>2:
Suffix = sys.argv[2].upper()

if len(Suffix)!=2:
print """
Suffix or postcode incorrect.
Please use postcodes without spaces or enclosed in ""
Suffix must be 2 characters (or blank to use 9Z)
"""
sys.exit()

# Check for correctness of postcode. Regular expression found at http://regexlib.com/REDetails.aspx?regexp_id=260
# (notice – not at top of the page, but on one of the bottom comments)
if not re.match("^(([A-PR-UWYZ]{1,2}[0-9]{1,2}[ABEHJMNPRVWXY]?)\s?([0-9][ABD-HJLNP-UW-Z]{2})|(GIR)\s?(0AA))$", sys.argv[1].upper()):
print "Postcode incorrect. Please use correct UK postcodes and either without spaces or enclosed in \"\""
sys.exit()

Postcode = sys.argv[1].upper().replace(‘ ‘,”) + Suffix

# Matrix as it appears on the PDF guide. Note arrays are referenced from 0..5
PostCodeMatrix = [
[’0′,’1′,’2′,’3′,’4′,’5′],
[’6′,’7′,’8′,’9′,‘A’,‘B’],
[‘C’,‘D’,‘E’,‘F’,‘G’,‘H’],
[‘I’,‘J’,‘K’,‘L’,‘M’,‘N’],
[‘O’,‘P’,‘Q’,‘R’,‘S’,‘T’],
[‘U’,‘V’,‘W’,‘X’,‘Y’,‘Z’]
]

# A ‘reverse-lookup’ dictionary for each postcode character
PostCodeDict = {}
for row in range (0,6):
for col in range (0,6):
PostCodeDict[(PostCodeMatrix[row][col])] = ((row+1)%6,(col+1)%6)

(rowSum, colSum) = (0,0)
for i in Postcode:
rowSum += PostCodeDict[i][0]
colSum += PostCodeDict[i][1]

Postcode += PostCodeMatrix[(rowSum%6)-1][(colSum%6)-1]
print "(%s)" % Postcode

sniffing some fresh tomatoes

Yoav Aner — Wed, 03 Sep 2008 21:31:23 +0000

Perhaps the title is a little misleading, but it’s an opportunity to combine two of my greatest loves: food and computers. I suppose even this intro is misleading. Oh. Forget it. Lets get down to business. And this time our business is rather short (and sweet).

Running tcpdump on my Linksys router (well, Buffalo WHR-54GS to be precise, but the same famous WRT54G clone that runs open source firmware).

I’ve been running Tomato firmware for quite a while now (from version 0.7 or something??). What a wonderful firmware. Definitely the very best. Neat. Stylish. Full of all the features one can think of. And then some. A true work of art.

Alas, I wanted to sniff some traffic to analyse using Wireshark (some SIP stuff, I might cover it some other time), and couldn’t be bothered getting my old heavy dusty hub out just to capture some packets. I was therefore turning to my fresh Tomato, but no such tool is available.

Well, not quite.

A few google searches concluded that I could probably get a pre-compiled binary, place it on a CIFS network share, and run it. So I did.

I download the latest ipkg of tcpdump off ipkg.nslu2-linux.org. Renamed the file to .tar.gz, untar’d it and extracted the binary inside data.tar.gz (/opt/bin/tcpdump).

I used a CIFS windows share feature on tomato. Straight under Administration->CIFS Client and pointed it to the share with the tcpdump binary. Telnet onto the router and off you go, saving the packet dump onto the same windows share. Then analyse it with Wireshark. Job Done. Smells delicious, doesn’t it?

# telnet 192.168.0.1
Trying 192.168.0.1…
Connected to 192.168.0.1.
Escape character is ‘^]’.
unknown login: root
Password:

Tomato v1.21.1515

BusyBox v1.2.2 (2008.07.26-14:43+0000) Built-in shell (ash)
Enter ‘help’ for a list of built-in commands.

# cd /cifs1
# ./tcpdump ip host 192.168.0.110 -w ./dumpfile.txt -s0
tcpdump: listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes

JDK 5 Debian etch Virtuozzo installation oddities

Yoav Aner — Thu, 03 Apr 2008 18:42:48 +0000

Not a particularly interesting post, unless you happen to be running Debian 4.0 etch running on Virtuozzo host and are trying to install JDK5. I saw a post with a user reporting the same error on the parallels forum , but couldn’t find a clear solution, until I bumped onto a weird forum in German, which luckily linked to a related bug report (in English).

So if you get this error message when installing jdk5:

Setting up sun-java5-bin (1.5.0-14-1etch1) …
Could not create the Java virtual machine.
dpkg: error processing sun-java5-bin (–configure):
subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
sun-java5-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)

The solution is to edit the postinst script

# vi /var/lib/dpkg/info/sun-java5-bin.postinst

comment out one line below:

# activate class data sharing
case i386 in i386|sparc)
rm -f $basedir/jre/lib/i386/client/classes.jsa
### $basedir/bin/java -client -Xshare:dump > /dev/null
esac

Then reinstall. Ugly, but it seems to do the trick

Turbogears, Lighttpd and WSGI for real dummies

Yoav Aner — Fri, 14 Sep 2007 19:49:39 +0000

Just so to set the records straight, I’m the real dummy here, as you would obviously see.

I absolutely have no experience with python, lighttpd, turbogears or wsgi, fcgi but I was given a task to set it up on a server. All I used was some online posts and common sense. Normally I wouldn’t have to post a blog entry about it, most of the stuff is already out there. On this occasion however, I felt there’s some missing documentation, or else – simply because I don’t know anything about any of these it was more tricky than it usually is. Perhaps it’s one indicator how this all seems from the sysadmin perspective, as opposed to a turbogears developer.

I’m not going to cover every step, because I feel the basics should be easy enough to follow if you’ve done some linux server setup. Documentation is pretty clear on the basic install stuff. Nevertheless, some things aren’t too clear, and I hope to cover at least to some extent. For example, I wasn’t quite sure whether I should rely on the debian packages or the python easy_setup ones, that can be retrieved automagically. I guess easy_setup is a good bet for the latest versions, as I’m using Debian 4 stable (etch), and try to avoid mixing stable and unstable, test – mostly because I’m not confident enough about not screwing things up

Ok, down to business then
I’m running:

First thing I wasn’t sure about, is where to place the turbogears apps. I then decided to create /var/turbogears and put them under there. I don’t think it makes a huge difference, but seemed sensible enough.

I then created a turbogears app using

/var/turbogears# tg-admin quickstart
Enter project name: newproject
Enter package name [newproject]:
Do you need Identity (usernames/passwords) in this project? [no]
…

so now there’s a new project called newproject under /var/turbogears/newproject.

I then tried to follow this link (or this one – which is essentially a mirror), to have it all served by lighty and wsgi, but then started having problems… Well – it simply wouldn’t work

I wasn’t very surprised considering all I did was a copy&paste job, without really understanding what’s going on. Getting to the root cause was somehow more of a challenge.

I wish I noted all the tiny different things I tried, but I’ll try to describe a couple of ones that helped along the way:

enabling more detailed debugging in lighttpd.conf:

debug.log-request-handling = “enable”
scgi.debug = 1

Another thing that took me ages to figure out from reading on the links above, what is actually /full/path/to/your/application supposed to point at?? I tried /var/turbogears/newproject/newproject-start-scgi.py (which is the new script I created by following the instructions) and /var/turbogears/newproject/ etc.

the correct answer is /var/turbogears/newproject/newproject (i.e. the folder where the application is UNDER the project, they are named the same when using the tg-admin quickstart)

so my lighttpd configuration looks like (after resolving it, I obviously took the debug statements out):

server.document-root = "/var/turbogears/newproject/newproject"
#debug.log-request-handling = "enable"
#scgi.debug = 1

$HTTP["url"] !~ "^/static/" {
scgi.server = (
"/" =>
( "127.0.0.1" =>
(
"host" => "127.0.0.1",
"port" => 4000,
"check-local" => "disable"
)
)
)
}

That didn’t solve the problem unfortunately. I was getting lots of these errors (in /var/log/lighttpd/error.log), even when running the new modified turbogears script (newproject-start-scgi.py):

2007-09-10 16:38:57: (response.c.258) URI-path : /
2007-09-10 16:38:57: (mod_scgi.c.2760) handling it in mod_scgi
2007-09-10 16:38:57: (response.c.372) — before doc_root
2007-09-10 16:38:57: (response.c.373) Doc-Root : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.374) Rel-Path : /
2007-09-10 16:38:57: (response.c.375) Path :
2007-09-10 16:38:57: (response.c.423) — after doc_root
2007-09-10 16:38:57: (response.c.424) Doc-Root : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.425) Rel-Path : /
2007-09-10 16:38:57: (response.c.426) Path : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.443) — logical -> physical
2007-09-10 16:38:57: (response.c.444) Doc-Root : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.445) Rel-Path : /
2007-09-10 16:38:57: (response.c.446) Path : /var/turbogears/newproject/
2007-09-10 16:38:57: (mod_scgi.c.1325) connect delayed, will continue later: 7
2007-09-10 16:38:57: (mod_scgi.c.2239) got proc: 7 0 4000 1
2007-09-10 16:38:57: (mod_scgi.c.1179) release proc: 7 0
2007-09-10 16:38:57: (mod_scgi.c.2029) proc: 127.0.0.1 4000 1 0 0 0
2007-09-10 16:38:58: (mod_scgi.c.2029) proc: 127.0.0.1 4000 1 0 0 0
2007-09-10 16:38:59: (mod_scgi.c.2029) proc: 127.0.0.1 4000 1 0 0 0

And at some other point (apologies, don’t remember exactly when was what)

2007-09-10 16:49:48: (response.c.206) URI-path : /
2007-09-10 16:49:48: (response.c.207) URI-query :
2007-09-10 16:49:48: (response.c.257) — sanatising URI
2007-09-10 16:49:48: (response.c.258) URI-path : /
2007-09-10 16:49:48: (mod_scgi.c.2825) no fcgi-handler found for: /
2007-09-10 16:49:48: (mod_scgi.c.2029) proc: 127.0.0.1 4000 5 0 0 0
2007-09-10 16:49:49: (mod_scgi.c.2029) proc: 127.0.0.1 4000 5 0 0 0

After very long googling experience, I came across this link! and then modified my turbogears startup script as it recommended, but with the following modifications where it said modulename=”gtextranet.config.app” I changed it to newproject.config (without the .app), and obviously the line that now says from newproject.controllers import Root

#!/usr/bin/env python
import pkg_resources
pkg_resources.require("TurboGears")

import turbogears
import cherrypy
cherrypy.lowercase_api = True

from os.path import *
import sys

# first look on the command line for a desired config file,
# if it’s not on the command line, then
# look for setup.py in this directory. If it’s not there, this script is
# probably installed
if len(sys.argv) > 1:
turbogears.update_config(configfile=sys.argv[1],
modulename="newproject.config")
elif exists(join(dirname(__file__), "setup.py")):
turbogears.update_config(configfile="dev.cfg",
modulename="newproject.config")
else:
turbogears.update_config(configfile="prod.cfg",
modulename="newproject.config")

from newproject.controllers import Root
from cherrypy._cpwsgi import wsgiApp
#from paste.util.scgiserver import serve_application
from flup.server.scgi import WSGIServer
#from flup.server.scgi_fork import WSGIServer

port = 4000
if len(sys.argv) > 2:
port = int(sys.argv[2])

cherrypy.config.update({
‘global’: {
‘autoreload.on’: False,
}})
cherrypy.root = Root()
cherrypy.server.start(initOnly=True, serverClass=None)
#serve_application(application=wsgiApp, prefix=’/', port=port)
WSGIServer(application=wsgiApp, bindAddress=(‘localhost’, port)).run()

Don’t ask me what made the difference with this script, as I said before I know nothing about python or turbogears, but it seemed to work.
I had to launch the new script

/var/turbogears/newproject #./newproject-start-scgi.py

last thing I did was to set up a bash script to launch this newproject-start-scgi.py script when the system boots. I used a rather basic bash script placed under /etc/init.d:

#!/bin/sh
SCRIPT="/var/turbogears/newproject/newproject-start-scgi.py"
case "$1" in
start) echo "STARTING $SCRIPT"
$SCRIPT &
;;
stop) echo "STOP $SCRIPT"
pid=`/bin/ps -ef | /usr/bin/awk ‘($9~"newproject" && $3==1){printf " "$2}’`
##
## please check if this computation of pid is correct on your system
##
if [ -n "$pid" ]; then
/bin/kill $pid
fi
;;
*) echo "Usage: $0 {start|stop}"
exit 2
;;
esac
exit 0

Even this script is probably crap and shows my lack of refined (or even basic) bash scripting experience, but it seemed to do the trick too