Building an AVR toolchain on OS X
I’ve recently got myself a couple of Arduino Duemilanove boards from Little Bird
Electronics. Now the Arduino is a pretty cool bit of kit, and it
comes with a great integrated development environment, which makes
writing simple little programs an snap. Of course, I’m too much of an
old curmudgeon to want to use and IDE and some
fancy-schmancy new language, I want make, I want
emacs, I want gcc. So I embarked on the
task of building a cross-compiler for the AVR board, that
will work on OS X.
Now, my ideal solution for this would have been to create a package
for Homebrew, my
current favourite package manager for OS X. Unfortunately, that just
isn’t going to happen right now. GCC toolchains pretty much
insist in putting things in $prefix/$target. So, in a
standard homebrew install, that would mean I need a directory
/usr/local/avr, unfortunately, Homebrew insists
your package only dump things in etc, bin,
sbin, include, share or
lib. Now, after wasting a bunch of time first fighting
GCC’s autogoat build system to try and convince it to conform
to Homebrew’s idea of a filesystem hierarchy, and then a whole
other bunch of time trying to learn Ruby and
convince Homebrew that other directories should be allowed, I took
the corwads way out and decided /opt/toolchains is
a perfectly respectable place for my cross-compilers to live.
And so, the cross compiler dance begins. We start with binutils, and as the code dictates, we start by trying with the latest version, which at time of writing is 2.20.1.
Of course, building a toolchain for a non-x86 based architecture
wouldn’t be the same if you didn’t need to patch the
source. The patches for the avr are mostly minimal; adding
some extra device definitions, for devices you probably don’t have
anyway. If you miss this step, expect some pain when you try to compile
avr-libc. I got my patches from the FreeBSD AVR binutils patchset. I try to avoid
patches I don’t need so have only applied the patch-newdevices patch. If things break for you, you might want to try the other patches as well.
$ mkdir avr-toolchain-build$ cd avr-toolchain-build$ wget http://ftp.gnu.org/gnu/binutils/binutils-2.20.1.tar.bz2$ wget "http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/ports/devel/avr-binutils/files/patch-newdevices?rev=1.16;content-type=text%2Fplain" -O patch-newdevices$ tar jxf binutils-2.20.1.tar.bz2$ cd binutils-2.20.1$ patch -p0 < ../patch-newdevices$ cd ..$ mkdir binutils-build$ cd binutils-build$ ../binutils-2.20.1/configure --target=avr --prefix=/opt/toolchains/ --disable-werror$ make -j2$ make install$ cd ../
Fairly simple, the only gotcha is the
--disable-werror, seems that binutils added the
-Werror flag to the build, good move!,
unfortunately, seems that the code isn’t warning free, so things barf
(bad move!), but it comes with a free flag to disable the
error... that’s good.
So, now we move on to GCC. Now, since I’m really only interested in compiling C code, (remember the bit about being a curmudgeon, none of this fancy C++ stuff for more, no siree!), we can just grab GCC’s core package.
Now GCC has some dependencies these days, so you need to get gmp, mpc and mpfr installed somehow. I suggest using Homwbrew. In fact it was this step with my old Mac Ports setup that forced me to switch to Homebrew; too many weird library conflicts with iconv between the OS X version and the ports version; but hey, you mileage may vary! (Note: make sure you have the latest version of Homebrew that includes my fix for building mpfr).
And of course, just like binutils, you need some patches as well.
Unfortunately things here aren’t as easy. There doesn’t appear to be
any semi-official patch for new devices for gcc 4.5.0! So, based on this
patch, I managed to hack something
together. The formats have changed a little, so I’m not 100%
confident that it works, if you are actually trying to use one of the
new devices in this patch, be a little skeptical, and
double-check my patch. If you are using an existing supported
AVR like the atmega328p, then the patch should work
fine (well, it works for me).
$ brew install gmp libmpc mpfr$ wget http://ftp.gnu.org/gnu/gcc/gcc-4.5.0/gcc-core-4.5.0.tar.bz2$ wget http://benno.id.au/drop/patch-gcc-4.5.0-avr-new-devices$ tar jxf gcc-core-4.5.0.tar.bz2$ cd gcc-core-4.5.0$ patch -p1 < ../patch-gcc-4.5.0-avr-new-devices$ mkdir gcc-build$ cd gcc-build$ ../gcc-4.5.0/configure --target=avr --prefix=/opt/toolchains$ make -j2$ make install$ cd ..
So, the final piece to complete the toolchain is getting a C library. You almost certainly want AVR Libc, or you can be really hard-core and go without a C library at all, your call.
$ wget http://mirror.veriportal.com/savannah/avr-libc/avr-libc-1.7.0.tar.bz2$ tar jxf avr-libc-1.7.0.tar.bz2$ mkdir avr-libc-build$ cd avr-libc-build$ ../avr-libc-1.7.0/configure --prefix=/opt/toolchains --host=avr --build=$ make$ make install
This should work with out any problems, however, if you messed up the new devices patches when building GCC and binutils, you might get errors at this point.
So, now we have a working GCC toolchain for the AVR we can start programming. Of course, you really want to have a good reason to do things this way, otherwise I really recommend just using the Arduino development environment..
Next time I’ll be looking at getting some example C programs running on the Arduino.
References
Mass market smartphones
The gap between so called feature phones and smart phones continues shrink. Features phones with their, typically, closed and proprietary operating systems have generally had an edge in terms of price by requiring fewer hardware resources and forgoing a strong separation between the baseband processing software and the primary user-interface software. However, with recent software innovations there is no reason you can’t effectively run an open smartphone OS such as Android on the hardware that is typically used in feature phones today. In the near future rather than having feature phone and smart phones we will have mass-market smartphones and high-end smartphones all running open operating systems such as Symbian S3 and Android.
The first enabler is fast-context switching approach which allows better performance of operating systems that use protected processes, such as Symbian and Android, on ARM9 based CPUs (which are commonly used in feature phones).
Secondly, continual improvement in smart-phone OS performance enables those OSes to run on lower-end hardware. An example of this is the recently announced Dalvik JIT in the latest Froyo release of Android.
Finally, mobile virtualization provides the separation guarantees required to allow a baseband stack, and an open OS to run on the same piece of hardware.
As much as I love getting and playing with fancy new hardware, I’m looking forward to seeing a wider adoption of open OSes like Android across a wider range of the mobile phone market.
Android Emulator Internals — Bus Scanning
Wow, I can’t believe it has been over two years since I last wrote about Android’s for of the QEMU emulator. Turns out there have been some changes since I last looked at it.
The most important is that the Android emulator no longer has a
fixed layout of devices in the physical memory address space. So,
while it may have previously been the case that the event device was
always at 0xff007000, now it might be at
0xff008000, or 0xff009000, depending on what
other devices have been configured for a particular device
configuration.
Now, if a device may exist at some random physical address, how
does the OS know how to setup the devices drivers? Well, as I’m
sure you’ve guessed, the addresses and really random, they
are located at page-offset addresses through a restricted range of
memory. OK, so how does the OS know what the range is? Well, there is
the goldfish_device_bus device.
Basically, this device provides a mechanism to enumerate the
devices on the bus. The driver writes PDEV_BUS_OP_INIT to
the PDEV_BUS_OP register, the
goldfish_device_bus then raises an interrupt. The driver
the reads the PDEV_BUS_OP register. Each time the value
is PDEV_BUS_OP_ADD_DEV, the driver can read the other
registers such as PDEV_BUS_IO_BASE,
PDEV_BUS_IO_SIZE, PDEV_BUS_IRQ, to determine
the properties of the new device. It continues doing this until it
reads a PDEV_BUS_OP_DONE, which indicates the bus scan
has finished.
The driver can determine what type of device it has found by
writing a pointer to the PDEV_BUS_GET_NAME register. When
this happens the device writes an the device’s name (as an ASCII
string) to the pointer.
Linux uses these strings to perform device to driver matching as described in the Platform Devices and Drivers document.
Weird things about JavaScript
JavaScript keeps throwing up interesting new tidbits for me. One that kind of freaked me out the other day is when functions and variables are created in a given block of code.
My expectation was that in a block of code, a sequence of statements, the statements would be executed in order, one after another, just like in most imperative languages. So for example in Python I can do something like:
def foo(): pass print foo
and expect output such as:
<function foo at 0x100407b18>
However, I would not expect the same output if I wrote the code something like this:
print foo def foo(): pass
and in fact, I don’t. I get something close to:
NameError: name 'foo' is not defined
Now in JavaScript, by contrast, I can do something like:
function foo() { }
console.log(foo);
and the console will dutifully have foo () printed in the log.
However, and here comes the fun part, I can also do this:
console.log(foo);
function foo() { }
which blows my mind. I guess this really comes in handy when... nope, can’t think of any good reason why this is a useful feature. (I’m sure there must be a good reason, but damned if I can work it out. But this is only where the fun begins. Because the same thing works for variables!
Now usually in Javascript, if you have a function, and write code like:
function foo() {
x = 37;
}
foo();
console.log("x:", x);
You find out you’ve stuffed up, and accidently written to the global object because for some brain-dead reason when you assign to a variable that doesn’t exist in JavaScript it will merrily walk up the scope chain to the global object and plonk it in there. If you do something like:
function foo() {
var x;
x = 37;
}
foo();
console.log("x:", x);
x will be undefined, and the x =
37 line will update the function’s locally scoped variable, and
not mess with the global object. But now comes the part that screws with
your head. You can just as easily write this as:
function foo() {
x = 37;
var x;
}
foo();
console.log("x:", x);
and it will have exactly the same effect. Now it is fairly clear what is happening here; as a function is parsed any variables and functions are created at that time. It turns out though that although variables are created, they are not initialised, so code such as:
var x = 12;
function foo() {
console.log("x:", x);
var x = 37;
}
foo();
will output x as undefined, (not 37, or 12). Now this
behaviour isn’t wrong, or necessarily bad, but it was certainly counter
to my expectation and experience in other languages.
Monkey Patching Javascript
Javascript is a very permissive language; you can go messing around of the innards of other classes to your heart’s content. Of course the question is should you?.
Currently I’m playing around with the channel
messaging feature in HTML5. In a nutshell, this API lets you
create communication MessageChannel, which have two
MessagePort object associated with it. When you send a
message on port1 an event is triggered on port2 (and vice-versa). You send a message
by calling the postMessage method on a port object. E.g:
port.postMessage("Hello world");
This opens up a range of interesting possibilities for web developers, but this blog post is about software design, not cool HTML5 features.
Unfortunately the postMessage is very new, and the implementation
has not yet caught up with the specification. Although you are meant to be
able to transfer objects using postMessage currently only strings
are supported, and any other objects are coerced into strings. This has an
interesting side-effect. If we have code such as:
port.postMessage({"op" : "test"});
the receiver of the message ends up with the string
[Object object], which is mostly
useless; actually it is completely useless. So, we want to transfer
structured data over a pipe that just supports standard strings, sounds
like a job for JSON.
So, now my code ends up looking like:
port.postMessage(JSON.stringify({"op" : "test"});
Now this is all good, but it gets a bit tedious having to type that out
every time I want to post a message, so a naïve approach we can simple create
a new function, postObject:
function postObject(port, object) {
return port.postMessage(JSON.stringify(object));
}
postObject(port, {"op" : "test"});
OK, so this works, and it is pretty simple to use, but there
aesthetic here that makes this grate just a bit, why do I have to do
postObject(port, object), why can’t I do
port.postObject(object), that is more “object-oriented”,
so, thankfully JavaScript lets us monkey patch objects
at run time. So, if we do this:
port.postObject = function (object) {
return this.postMessage(JSON.stringify(object));
}
port.postObject({"op" : "test"});
OK, so far so good. What problems does this have? Well, firstly we are creating a new function for each port object, which isn’t great for either execution time, or memory usage if we have a large number of ports. So instead we could do:
function postObject(object) {
return this.postMessage(JSON.stringify(object));
}
port.postObject = postObject;
port.postObject({"op" : "test"});
This works well, except we have two problems. The
postObject function ends up as a global function, and if
you called it as a global function, the this parameter
would be the global, window object, rather than a port object, which
would be an easy mistake to make, and difficult to
debug. Additionally, we end up with additional per-object data for
storing the pointer. Thankfully javascript has a powerful mechanism
for solving both the problems: the prototype
object (not to be confused with the javascript framework of the
same name).
So, if we update the prototype object, instead of object directly, we don’t need to add the function to the global namespace, we avoid per-object memory usage, and we avoid extra code having to remember to set it for every port object:
MessagePort.prototype.postObject = function postObject(object) {
return this.postMessage(JSON.stringify(object));
}
port.postObject({"op" : "test"});
Now, this ends up working pretty well. The real question is
should be do this? Or rather which of these options should we
choose? There is an argument to be made that monkey-patching anything
is just plain wrong, and it should be avoided. For example rather than
extending the base MessagePort class, we could create a
sub-class (Exactly how you create sub-classes in JavaScript is another
matter!).
Unfortunately sub-classing doesn’t get us to far, as we are not the
ones directly creating the MessagePort instance, the
MessageChannel construct does this for us. (I guess we
could monkey-patch MessageChannel, but that defeats the
purpose of avoiding monkey-patching!).
Of course, another option would be to create a class that encapsulates a port, taking one as a constructor. E.g:
function MyPort(port) {
this.port = port;
}
MyPort.prototype.postObject = function postObject(object) {
this.port.postMessage(JSON.stringify(object));
}
port = MyPort(port);
port.postObject({"op" : "test"});
Of course, this means we have to remember to wrap all our port
objects in this MyPort class. This is kind of messy (in
my opinion). Also we can no longer call the standard port methods. Of
course, we could create wrappers for all these methods too, but then
things are getting quite verbose, and we are stuffed if it comes to
inspecting properties.
Unlike some other object-oriented languages, Javascript provides another alternative, we could change the class (i.e: prototype) of the object at runtime. E.g:
function MyPort(port) { }
MyPort.deriveFrom(MessagePort); /* Assume this is how we create sub-classes */
MyPort.prototype.postObject = function (object) {
this.portMessage(JSON.stringify(object));
}
port.__proto__ = MyPort.prototype; /* Change class at runtime */
port.postObject({"op" : "test"});
This solves most of the problems of the encapsulted approach, but
we still have to remember to adapt the object, and aditionally, __proto__
is a non-standard extension.
OK, so after quickly looking at the sub-classing approaches I think it is fair to discount them. We are still left with trying to determine if any of the monkey-patching approaches is better than a simple function call.
So, there is mostly a consesus out there that monkey patching the base object is verboten, but what about other objects?
Well, if you are in you own code, I think it is a case of anything
goes, but what if we are providing reusable code modules for other
people? (Of course, even in your own code, there might be libraries
that you include that are affected by your overloading). When base
objects start working in weird and wonderful ways just because you
import a module debugging becomes quite painful. So I think
changing the underlying implementation (like the example does
when monkey-patching the postMessage method) should
probably be avoided.
OK, so now the choices are down to function vs. add a method to the
built-in class’s prototype. So if we just add a new global function we
could be conflicting with any libraries that also name a global
function in the same way. If we add a method to the prototype, at
least we are limiting of messing with the namespace to just the
MessagePort object; but really, both the options aren’t
ideal.
The accepted way to get around this problem is to create a module specific namespace. This reduces the number of potential conflicts. E.g:
var Benno = {};
Benno.postObject = function postObject(port, object) {
return port.postMessage(JSON.stringify(object));
}
Benno.postObject(port, {"op" : "test"});
Now, this avoids polluting the global namespace (except for the
single Benno object). So, it would have to come out above
the prototype extension approach. Now, we should consider if it is possible
to play any namespace tricks with the prototype approach. It might be nice
to think we could do something like:
MessagePort.prototype.Benno = {}
MessagePort.prototype.Benno.postObject = function postObject(object) {
return this.postMessage(JSON.stringify(object));
}
port.Benno.postObject(object);
but this doesn’t work because of the way in which methods and the
this object work. this in the function ends
up referring to the Benno module object, rather than the
MessagePort object.
Even assuming this did work, the function approach has some additional benefits. If the user wants to reduce the typing they can do something like:
var $B = Benno; $B.postObject(port, object);
or even,
var $P = Benno.postObject; $P(port, object);
The other advantage of this scheme is that for someone debugging
the code it should be much more obvious where to look for the code and
to understand what is happening. If you were reading code and saw
Benno.postObject(port, object), it would be much more
obvious where the code came from, and where to start looking to debug
things.
So, in conclusion, the best approach is also the simplest: just
write a function. (But put it in a decent namespace first). Sure
instance.operation(args) looks nicer than
operation(instance, args), but in the end the ability
to namespace the function, along with the advantage of making a clear
distinction between built-in and added functionality means that
the latter solution wins to day in my eyes.
If you have some other ideas on this I’d love to hear them, so please drop me an e-mail. Thanks to Nicholas for his insights here.
HTML5 FileApi and Jpeg Meta-data
I’m really impressed with the way the HTML5 spec is going, and the fact that it is quickly going to become the default choice for portable application development.
One of the lastest additions to help support application
development is the File
API. This API enables a developer to gain access to the contents
of files locally. The main new data structure that a developer if
provided with is a FileList objects which represents an
array of File objects. FileList objects
can be obtained from two places; input form elements
and from drag & drop DataTransfer objects.
Based on this latest API, I’ve created a simple library, JsJpegMeta for parsing Jpeg meta data.
I’ve hacked together a example that
demonstrates the library. Just select a JPEG file from the form, or
drag a JPEG file onto the window. For large JPEG files you might need
to be a little bit patient, as it can be a little slow. This slowness, suprisingly,
doesn’t appear to be the Javascript part, but rather Firefox’s handling of large
data: URLs and JPEG display in general.
The rest of this post goes into some of the details. Unfortunately only Firefox 3.6 supports these new APIs right now.
Using the File API
Here is an example of how to get access to a FileList.
When the user chooses a file, it calls the Javascript function
loadFiles. (Assuming you have already defined that
function).
<form id="form" action="javascript:void(0)">
<p>Choose file: <input type="file" onchange="loadFiles(this.files)" /></p>
</form>
A File object just provides a reference to a file; to
actually get some data out of the file you need to use a
FileReader object. The FileReader object
provides an asynchronous API for reading the file data into
memory. Three different methods are provided by the
FileReader object; readAsBinaryString,
readAsText and readAsDataURL. A callback,
onloadend, is executed when the file has been read into
memory, the data is then available via the result field.
Here example of what the loadFiles function might look like:
function loadFiles(files) {
var binary_reader = new FileReader();
binary_reader.file = files[0];
binary_reader.onloadend = function() {
alert("Loaded file: " + this.file.name + " length: " + this.result.length);
}
binary_reader.readAsBinaryString(files[0]);
$("form").reset();
}
Note the $("form").reset(); clears the input form.
Drag & Drop
Forms are not the only way to get a FileList, you can
also get files from drag and drop
event. You need to handle three events; dragenter, dragover and drop.
<body ondragenter="dragEnterHandler(event)" ondragover="dragOverHandler(event)" ondrop="dropHandler(event)">
The default handling of these are fairly striaght forward:
function dragEnterHandler(e) { e.preventDefault(); }
function dragOverHandler(e) { e.preventDefault(); }
function dropHandler(e) {
e.preventDefault();
loadFiles(e.dataTransfer.files);
}
Parsing files
The interesting thing here is the readAsBinaryString,
when this method is used result ends up being a
binary string. This is pretty new because, as far
as I know, there hasn’t really been a good way to access binary data
in Javascript before. Each character in the binary string represents a
byte, and has a character code in the range [0..255].
This is great, because it means that we can parse binary strings locally, without having to upload files to a server for processing. Unfortunately there isn’t a great deal of support for handling binary data in Javacript; there isn’t anything like Python’s struct module.
Luckily it isn’t too hard to write something close to this. Mostly we wanted to parse unsigned and signed integers of arbitrary length. To be useful, we need to handle both little and big endianess. A very simple implementation of parsing an unsigned integer is:
function parseNum(endian, data, offset, size) {
var i;
var ret;
var big_endian = (endian === ">");
if (offset === undefined) offset = 0;
if (size === undefined) size = data.length - offset;
for (big_endian ? i = offset : i = offset + size - 1;
big_endian ? i < offset + size : i >= offset;
big_endian ? i++ : i--) {
ret <<= 8;
ret += data.charCodeAt(i);
}
return ret;
}
endian specifies the endianess; the string literal ">"
for big-endian and "<" for little-endian. (Copying the Python struct
module). data is the binary data to parse. An
offset can be specified to enable parsing from the middle
of a binary structure; this defaults to zero. The size of
the integer to parse can also be specified; it defaults to the
remainder of the string.
Signed integers require a little bit more work. Although there are
multiple ways of representing
signed numbers, by far the most common is the two’s
complement method. A function that has the same inputs as parseNum
is:
function parseSnum(endian, data, offset, size) {
var i;
var ret;
var neg;
var big_endian = (endian === ">");
if (offset === undefined) offset = 0;
if (size === undefined) size = data.length - offset;
for (big_endian ? i = offset : i = offset + size - 1;
big_endian ? i < offset + size : i >= offset;
big_endian ? i++ : i--) {
if (neg === undefined) {
/* Negative if top bit is set */
neg = (data.charCodeAt(i) & 0x80) === 0x80;
}
ret <<= 8;
/* If it is negative we invert the bits */
ret += neg ? ~data.charCodeAt(i) & 0xff: data.charCodeAt(i);
}
if (neg) {
/* If it is negative we do two's complement */
ret += 1;
ret *= -1;
}
return ret;
}
JpegMeta API
JpegMeta is a
simple, pure Javascript library for parsing Jpeg meta-data. To use it
include the jpegmeta.js file. This creates a single,
global, module object JpegMeta. The JpegMeta
module object has one public interface of use, the JpegFile
class. You can use this to construct new JpegFile class instances. The input
is a binary string (for example as returned from a FileReader object.
An example is:
var jpeg = new JpegMeta.JpegFile(this.result, this.file.name);
After creation you can then access various meta-data properties, categorised by meta-data groups. The main groups of meta-data are:
- general
- Information extracted from the JPEG SOF segment. In particular the hieght, width and colour depth.
- jfif
- Meta-data from the JFIF APP0 segment. This usually includes resolution, aspect ratio and colour space meta-data.
- tiff
- Generic TIFF meta-data extracted from the Exif meta-data APP1 segment. Includes things such as camera make and model, orientation and date-time.
- exif
- Exif specific meta-data extracted from the Exif meta-data APP1 segment. Includes camera specific things such as white balance, flash, metering mode, etc.
- gps
- GPS related information extracted from the Exif meta-data APP1 segment. Includes atitude, longitude etc.
Meta-data groups can be access directly, for example:
var group = jpeg.gps;
A lookup table is also provided: jpeg.metaGroups. This
associative array can be used to determine which meta-groups a
particular jpeg file instance actually has.
The MetaGroup object has a name field, a description field and
an associative array of properties.
Properties in a given group can be accessed directly. E.g:
var lat = jpeg.gps.latitude;
Alternatively, the metaProps associative array provides
can be used to determine which properties are available.
The metaProp object has a name field,
description field, and also a value
field.
Conclusion
The File API adds a poweful new capability to native HTML5 applications.
Un-shortening twitter URLs
Introduction
I got a little annoyed at twitter the other day for gratiutiously tinyurl-ing a link, even when my message was under the magic 140 characters. Anyway, as far as I can tell, there isn’t really anyway to avoid this.
As it happens there has been quite a kerfuffle about this in the blog-o-sphere of late, with Josh Schachter (founder of del.icio.us) discussing some of the problems with URL shorteners. While, I agree with most of what is written there, I think some of his points are a little over the top. Personally, i don’t think that URL shorteners are evil, but they can certainly be annoying. I regularly mouseover links to see where they are going, and like the visual feedback of seeing visited links, both of which are broken by URL shorteners.
So, for my problem, there is a pretty easy solution, somehow expand the URL, and show that instead of the shortened URL. There are clearly already solutions to solve this, but I was interested in seeing if this could be done purely within the browser, using JavaScript and standard (or proposed standard) DOM APIs.
Overall approach
Now, in general, these URL shortening services do the “right thing”, and provide a 301 permanent redirect response. So my basic thinking is something like:
- get the tweets from twitter
- find all the URLs in the tweets
- for each URL, do an HTTP HEAD request
- if the response code is 301, replace the link text (and href) with the response location.
.. and the plan is to try and do this on the client. Now, if you’ve done something like this before, you can probably already guess all the pitfalls!
Render the tweet data
Twitter quite conveniently provides its data in a variety of useful machine readable formats. The most useful for my purposes here is the twitter XML format. Now, I’m leaving it as an exercise for the reader how to get an (authenticated) bit of twitter XML into the client at run-time. For now, I’ve downloaded a my latest timeline in XML format and made it available at http://benno.id.au/twit/twit.xml.
Now, in these days of JSON and JavaScript, no-one much seems to like XSLT langauge, but I’m quite partial to it, and as far as I’m concerned it is the best way to take one XML document (the tweet timeline) and convert it into another XML document (the HTML output). (Technically, I’ll be converting it into an XML document fragment.)
I’ll leave out how to create an XSLT processor in Javascript as an exercise for the reader, but the XSLT script itself is of some interest. At least I think so!
Now, for the most part converting the Twitter status element into an appropriate <div> and <p> tags is straight forward:
<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" />
<xsl:template match="statuses">
<h1>Tweet!</h1>
<xsl:apply-templates/>
</xsl:template>
<xsl:template match="status">
<div class="tweet">
<div class="text">
<p><xsl:value-of select="text" /></p>
</div>
<div class="meta">
<span><xsl:value-of select="created_at" /> from <xsl:value-of select="source"/></span>
</div>
</div>
</xsl:template>
</xsl:stylesheet>
What this script basically does is find a tag <statuses> and output the heading tag. The for each <status> tag, it generates a div with the actual tweet. Now this works pretty well, but, the tweets are stored in plain text, without the URLs marked up in anyway! E.g:
<status> <created_at>Thu Apr 23 08:06:49 +0000 2009</created_at> <id>1592566033</id> <text>I can't believe the successor of the Radius protocol is the diameter protocol: http://tinyurl.com/c3zb2v</text> <source>web</source> .... </status>
This means the script currently only prints out URLs with no link markup. Unfortunately the XSLT language doesn’t have very powerful inbuilt string handling functions. Fortunately the XSLT language is pretty general purpose (and turing complete. Confusingly, functions are called templates, and the syntax to invoke a function is far from convenient, but it is relatively straight forward to parse a string to extract and markup links.
So, we write a function, err, template, called
parseurls. This template takes a single parameter called
text. It then outputs this text with anything that looks like a URL
replace with <a href="URL">URL</>. For
this proof-of-concept, anything that starts with http:// is
going to count as a URL, which works relatively well in practise.
The basic algorithm is to split the string into three parts: before-url, url, after-url. before-url is simply output as-is, url is output as described previously. If there is an after-url part, then we recursively call the template with this data.
<xsl:template name="parseurls">
<xsl:param name="text"/>
<xsl:choose>
<xsl:when test="contains($text, 'http://')">
<xsl:variable name="after_scheme" select="substring-after($text, 'http://')" />
<xsl:value-of select="substring-before($text, 'http://')"/>
<xsl:choose>
<xsl:when test="contains($after_scheme, ' ')">
<xsl:variable name="url" select="concat('http://', substring-before($after_scheme, ' '))" />
<xsl:call-template name="linkify"><xsl:with-param name="url" select="$url"/></xsl:call-template>
<xsl:text> </xsl:text>
<xsl:call-template name="parseurls">
<xsl:with-param name="text" select="substring-after($after_scheme, ' ')"/>
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
<xsl:variable name="url" select="concat('http://', $after_scheme)"/>
<xsl:call-template name="linkify"><xsl:with-param name="url" select="$url"/></xsl:call-template>
</xsl:otherwise>
</xsl:choose>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="$text"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
So, through the power of functional programming we can transform the raw tweet XML into a useful HTML document. Unfortunately making HTTP request is beyond the scope of XSLT, so for the next step, we need to use JavaScript.
Replacing URLs
OK, the next step is to go and change the links in the document. The goal is to find any links that have been shortened and replace them with the true destination.
Finding the links in the document is relatively striaght-forward, something
like: var atags = document.getElementsByTagName("body")[0].getElementsByTagName("a");,
does the trick. Now the aim is to go and see if any of these URL return a
redirect, and then update the element. What I want to do is something like:
var atags = document.getElementsByTagName("body")[0].getElementsByTagName("a");
for (var i = 0; i < atags.length; i++) {
var x = atags[i];
var xmlhttp = new XMLHttpRequest();
xmlhttp.update_x = x;
xmlhttp.open("HEAD", x.attributes.getNamedItem("href").value);
xmlhttp.onreadystatechange = function () {
if (this.readyState == 4) {
if (this.status == 301) {
this.update_x.innerHTML = this.getResponseHeader('Location');
}
}
}
xmlhttp.send(null);
}
Basically, what this code does (or what I wish it did), is grab the href attribute out of each link, and do a HEAD request on it. Recall, a head request will get the resource headers, but not the contents. Since we only care about the reponse code, and the location header, we do the network a favour, and don’t download all the uncessary data.
Unfortunately, at this stage, we end up pretty hard against some fundamental limitations of the XMLHTTPRequest, specifically, the same-origin policy. In short, (simplified) terms, the same-origin policy means you can’t make HTTP requests except to the domain where the page resides. This is done for a very good security reason, but is slightly frustrating at this point.
Now, I wasn’t going to let a simple thing like this stop me, so I
implemented a pretty simple server side proxy which let me avoid this
problem. (This is a pretty unsatisfactory solution, and I’m definitely
looking forward to some of the new cross-origin extensions coming out.)
So, basically, we change the open call to something like:
xmlhttp.open("HEAD", "proxy?" + x.attributes.getNamedItem("href").value);
OK, so the server-side proxy gets around our cross-origin restriction, unfortunately we now hit a new problem! It turns out that XMLHttpRequest is specified so that any redirects are automatically followed by the implementation. Which means, we are stuck again, because we don’t even get a chance to find out about redirections! To get around this I hacked up my proxy, so that it converted 301 response codes into a 531 response code. (No particular reason for choosing that number over any other available reponse code).
Putting all this together gives us a solution for doing URL elongating (mostly) on the client-side. I’ve put an example up at http://benno.id.au/twit/.
Conclusions and further work
As you can see from the example, modifying the links at afer the page has already rendered can be somewhat distracting. An alternative user-interface would be to hook the mouse-over event, and in those cases display the real URL in the status-bar.
Obviously this has the potential to create a large number of network requests, but this would be no different to a desktop application, or browser plugin, so not really much that can be done there. It might be possible to batch a number of requests to the proxy, and also have the proxy cache requests, but I’d prefer to find solutions to needing the proxy cache in the first place!
The W3C Cross-Origin Resource Working Draft, provides a mechanism which allows holes to be punched in the same-origin restricting. If URL shortening services allowed their resources to be shared cross-origin, by implementing the Access-Control-Allow-Origin http header, the need for the proxying mechanism would go away.
Finally, I would propose that the XMLHttpRequest API be updated to enable a mechanism to avoid following redirects. The W3C working draft notes that a Property to disable following redirects; is being considered for future version of the specification. I would be in favour of this.
Unfortuantely the conclusion needs to be that with the current API limitations and security models, it is not possible to write this kind of application in a pure client-side manner with current web technologies.