Benno's Blog

distrusting git

Sat, 01 Oct 2011 07:06:11 +0000

tl;dr git destroyed my data; my team now has severe trust issues with git

We ask a lot from our source control systems. We want them to be flexible, fast, distributed, clever and even easy-to-use. But the number 1 thing we should demand from a source control system is that it doesn’t destroy our data. Probably most importantly, it shouldn’t ever lose stuff that has been committed, but just behind that it really shouldn’t destroy data in our working directory.

When you find out that your source control system has lost your data you end up in a very bad place. Once your source control system destroys your data once, you immediately have a severe break-down of trust between yourself and your tool. You revert to using cp -R to create backups before doing anything with the tool, just in case it destroys your data again.

Development was proceeding along at a cracking pace, and I was getting ready to commit a series of important changes. Before doing so, I want to merge in the recent changes from the remote master, so I do the familiar git pull. It complained about some files that would be overwritten by the merge, so I saved a backup of my changes, then reverted my changes in those specific files, and proceeded. The merge went went fine, and pulled in all the remote changes to the working directory. Getting ready for the commit, I do a git status and start to get a little concerned; one of the files I’ve been heavily editting doesn’t show up in the status list. I cat the file to see what is going on; seems none of my changes are there. Now, I’m getting concerned, maybe I’m going slightly crazy after 3 days straight hacking, but I’m sure I made some changes to this file. I scroll up the terminal backlog to the git status I did before the pull. Sure enough, the file is marked as changed there, but not after the merge. I carefully read the full details from the merge; my file isn’t listed being touched there. Now I am really worried. git has just gone and destroyed the last 5 or 6 hours worht of work. Not happy!

Luckily, I was able to reconstruct most of the work from editor buffers, which I luckily still had open.

But, now I am worried. Why the fuck did git decide to destroy data in my working directory, without even telling me!. Did I do something wrong? Is this a case I should know about? I had to investigate.

So, I took a snapshot of my repository, rolled back to the revision before the merge, mad some minor modifications to my file, the ran the merge again. And, again, git destroys the change in the working directory. Now this isn’t normal behaviour, something is really fucked. The only thing slightly interesting about the file in question is that it had been recently renamed. Somehow this rename had confused the merge, and the merge was silently overwriting files.

Now git has a few different merge strategies, so I tried out some different ones. This was a simple pretty simple merge with 2-heads so the options were really recursive or resolve. git picks recursive be default, so I tried resolve instead. This worked fine. Surprsingly this made me feel a little better, I wasn’t completely crazy, silently updating files in my working directory wasn’t intended behaviour, there had to be something wrong in recursive merge.

So, I updated to the latest version in homebrew. Same problem.

Then it was time to start debugging git for real. So I downloaded the source (using git of course). I started having a look through merge-recursive.c. It didn’t look too bad, but there was clearly going to be a lot to learn if I was going to debug this. Before I started literring the code with prints I thought I better just see if head had the same problem. Lo and behold head worked! OK, cool, they fixed the bug. But that isn’t really a satisfying answer. Just for fun I checked out some random version to try and narrow down when the bug was fixed. In doing so I found that actually it worked in some old vesions, then didn’t work, and then finally worked again in the very latest. Here are my raw notes:

1.7.1               => good
1.7.2               => good
1.7.3               => good
1.7.4               => bad
1.7.5               => bad              
1.7.6.1 (installed) => bad
1.7.6.1 (checkout)  => bad
1.7.6.4             => bad
1.7.7-rc0           => fail
1.7.7-rc1           => pass
1.7.7-rc3           => pass

OK, this is getting more interesting. So somewhere between 1.7.2 and 1.7.3 this bug was introduced. I started using git bisect to narrow things down. I quickly got bored of manually doing git bisect good and git bisect bad, luckily I stumbled upon git bisect run that automates the whole process. After about 20 minutes compiling and testing it found the bad commit.

commit 882fd11aff6f0e8add77e75924678cce875a0eaf
Author: Elijah Newren 
Date:   Mon Sep 20 02:29:03 2010 -0600

    merge-recursive: Delay content merging for renames
    
    Move the handling of content merging for renames from process_renames() to
    process_df_entry().
    
    Signed-off-by: Elijah Newren 
    Signed-off-by: Junio C Hamano

OK, lots of talk about merge-recursive and renames. That sounds like it makes sense; at least there is a specific bit of code that I can blame for my data destruction, maybe I don’t have to distrust the whole tool.

But to really be confident, I want to think that the fix isn’t just something random, and was actually done to fix this this problem. So I switched the return code in my test script, and ran git bisect again to find when the bug was fixed. Eventually it found this commit:

commit 5b448b8530308b1f5a7a721cb1bf0ba557b5c78d
Author: Elijah Newren 
Date:   Thu Aug 11 23:20:10 2011 -0600

    merge-recursive: When we detect we can skip an update, actually skip it
    
    In 882fd11 (merge-recursive: Delay content merging for renames 2010-09-20),
    there was code that checked for whether we could skip updating a file in
    the working directory, based on whether the merged version matched the
    current working copy.  Due to the desire to handle directory/file conflicts
    that were resolvable, that commit deferred content merging by first
    updating the index with the unmerged entries and then moving the actual
    merging (along with the skip-the-content-update check) to another function
    that ran later in the merge process.  As part moving the content merging
    code, a bug was introduced such that although the message about skipping
    the update would be printed (whenever GIT_MERGE_VERBOSITY was sufficiently
    high), the file would be unconditionally updated in the working copy
    anyway.
    
    When we detect that the file does not need to be updated in the working
    copy, update the index appropriately and then return early before updating
    the working copy.
    
    Note that there was a similar change in b2c8c0a (merge-recursive: When we
    detect we can skip an update, actually skip it 2011-02-28), but it was
    reverted by 6db4105 (Revert "Merge branch 'en/merge-recursive'"
    2011-05-19) since it did not fix both of the relevant types of unnecessary
    update breakages and, worse, it made use of some band-aids that caused
    other problems.  The reason this change works is due to the changes earlier
    in this series to (a) record_df_conflict_files instead of just unlinking
    them early, (b) allowing make_room_for_path() to remove D/F entries,
    (c) the splitting of update_stages_and_entry() to have its functionality
    called at different points, and (d) making the pathnames of the files
    involved in the merge available to merge_content().
    
    Signed-off-by: Elijah Newren 
    Signed-off-by: Junio C Hamano

OK, this is good. Looks like they fixed the bug, and it even references the bad commit that I had narrowed things down to.

So I’m a little bit dismayed that this bug existed for almost a full year before being fixed. I can’t be the only person to have been hit by this problem can I? I looked at the release notes for v1.7.7. This is what they have to say abou the issue:

 The recursive merge strategy implementation got a fairly large
 fix for many corner cases that may rarely happen in real world
 projects (it has been verified that none of the 16000+ merges in
 the Linux kernel history back to v2.6.12 is affected with the
 corner case bugs this update fixes).

OK, so the bug never trigged in 16,000+ Linux kernel merges. Strangely that doesn’t actually make me feel any better.

So, I don’t think git sucks. All software has bugs, but bugs that destroy data are pretty devastating. It is a little hard to trust git merge operations now. I’ll probably try to make sure I don’t merge on to a working directory (i.e: stash my changes first, since then they are at least backed up on the object database).

Of course convincing my colleagues, who were also affected by this bug, and didn’t really have any love for git in the first place, that git isn’t completely broken is going to be a tough sell.

node.js and IPv6

Sat, 13 Aug 2011 09:46:35 +0000

tl;dr

node.js makes it real easy to create an IPv6 server.
just do: server.listen(, ). E.g: server.listen(80, '::1')
my javascript style is weird
IPv6 URls are a little weird, the IPv6 address is surround by square brackets. E.g: http://[::1]:80/

OK, my last few posts on node.js may have seemed a little negative. While there are some things in node.js that seem a little more complicated than necessary, there are some things that are nice and simple, such as getting your server to run on both IPv4 and IPv6. This post is a little late for World IPv6 Day, but better late than never!

So this post isn’t about configuring IPv6 on your machine in general. I’m going to assume that your local network interface has an IPv6 address. You can probably check this with the output of ifconfig. On my Darwin box it looks something like:

benno@ff:~% ifconfig lo0
lo0: flags=8049 mtu 16384
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
	inet 127.0.0.1 netmask 0xff000000

You should see the local interface bound to the IPv6 localhost address ::1 as well the IPv4 localhost address 127.0.0.1. So lets get started with a simple little IPv4 server.

var http = require('http')
var server

function onRequest(req, res) {
    console.log(req.method, req.url)
    res.writeHead(200, {'Content-Type': 'text/plain'})
    res.end('Hello World\n')
}

function onListening() {
    console.log('Listening at http://' + this.address().address + ':' + this.address().port + '/')
}

server = http.createServer()
server.on('request', onRequest)
server.on('listening', onListening)
server.listen(1337, '127.0.0.1')

This is a slight variation on the canonical node.js Hello World example. A few things worth noting:

I don’t like semi-colons. (And this is coming from a guy who mostly program in C!)
I don’t like anonymous functions. (But that is probably a whole other post!)
I prefer using the on method rather than the verbose addListener
I like adding my event handlers before calling a method that might emit the event.
I prefer to explicitly register my event handlers, rather than have them magically added for me.
I like displaying important messages in asynchronous callback, rather than assuming everything just worked.
I don’t like duplicating data, even in a small test program!

So, apart from my stylistic quirks, the above should be fairly straight forward. The only new thing functionality wise compared to the normal node.js example is the addition of some trivial logging in the request handler.

So our quest is going to be to add support for IPv6. Before we do that though, I’m going to improve our logging a bit. Just because we are supporting IPv6, doesn’t mean we want to stop our server running on IPv4, so we are going to end up with multiple servers running at once. Once this happens, our logging might get a bit confusing. So we’re going to give our servers a name, and include that in the logging.

var http = require('http')
var server

function onRequest(req, res) {
    console.log('[' + this.name + ']', req.method, req.url)
    res.writeHead(200, {'Content-Type': 'text/plain'})
    res.end('Hello World\n')
}

function onListening() {
    console.log('[' + this.name + '] Listening at http://' + this.address().address + ':' + this.address().port + '/')
}

server = http.createServer()
server.name = 'ipv4server'
server.on('request', onRequest)
server.on('listening', onListening)
server.listen(1337, '127.0.0.1')

Because Javascript objects are open we can trivially add a name field to our objects, and then use this when logging. In general I avoid messing with objects created by other modules, but it is the quick and easy approach in this case.

OK, so on to IPv6. As a first stab at it, we get something like this:

var http = require('http')
var server

function onRequest(req, res) {
    console.log('[' + this.name + ']', req.method, req.url)
    res.writeHead(200, {'Content-Type': 'text/plain'})
    res.end('Hello World\n')
}

function onListening() {
    console.log('[' + this.name + '] Listening at http://' + this.address().address + ':' + this.address().port + '/')
}

ipv4server = http.createServer()
ipv6server = http.createServer()

ipv4server.name = 'ipv4server'
ipv6server.name = 'ipv6server'

ipv4server.on('request', onRequest)
ipv6server.on('request', onRequest)

ipv4server.on('listening', onListening)
ipv6server.on('listening', onListening)

ipv4server.listen(1337, '127.0.0.1')
ipv6server.listen(1337, '::1')

Basically, creating an IPv6 server is exactly the same as creating an IPv4 server, except you use an IPv6 address literal (i.e: ::1) to specify the local address to bind to, rather than an IPv4 address literal. You can see that there is absolutely no problem sharing the event handlers between the two different servers. The this variable in each event handler function refers to the server itself, so you can handle cases that are server specific if necessary.

When you run this you should get some output like:

[ipv4server] Listening at http://127.0.0.1:1337/
[ipv6server] Listening at http://::1:1337/

Which looks pretty good. You can try going to the IPv4 server URL in your browser. If you try the IPv6 URL, you will probably run in to some problems. This is because you need some escaping of the IPv6 literal address in the URL, or it can’t be parsed correctly (what with there being all those colons which are usually used for separating the port number). So the correct URL should be: http://[::1]:1337/. We better fix this bug in the code:

function onListening() {
    var hostname = this.type === 'tcp4' ? this.address().address : '[' + this.address().address + ']'

    console.log('[' + this.name + '] Listening at http://' + hostname + ':' + this.address().port + '/')
}

OK, that’s looking pretty good now, if you start hitting those URLs on the different address you should get some useful output such as:

[ipv4server] Listening at http://127.0.0.1:1337/
[ipv6server] Listening at http://[::1]:1337/
[ipv4server] GET /
[ipv4server] GET /favicon.ico
[ipv6server] GET /
[ipv6server] GET /favicon.ico

Now, I mentioned earlier I don’t like duplicating data. I also don’t like duplicating code either, so let’s refactor this a little:

function startServer(name, address, port) {
    var server = http.createServer()
    server.name = name
    server.on('request', onRequest)
    server.on('listening', onListening)
    server.listen(port, address)
    return server
}

startServer('ipv4server', '127.0.0.1', 1337)
startServer('ipv6server', '::1', 1337)

So, in conclusion it is easy-peasy to run your web application on IPv6, and even on IPv4 and IPv6 using the exact same script.

Safely dropping privileges in node.js

Tue, 09 Aug 2011 15:12:25 +0000

tl;dr

Don’t run as root, use process.setuid to drop privileges.
Callbacks make it difficult to work out the exactly right time to call setuid(); be careful!

So, you want to run some kind of TCP server, and you’d like to run it on one of those fancy ports with a number less-than 1024. Well, unfortunately you got to be root to bind to a low-numbered port. Of course, we don’t want to run our network server as root, because that would be, well, really silly, wouldn’t it! Luckily, POSIX gives us a simple way of breaking this little problem. You start your program running with root privileges, grab all the resources you need, and then drop back to running as an unprivileged user using the setuid syscall.

Now, if you are writing a network server you probably know the drill, you create a socket(), then you bind(), and then you starts to listen() for connections, occasionally calling accept() when you decide you want to actually do something with an incoming request. So, the question is, at which point do you drop the privileges? Well, the important part is that you need privileges to bind(), but once you have bound to an address and port, you no longer need root privileges. So ideally, you call setuid() after you bind(). You want to get this right. Drop privileges too early and you can’t correctly bind to the address, drop too late and you unnecessarily expose yourself to potential exploits.

Now, if you are doing something in normal synchronous programming you would do something like:

fd = socket(...)
bind(fd, ...)
setuid(...)
listen(fd, ...)

But good luck on things being so simple in node.js. In my last post I described these semi-asynchronous functions, which you probably thought was just a bit of an academic exercise. Well, it turns out that, depending on the arguments, the listen method behaves in this semi-asynchronous manner.

Specifically, when the listen function returns, the bind() operation has completed, but the listen() operation hasn’t. Which means that calling process.setuid() immediately after server.listen() will end up dropping privileges at the ideal time.

This technique is explained in this excellent post on the subject. However, I’m not 100% satisfied with this solution. My unease with this approach comes down to the fact that there is no documented guarantee that the bind() must have occurred when the function returns, it could change in the next version. In fact, depending on the arguments passed to listen, it may not happen that way. If instead of using an IP address to specify the local address to bind to, you use a domain name, then an asynchronous DNS lookup occurs before the call to bind(), which means that when server.listen() returns the bind call has not yet happened, if you drop the privileges at this point then you will hit an exception later when the bind() happens. Of course, specifying the local address to which your server binds using a DNS name is a little bits silly in the first place, but that is another matter.

So, if we can’t rely on the bind() having occurred when server.listen() returns then the only other option is to call setuid in the listen callback function. This is probably a reasonable approach, but it does mean that we hold privileges longer than strictly necessary. In this case, there probably isn’t really very much that happens between the bind() call and when the listen event triggers, so it doesn’t really matter, but I’d still like to find a solution that avoids both of these problems.

Thankfully, node.js is pretty flexible and provides a listenFD() method that we can take advantage of. This lets us set up our own socket first, with whatever exact timings we want, and then let the class know about the socket we created.

It turns out that writing function to create an appropriate socket isn’t too hard as most of the low-level functions are available if you know where to look. So I present you with safeListen

function safeListen(server, port, address, user) {
    var ip_ver = net_binding.isIP(address)
    var fd
    var type

    switch (ip_ver) {
    case 4:
	type = 'tcp4'
	break
    case 6:
	type = 'tcp6'
	break
    default:
	throw new Error("Address must be a valid IPv4 or IPv6 address.")
    }

    fd = net_binding.socket(type)

    net_binding.bind(fd, port, address)

    if (user) {
	process.setuid(user)
    }

    net_binding.listen(fd, server._backlog || 128)

    /* Following the net.js listen implementation we do this in the
     nextTick so that people potentially have time to register
     'listening' listeners. */
    process.nextTick(function() {
	server.listenFD(fd, type)
    })
}

Instead of using server.listen(address, port) use safeListen(server, address, port, user). If you like monkey patching you can probably attach the function as a method to the server object and then make the call look like server.safeListen(address, port, user). This function essentially does the same thing as listen but if a user argument is specified, it will call setuid to drop privileges after calling bind(). The main limitation compared to the normal listen() method is that the address must be specified, and must be an IP address, rather than a hostname.

node.js semi-asynchronous functions

Mon, 08 Aug 2011 19:19:28 +0000

tl;dr

After you call an asynchronous function you can’t make any assumptions about the state of the objects the asynch function touches.
When you pass a callback to a function it may be called on the current stack, or a new stack
Existing documented patterns on how to do loops with asynch functions make big assumptions about callbacks being executed on new stack.
If you provide an API please document what assumptions a caller can and can’t make about the state of objects both immediately after calling, and after the callback completes.
If you provide an API which takes a callback, always execute that callback on a new stack.

Last time I wrote about some of the idiosyncrasies in the way in which you deal with exceptions in node.js. This time, I’m looking at a phenomenon I’m calling semi-asynchronous functions.

Let’s start with a simple asynchronous function. We have a function x which sets the value of two global variables. Of course global variables are bad, so you could imagine that x is a method and it is updating some fields on the current object if it makes you feel better. Of course some will argue that any mutable state is bad, but now we are getting side-tracked!

var a = 0
var b = 0

function x(new_a, new_b) {
    a = new_a
    b = new_b
}

So, here was have a pretty simple function, and it is pretty easy to state the post-condition that we expect, specifically that when x returns a will have the value of the first argument and b will have the value of the second argument.

So, let’s just write some code to quickly test our expectations:

x(5, 6)
console.log(a, b)

As expected this will print 5 6 to the console.

Now, if x is changed to be an asynchronous function things get a little bit more interesting. We’ll make x asynchronous by doing the work on the next tick:

function x(new_a, new_b, callback) {
    function doIt() {
	a = new_a
	b = new_b
	callback()
    }
    process.nextTick(doIt)
}

Now, we can guarantee something about the values of a and b when the callback is executed, but what about immediately after calling? Well, with this particular implementation, we can guarantee that a and b will be unchanged.

function done() {
    console.log("Done", a, b)
}

x(5, 6, done)
console.log("Called", a, b)

Running this we see that our expectations hold. a and b are 0 after x is called, but are 5 and 6 by the time the callback is executed.

Of course, another valid implementation of x could really mess up some of these assumptions. We could instead implement it like so:

function x(new_a, new_b, callback) {
    a = new_a
    function doIt() {
	b = new_b
	callback()
    }
    process.nextTick(doIt)
}

Now we get quite a different result. After x is called a has been modified, but b remains unchanged. This is what I call a semi-asynchronous asynchronous function; part of the work is done synchronously, while the remainder happens some time later.

Just in case you are thinking at this point that this is slightly academic, there are real functions in the node.js library that are implemented in this semi-asynchronous fashion.

Now as a caller, faced with this semi-asynchronous functions, how exactly should you use it? If it is clearly documented which parts happen asynchronously and which parts happen synchronously and that is part of the interface, then it is relatively simple, however most functions are not documented this way, so we can only make assumptions.

If we are conservative, then we really need to assume that anything modified by the function must be in an undefined state until the callback is executed. Hopefully the documentation makes it clear what is being mutated so we don’t have to assume the state of the entire program is undefined.

Put another way, after calling x we should not rely on the values a and b in anyway, and the implementer of x should feel free to change when in the program flow a and/or b is updated.

So can we rely on anything? Well, it might be nice to rely on the order in which some code is executed. With both the implementation of x so far, we have been able to guarantee that the code immediately following the function executes before the asynchronous callback executes. Well, that would be nice, but what if x is implemented like so:

function x(new_a, new_b, callback) {
    a = new_a
    b = new_b
    callback()
}

In this case, the callback will be executed before the code following the call to x. So, there are two questions to think about. Is the current formulation of x a valid approach? And secondly, is it valid to rely on the code ordering?

While you think about that, let me introduce another interesting issue. Let’s say we want to execute x many times in series (i.e: don’t start the next x operation until the previous one has finished, i.e: it has executed the callback.). Well, of course, you can’t just use something as simple as a for loop that would be far too easy, and it would be difficult to prove how cool you are at programming if you could just use a for loop. No instead, you need to do something like this:

var length = 100000;
function repeater(i) {
  if( i < length ) {
      x(i, i,  function(){
	  repeater(i + 1)
      })
  }
}
repeater(0)

This appears to be the most widely used approach. Well there is at least one blog post about this technique, and it has been tied up into a nice library. Now, this works great with our original implementations of x. But try it with the latest one (i.e: the one that does the callback immediately). What happens? Stack overflow happens:

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
RangeError: Maximum call stack size exceeded

So now the question isn’t just about whether the code ordering is a reasonable assumption to make, now we need to work out whether it is a reasonable assumption to make that the callback gets a new stack each time it is called! Once again, if it is clearly documented it isn’t that much of a problem, but none of the standard library functions document whether they create a new stack or not.

The problem here is that common usage is conflicting. There is a lot of advice and existing libraries that make the assumption that a callback implies a new stack. At the same time there is existing code within the standard library that does not create a new stack each time. To make matters worse, this is not always consistent either, it can often depend on the actual arguments passed to the function as to whether a new stack is created, or the callback is executed on the existing stack!

What then can we make of this mess? Well, once again, as a caller you need to make sure you understand when the state is going to be mutated by the function, and also exactly when, and on which stack your callback will be executed.

As an API provider as always, you need to document this stuff, but lets try to stick to some common ground; callback should always be executed in a new stack, not on the existing one.

node.js exceptions

Mon, 08 Aug 2011 16:10:28 +0000

tl;dr

If you are calling functions in node.js work out if they throw exceptions or emit error events (or both). Handle these appropriately.
If you are writing functions, clearly document which exceptions you throw and if you raise error events.

One of the best things about asynchronous, callback based programming is that basically all those regular flow control constructs you are used to are completely broken. However, the one I find most broken is the handling of exceptions.

Javascript provides a fairly familiar try...catch construct for dealing with exceptions. The problems with exceptions is that they provide a great way of short-cutting errors up a call stack, but end up being completely useless of the error happens on a different stack.

Here is a simple example to get started:

function x() {
    throw new Error('my silly error')
}

x()

If we run this in node, the result is fairly intuitive, we get a nasty traceback:

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: my silly error
    at x (/Users/benno/apkudo/test.js:2:11)
    at Object. (/Users/benno/apkudo/test.js:5:1)
    at Module._compile (module.js:402:26)
    at Object..js (module.js:408:10)
    at Module.load (module.js:334:31)
    at Function._load (module.js:293:12)
    at Array. (module.js:421:10)
    at EventEmitter._tickCallback (node.js:126:26)

Now if we ignore the first few lines of junk, the rest is a fairly familiar traceback. You’ll note that we are already pretty deep in a stack trace even for this very simple function. You can mostly ignore everything from Module._compile onwards.

Now instead of doing this we might want to, instead, catch this error and write some code to handle the error. We aren’t going to do anything earth shattering in the handler, just print out the exception and continue on our merry way.

function x() {
    throw new Error('my silly error')
}

try {
    x()
} catch (err) {
    console.log("Error:", err)
}

Now, if you run this you get:

Error: { stack: [Getter/Setter],
  arguments: undefined,
  type: undefined,
  message: 'my silly error' }

So far, so good. Just what you would expect in the normal world of programming. Let’s spice things up a bit; let’s make x asynchronous. We’ll create a wrapper function y which will take two arguments. The first argument indicates whether to execute x synchronously or asynchronously. The second argument is a function that will be called on completion. Something like this:

function y(arg, callback) {
    if (arg === 1) {
	x()
	callback()
    } else {
	function onTick() {
	    x()
	    callback()
	}
	process.nextTick(onTick())
    }
}

Now this setup may seem a tad contrived, but in the real world we get situations not all that different to this. For example the built-in listen method may do a DNS lookup on the host argument if it is not a dotted decimal. If it is a dotted decimal though, no lookup is required. So, we change our calling code appropriately:

try {
    y(1, function () { console.log("Callback") })
} catch (err) {
    console.log("Error:", err)
}

Running this gets us essentially the same output as before: we successfully catch the exception and then we are on our way. Let’s change our calling code slightly though, so that we hit the asynchronous path:

try {
    y(0, function () { console.log("Callback") })
} catch (err) {
    console.log("Error:", err)
}

Running this we now find that we get an ugly traceback. We completely failed in catching the exception:

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: my silly error
    at x (/Users/benno/apkudo/test.js:2:11)
    at Array.onTick (/Users/benno/apkudo/test.js:11:6)
    at EventEmitter._tickCallback (node.js:126:26)

What happened here is that when y hits the asynchronous path it creates an entirely new call stack, on that isn’t protected by a try..catch block at the top of the call stack. So we end up with the default node exception handling code. You can see how the call stack in this case is much shorter.

How can we deal with this? Well, one way is that we just don’t do exception like things, and always explicitly return errors or pass them as arguments to callbacks. The other option is to use the event system provided by node.js. That is what we will look at next as it is what node.js uses internally. We are going to change our code so that y emits a myerror event rather than the exception bubbling up.

var events = require('events')
emitter = new events.EventEmitter()

function y(arg, callback) {
    if (arg === 1) {
	x()
	callback()
    } else {
	function onTick() {
	    try {
		x()
	    } catch(err) {
		emitter.emit('myerror', err)
		return
	    }
	    callback()
	}
	process.nextTick(onTick)
    }
}

In this example we are just using a global emitter object. In a real example x would likely be a method on an object that sub-classed the EventEmitter class. If we run the code now don’t get any output at all! This is because we haven’t yet attached a listener to the myerror event. We can do that like so:

emitter.on('myerror', function(err) { console.log("Error:", err) })
y(0, function () { console.log("Callback") })

Now, when we run it we get the same type of output as we did when we were catching exceptions:

Error: { stack: [Getter/Setter],
  arguments: undefined,
  type: undefined,
  message: 'my silly error' }

Now, you might have picked up a slight problem with the above approach. If we don’t catch the exception by registering a handler for the myerror event nothing happens; the exception is essentially ignored. This is different to normal exceptions in Javascript that will bubble right up to the run-time for reporting.

Now, we could ensure that there is always a default handler for the myerror event which dumped the traceback and exits, but you would also need to work out if another listener already handled the error or not, and so on. It turns out that node.js has already solved this problem, so instead of inventing our own event name we can use the special error event. Events called error are treated very specially by the node code. From the emit code in event.js:

  if (type === 'error') {
    if (!this._events || !this._events.error ||
        (isArray(this._events.error) && !this._events.error.length))
    {
      if (arguments[1] instanceof Error) {
        throw arguments[1]; // Unhandled 'error' event
      } else {
        throw new Error("Uncaught, unspecified 'error' event.");
      }
      return false;
    }
  }

Of course, registering an error event listener doesn’t magically trap any exceptions that might be raised in the old fashioned way. This means if you are writing code, you really need to understand which type of error handling approach the code you are calling uses and ensure you handle it appropriatley.

Unfortunately, most APIs don’t actually document this error handling behaviour, so you are forced to go and read the code to work out exactly what is going on. If you are writing an API, please make it easy on your fellow developer by documenting this kind of thing.

Supporting the Ada Initiative

Tue, 21 Jun 2011 20:00:06 +0000

The Ada Initiative is a relatively new organisation that has been started to help increase the participation of women in, among other things, open source software. Now the question some people might ask, in fact do ask, is why does the open source community need such an organisation at all? Well, firstly the participation rate is currently very small (< 2% according a 2002 survey [PDF]). I think this low participartion rate matters on two fronts.

Firstly, it has an overall negative impact of the open source community. There is the direct loss attributable to the fact that we miss out on the contributions of many excellent developers. Additionally, the are indirect costs. Also I think that having a diverse community working on any project brings a variety of ideas to the project that can dramatically improve the project.

Secondly, and more importantly, it matters to all the inviduals who miss out on participating in the open source community. I don’t think I really appreciated this perspective before becoming a father. I’d be pretty upset if my daughter missed out on being involved in the open source community because of some of the unnecessary challenges that currently exist for women in the community.

I really hope that if my daughter wants to get involved in the open source the Ada Initiative will be there to support. With any luck many of the challenges women currently face in the open source community will have been solved.

If your are involved in the open source community and would like to see more done to support women with in the community, I’d encourage you to become an Ada Initiative supporter.

Determining Dependencies

Tue, 15 Mar 2011 18:58:48 +0000

One of the best ways to make a build system fast is to avoid the unnecessary rebuilding of files. Build tools have a variety ways of achieving this. To better discuss this, let’s first define some terms. One way to look at a build system is that it takes some set of inputs in to some set of outputs. Usually these inputs and outputs are files in the file-system, but could potentially be something else, like tables in a database or anything else. Usually the build system consists of a set of build rules; each build rule has some set of inputs, and produces a set of outputs, by running a given build command. You’ll have to forgive the abstruse nature of these definitions, but I’m attempting to keep the design space as open as possible!

So to improve the speed of the build system we want to avoid executing unnecessary build commands. To do this in any reasonable way requires making some assumptions about the individual build rules, specifically that the output for a build rule only depends on the build rule’s inputs. With such an assumption there is no need to rerun a build command if the inputs of a build rule have not changed.

This in itself is an interesting restriction as the inputs to a build rule may not be entirely obvious. For example, the output of a build rule may depend on the time, or the user name of the hostname. The other problem is build commands that have inputs which are also outputs (i.e: commands that modify files such as ar). And of course the given command for a build rule may also potentially change. For example, imagine a build system that support a --ndebug argument, which causes compile command to have an extra -DNDEBUG argument.

So the aim of this article is to explore the design space of how build tools handle the specification of the inputs to a given build rule.

Explicit

Now the easiest approach is that the build system explicitly lists the inputs for each build rule. This is the base line kind of approach for something like make. The difficulty with this approach is that it can be error prone. A prototypical extract from a Makefile might look something like:


foo.o: foo.c
     gcc foo.c -o foo.o

Now if foo.c includes foo.h then there is a problem. Since foo.h is not captured as one of the inputs to the build rule, if foo.h changes, then the build command will not be re-run.

Of course, it is quite simple to include foo.h as one of the inputs for this specific build rule, but that is pretty brittle. C already sucks enough having to both declare and define public functions, without making it even more annoying by requiring you to update the build system every time you add a header to a source file. (And of course it should be removed from the build system when it is removed from the C file, however forgetting to do this will just affect performance, not correctness of the build system).

Another possibility is to treat each of the include path directories as an input to the build rule. There are some other issues with determining if a directory has changed from one build to the next, but we’ll ignore that for now. This approach should be relatively easy to use, and should be correct most of the time, but has a performance drawback if the include path has many include files, and most source files only include one or two headers.

Rule Specific Scanners

Rather than having to manually define each and every input file another approach is to have some rule specific process that can determine the correct inputs for a given rule. gcc has a -MM option which can be used to determine which header files would be used to compile a given file. This can be used in conjuction with make to automatically determine the inputs for any compile rules, which is a great improvement over manually managing the dependencies for any given source file, however there are some drawbacks.

The first is that the overall compile time is affected, as each time a file is compiled it must also generate the dependencies. In practise this isn't too bad; scanning for headers isn’t particularly CPU intensive, and since the compile will touch the same files doesn’t result in any extra I/O (and if the files weren’t cached, it primes the cache for the compile anyway).

The second problem, is that gcc -M is a gcc specific thing that isn’t going to work with other compilers, or other tools more generally. Of course, it would be possible to write a generate dependencies script for each type of build rule, but this is potentially a lot of work, and can have accuracy problem as the actual build command may in fact work slightly differently to how dependency script works, which risks having incorrect builds.

The next problem is to do with generated header files. If a source file includes a generated header file, and that generated header file does not yet exist, then gcc -MM will result in an error. gcc provides a -MG option to help account for this problem, however it is far from perfect. It assumes that the include path of the generated header is the current working directory which may not actually be the case. Generated files are not necessarily a problem, depending on some other design decisions it is possible to ensure that this dependency scanning occurs at the same time as compilation, so missing includes would be an error.

Another way to avoid the generated header file problem if the scanning operation is aware of the rest of the build rules. For example, when searching for a specific include file, the scanning tool could check not just for specific files in the file system itself, but also check for known outputs from other build rules in the build system. This approach has the drawback that the accuracy of scanning might not be adequate. For example, the SCons build tool uses this approach but can generate incorrect set of inputs when include files are conditionally included, or when headers are included through a macro. E.g:


#define ARCH mips
#define ARCH_INC(x) 
#include ARCH_INC(foo.h)

Of course you can argue that such a construct is probably less than ideal, however any approach like this is going to be prone to the same class of errors.

Depending on the exact model for determining the execution order of build commands in the system (the subject of a later article) the time at which the scanning occurs can have a major impact on performance.

A final problem with this approach in practise is that it can actually miss some dependencies. Consider the a command such as gcc -Iinc1 -Iinc2 foo.c. If foo.c includes foo.h and foo.h resides in the inc2 directory then this approach will generally report that foo.c depends on inc2/foo.h. However, this misses an important bit of information; the command is dependent on the non-existence of foo.h in the inc1 directory. If foo.h in added to inc1 then the output of the command will be different but an incremental build would miss this and not cause a rebuild. In theory there should be no reason why such a tool can’t report that a build rule depends on the non-existence of files as well. And indeed why explicit noting of inputs can’t do the same thing.

Tracing command execution

The only other approach (that I know of) is to track what files the build command actually touches when it executes. There are a couple of ways in which this can be done, but all approaches conceptually track when the build command opens files. One example of this is the memoize.py build tool, which uses strace to trace which files are touched by a given build command.

This approach has the very large advantage of being pretty accurate and capturing all the files that a build command touches and of not needing any build rule specific logic to determine the input files.

This approach can also easily capture files that were attempted to be opened

There are of course a few disadvantages. Firstly there is no standard API for tracing, so this part of any build tool ends up being OS specific, which is not ideal. Also tracing can add some significant overhead to the execution of build commands. Benchmarks are required to see what the difference in performance is between simply tracing build commands and running some rule specific scanner / dependency generation.

A potential disadvantage of this approach is false dependencies. If a build command opens a file but doesn’t actually consider the contents of the file during the execution of the command it will still be marked as a dependency, although there would be no need to rebuild if that file changed. This is is not a correctness problem, but could cause excessive unnecessary rebuilds.

Probably the biggest disadvantage of this approach is that all the input files for a build rule are not known until after the build command has executed. This has some pretty significant impacts on the different approaches that can be used for choosing the build rules execution order.

Summary

There are different approaches that can used to determine the set of input files for a given build rule; each has pros and cons, there is no clear winner.

My preference is the automatic detection of inputs for and build rule using a tracing approach of some kind. It wins in terms of correctness and also ease-of-use. Performance is a little bit unknown, however it should approach the performance of using a secondary script for determining the inputs.

Disagree with my analysis? Can you suggest some alternatives? Please leave a comment.

Build System Requirements

Sun, 13 Mar 2011 14:46:30 +0000

On just about any project that is larger than hello world you are going to want some kind of tool that can automate all the steps required to build a program. In general we give the name build sytem to the various compenents that go in to generating the final program. Build systems are probably up there with revision control systems as one of the most important software engineering tools. In this first post of a series about build systems I'm going to try and put together some of the requirements that a project’s build system should have.

So what is a build system? Basically, something that takes in the project source and generates the project artifacts (generally programs, libraries, tarballs, etc).

Before getting too far into the discussion is useful to draw the distinction between a project’s build system and the underlying build tool (e.g: make, scons, etc). This post is primarily about the former, not the latter. Of course the choice of tool can make it easier or harder to achieve the goals of the project’s build system, but it is possible to build good or bad systems regardless of the underlying tool.

Completeness

The build system should make it easy to build all the project’s outputs right up to something in releasable format. See also The Joel Test #2. This is not always easy to achieve, especially if some of your build process includes tools that are not easily scriptable.

Ideally this should include all the artifacts that are related to the release, including user manuals, release notes, test reports and so forth. Enabling the build system to automate the generation of all these artifacts requires some pretty careful thinking about the entire software development process.

For example, release notes may be as simple as pulling all the commit messages from yoru revision control system. Of course, depending on your choice of revision control system this may more or less difficult. Ofcourse there are plently of reasons why simple pulling commit messages it not the best approach to release notes; a higher level feature oriented change list is often more appropriate.

Another area is how testing will work. Clearly such an approach pushes for automated testing procedures, but this is not always so straight forward, especially for embedded software where the final test requires some form of manual testing.

Another point of consideration is build variants. Although most developers will only be building and testing a certain configuration or target at once, it should be possible to build all the supported variants of the project in one go.

Correctness

The output of any invocation of the build sytem should be correct. This seems obvious, but can actually be very difficult to get right. The main place that this is relevant is when thinking about incremental builds. Incremental builds are a pretty common optimisation that attempts to avoid doing unnecessary work to regenerate the outputs when the inputs haven’t changed. In such case, the result of doing an incremental build should be the same as doing a full build.

The usual way this can go wrong is that the dependencies for an output command are incorrecty specified or determined; this can lead to a file changing (such as a header), and output file not being rebuilt.

Performance

The build system should be fast! Building the entire project itself should be fast, and incremental builds should also fast. As a programmer you really want to get the edit-compile-test cycle to be fast; it doesn’t need to be too long before you lose concentration and just quickly check twitter or hackernews.

Incremental builds are normally the main thing required for performance, but an oft overlooked aspect is partial build where only some subset of the outputs are required.

Easy to use

To be effective the build system should be easy to use for all the developers. Of course there are a lot of aspects of ease-of-use. Firstly, it should be simple to start and execute a build; this part is usually pretty easy to achieve. It should also be easy to understand what the different targets are, and build for a specific target; this can be slightly harder to get right.

Ideally it would be easy and straight-forward for developers to change the build, for exmaple adding or removing targets, changing the build line, and so forth. This is the bit that many build systems fail at. It’s often very difficult to work out where a particular target is defined, and where the various different flags are defined; especially when flags are often the accumulation of options set in various different places.

Summary

So, this has been a pretty high-level overview of the requirements of a build system. In future posts, I'm going to look at the design of various build tools, and how the design of these tools can help achieve the goals outlined in this post.

If you have other requirements on a build system, it would be great if you could leave a comment or send me an e-mail.

Musings on C declaration syntax and style

Thu, 10 Mar 2011 21:27:08 +0000

So, today I was asked why I declared my pointer variables like:


int *f;

rather than:


int* f;

After all, I want a variable with the type pointer to int; the pointer is part of the type, so the asterisk rightly belongs with the int part. Which, on the face of it is a pretty reasonable argument. And to be honest, may be valid in the overall scheme of things.

The reason for my preference really stems from the syntax of declarations in C. The mental model that many of us have is that the syntax is:


type-name variable-name

If this was the reality of the situation then int* foo would be a pretty reasonable way to declare a variable. Unfortunately the reality of the situation is that syntax for declarations in C is actually:


delcaration-specifiers init-declarator-list_opt

At this point even an experienced C developers could be forgiven for thinking to herself, wtf is a declaration-specifier, and for that matter an init-declarator??.

Well, the full answers to such questions are found in everyone’s favourite piece of literature ISO/IEC 9899:1999 (of course to get the actual standard costs money, so most of us make do with WG14 N1256 [pdf], which is the final draft of the standard and as far as I’m aware there are no significant changes between the draft and the published standard) but I’m going to try and give a less precise, and hopefully more readable overview of what it means.

Declaration Specifiers

The declaration-specifiers consists of the storage-class specifier, the type-specifier and the type-qualifier. So, basically your storage class specifier is the extern or static. You can only have one of these in your declaration-specifier and should go at the front of your declaration.

Next up is the type-specifier, this is your void, char int, short, long, signed, unsigned, float, double. It can also be a struct, union or enum specifier, or a typedef name if you are feeling particularly out there.

Now, you’ve got to have at least one type-specifier in your declaration, but you can have more than one, such as unsigned int, for example. Interestingly, the order doesn’t matter, so int unsigned is the same as unsigned int, and the other kind of crazy thing is that type-specifiers can be mixed freely with the other specifiers, so int volatile long static unsigned is also perfectly valid!

Finally, you can optionally have a type-qualifer (or two), which are the volatile and const. These can also appear anywhere in your declaration-specifier and as a bonus party trick, if you have more than one of them that is fine. So int volatile volatile volatile is perfectly fine if you want to treat your C code as some form of absurdist poetry. (In C99 at least, not true in C89).

OK, so now you have a pretty good idea of what goes in to the magic declaration-specifier thing is. Now the important thing here is that pointer is not mentioned at all! And neither for that matter are arrays or functions. The pointer and array (and function) part of an identifier’s type don’t go in the declaration-specifier. Which just leaves us with the init-declarator-list thing; which is simply a list of declarators, which may be initialised. For this article we’ll not really worry about the initialisation part.

Declarators

So, a declarator contains the identifier (informally, the variable name), and additionally extra bits of type information, specifically whether the identifier is a pointer, array or function. Some example declarators:


x
*x
*const x
*const*const x
x[5]
x(int)
x(int x)

Now, for every declaration-specifier we can have a list of declarator, so the standard reason for putting a space between the type-specifier and the pointer is when declaring multiple pointers in the same declaration, there is less chance of getting things wrong. For example


int* x, y;

It is not clear whether the author intended for x to be a pointer to int, and y to be an int, or whether the intent was for both x and y to be pointers to int. Assuming the latter the right way to do it with such a formatting style would be:


int* x,* y;

which, is somewhat aesthetically unpleasing. By comparison:


int *x, *y;

is clearly two pointers, and


int *x, y;

is clearly one pointer without any real ambiguity. (Of course one could argue that declaring multiple identifiers with different types in the same declaration is probably not a crash hot idea anyway).

As an aside, while I’d never suggest doing this in real code, it is perfectly legal to declare a variable, pointer, array, function pointer, and function identifier within the same declaration, for example:


int *d1, d2, d3[1], (*d4)(void), d5(void);

Conclusions

So the main point here is that we shouldn’t think of declarations as type-name variable-name because that just isn’t how the language’s syntax works. Of course, there are other places where we do need to specify a full type, and that is when using the cast operator, however in the C specification a type-name is defined as “a declaration ... of that type that omits the identifier”, which is I format casts as (int *)foo rather than (int*)foo.

So, back to the topic at hand, int* foo vs int *foo. I don’t think there is any real defense for the first approach if your coding standard allows multiple identifiers to be declared within the same declaration.

I can see an argument being made that the C declaration rules are just too damn complex, and lets just pretend that declarations really are of the form type-name variable-name.

Of course one problem with that approach is that is is not possible to give identifiers complex types without the use of typedef. I guess this could be seen as a feature rather tahn a drawback.

Another argument for the space between type-specifier and pointer is that the pointer may include a type-qualifier, such as const. Compare int*const vs. int *const. In my opinion the latter is more aesthetically pleasing, but this is much weaker argument (and also motivates having a space between pointer and identifier *const ident vs *constident... the latter is not even syntactically correct).

So, a conclusion... I’m still going with int *foo in any of my coding standards because it most closely matches the underlying syntax of the langauge.

Do you have an opinion one way or the other? Do you have some good reasons to back it up? Please leave a comment!

Footnotes

If you want to know how to read C declarations I’d suggest understanding The Clockwise/Spiral Rule. Or if you are lazy try cdecl.

So, really, the C declaration syntax is kind of nuts, I much prefer what is done in Go. Go’s Declaration Syntax is worth a read.

Python getCaller

Thu, 06 Jan 2011 12:55:06 +0000

I’ve been doing a bit of Python programming of late, and thought I’d share a simple trick that I’ve found quite useful. When working with a large code-base it can sometimes be quite difficult to understand the system’s call-flow, which can make life trickier than necessary when refactoring or debugging.

A handy tool for this situation is to print out where a certain function is called from, Python makes this quite simple to do. Python’s inspect module is very powerful way of determining the current state of the Python interpreter. The stack function provides a mechanism to view the stack.

import inspect

print inspect.stack()

inspect.stack gives you a list of frame records. A frame record is a 6-tuple that, among other things, contains the filename and line number of the caller location. So, in your code you can do something like:

import inspect
_, filename, linenumber, _, _, _ = inspect.stack()[1]
print "Called from: %s:%d" % (filename, linenumber)

The list-index used is 1, which refers to the caller’s frame records. Index 0 returns the current function’s frame record.

Now, while this is just a simple little bit of code, it is nice to package it into something more reusable, so we can create a function:

def getCallsite():
    """Return a string representing where the function was called from in the form 'filename:linenumber'"""
    _, filename, linenumber, _, _, _ = inspect.stack()[2]
    return "%s:%d" % (filename, linenumber)

The tricky thing here is to realise that it is necessary to use list index 2 rather than 1.

The ability to inspect the stack provides the opportunity to do some truly awful things (like making the return value dependent on the caller), but that doesn’t mean it can’t be used for good as well.