So I recently stumbled across Jester, which is a mutation testing tool.Mutation testing is really a way of testing your tests. The basic idea is:
So in testing your code you should first aim for complete coverage, that is ensure that your test cases exercise each line of code. Once you get here, then mutation testing can help you find other holes in your test cases.
So the interesting thing here is what mutation operations (or mutagens), you should do on the original code. Jester provides only some very simple mutations, (at least according to this paper):
Unfortunately it seems that such an approach is, well, way too simplistic, at least according Jeff Offutt, who has published papers in the area. Basically, any rudimentary testing finds these changes, and doesn't really find the holes in your test suite.
A paper on Mothra, a mutation system for FORTRAN, describes a large number of different mutation operations. (As a coincident, my friend has a bug tracking system called Mothra.)
Anyway, so while Jester has a python port, I didn't quite like its approach (especially since I needed to install Java, which I didn't really feel like doing). So I decided to explore how this could be done in Python.
So the first set of mutation I wanted to play with is changing some constants
to different values. E.g: change 3 to (1 or 2). So this turns out to be reasonably
easy to do in python. I took functions as the unit of test I wanted to play with.
So, Python makes it easy to introspect function. You can get a list of
conants on a function like so: function.func_code.co_consts. Now,
you can't actually modify this, but what you can do is make a new
copy of the method with a different set of constants. This is conveniant because
in mutation testing we want to create mutants. So:
def f_new_consts(f, newconsts):
"""Return a copy of function f, but change its constants to newconsts."""
co = f.func_code
codeobj = type(co)(co.co_argcount, co.co_nlocals, co.co_stacksize,
co.co_flags, co.co_code, tuple(newconsts), co.co_names,
co.co_varnames, co.co_filename, co.co_name,
co.co_firstlineno, co.co_lnotab, co.co_freevars,
co.co_cellvars)
new_function = type(f)(codeobj, f.func_globals, f.func_name, f.func_defaults,
f.func_closure)
return new_function
So that is the basic mechanism I'm using for creating mutant function with
different constants. The other mutants I wanted to make where those where
I change comparison operators. E.g: changing < to <= or >. This is a bit trickier,
it means getting down and dirty with the python byte code, or at least this is my
preferred approach. I guess you could also do syntactic changes and recompile.
Anyway, you can get a string representation of a function's bytecode like so:
function.func_code.co_code. To do something useful with this you
really want convert it to a list of integers, which is easy as so:
[ord(x) for x in function.func_code.co_code].
So far, so good, what you need to be able to do next is iterate through the different bytecode, this is a little tricky as they can be either 1 or 3 bytes long. A loop like so:
from opcode import opname, HAVE_ARGUMENT
i = 0
opcodes = [ord(x) for x in co.co_code]
while i < len(opcodes):
print opname[opcode]
i += 1
if opcode >= HAVE_ARGUMENT:
i+= 2
will iterate through the opcodes and printing each one. Basically all bytecodes larger than the constant HAVE_ARGUMENT, are three bytes long.
From here it is possible to find all the COMPARE_OP
byte codes. The first argument byte specifies the type of compare operations,
these can be decoded using the opcode.cmp_op tuple.
So, at this point I have some basic proof-of-concept code for creating some very simple mutants. The next step is to try and integrate this with my unit testing and coverage testing framework, and then my code will never have bugs in it again!
Damn, conkeror is the most awesome thing ever. It basically gives you an emacs style interface to firefox. This is not just emacs key bindings for text boxes and stuff but a full on emacs style interface, with a status bar and entry box at the bootom, and none of this toolbar and menu stuff crufty up the page.
Instead of tabs you just have multiple buffers, which you can move between with the normal emacs key combinations. Huzzah!
I guess it isn't too suprising that this masterpiece comes from Shawn Betts, who also gave us ratpoison.
Of course it ain't perfect yet. It would be amazingly cool if it supported
C-x 2 (split windows) and C-x B (list buffers). Also
if there was a way I could still use the developer toolbar that would be good too.
The ssh ProxyCommand option is just really insanely useful. The reason I want to use it is that it makes it easy to tunnel ssh through a firewall. So for example you have a machine on your corporate network that is is sitting behind a firewall, but you have a login to the firewall. Now of course you could just do:
laptop$ ssh gateway gateway$ ssh internal internal$
or something like:
laptop$ ssh -t gateway ssh internal internal$
But that really doesn't work very well if you use scp or sftp or some other service like revision control that runs on top of it. This is where you can use ProxyCommand option to make your life easier. Basically you want to put something like this into you .ssh/config:
Host internal
ProxyCommand ssh gw nc -w 1 internal 22
For this to work you will need netcat (nc) installed on the gw, but that generally isn't too much of a problem. Now you can transparently ssh, scp or whatever to internal, and ssh deals with it.
Now of course if you can't get into the network full stop, you need some reverse tunneling tricks and some other host on the interweb which you can use to tunnel through. So something like:
ssh -f -nNT -R 1100:localhost:22 somehost
Will setup a remote tunnel, which basically means if you connect to
port 1100, it will be forwarded to port 22 on the machine on which you
ran ssh -R. Unfortunately unless you have full control
over the host you are creating the reverse tunnel too, you will find
that port 1100, will only be bound to localhost, which means you will
probably still need to use the trick mentioned above to get seemless
access to it.
I've always been a fan of using -Wall -Werror
when programming C and also tools such as flint and splint.
For some reason, despite coding Python since back in the 1.5
days, I had never really used a similar tool for Python.
This is where pylint comes in. This is a fantastic tool that helps find code that is likely to fail, and helps you adhere to coding standards.
pylint divides the things it checks into:
The project webpage has a full list of checks that pylint performs.
One of the really neat things I found from pylint is that it will test code paths that you test cases might not cover, (which isn't to say your test cases should have complete coverage), this is great when you have a dynamic language. It still won't give you the kind of guarentees you get from type checking, but it goes a long way to finding most stupid bugs.
I've found that the things that it has picked up for me has mostly been convention and warnings about missing docstrings, which is great. I think these are important things, but I often forget to follow them and am definately too lazy to manually check my code for this kind of thing. Pylint provides a check crutch to lean on, which in the end produces better, more readable code, with less effort on my end, which is what any good tool should do. The other great thing is that it is configurable so you can enable to tests you think are important, and tweak the parameters of other tests. I definately recommend using it to any who uses python on a regular (or irregular) basis.
I'm sometimes suprised of the things I don't know. For
example, after using Linux as my primary machine I only just
found out that less -R will make less correctly
markup ANSI escape sequences, so when I pipe
ls --color or pylint
through it I get something readable, rather than raw escape codes.
When you write scripts that run as cron jobs, and send email to people, and have the potential to send a *lot* of email to people you really don't want to screw up.
Unfortunately I did screw up when writing one of these. It was a pretty simple 200 lines or so python script that would find any new revisions that had been commited since the last time it ran, and email commit messages to developers.
The idea was simple, a file kept a list of the last seen revisions, I would go through the archive find new revisions, mail them out, and then finally write out the file with the latest files.
Spot the bug, or at least the design error? When our server ran out of disk space, the stage of writing out the the file with the last seen revisions failed, and created an empty file. So next time the script ran it thought all the revisions were new, resulting in thousands of email about revisions committed years ago. I pity our poor sysadmin who not only had to deal with out of disk problems but now also with a mail queue with thousands of messages.
Solution to the problem of course is try and write out the new revision file before sending the email, and write it to a temporary file, instead of blasting the last known good out of existance by writing directly over the top of it.
I guess the moral is designing these little scripts actually requires more care than I usually give them.
I do a lot of hacking with binary files at work, and simply doing a diff to check if two files are the same isn't really useful, I usually want to preprocess the file with some command to make it into useful plain text. This ends up being quite tedious by hand. I came up with this simple script, diffcmd, but I'm sure the lazyweb will let me know of anything better if it exists.
Usage: % diffcmd "readelf -a" file1 file2
#!/bin/sh CMD=$1 FILE1=$2 FILE2=$3 TMP1=`tempfile` TMP2=`tempfile` $CMD $FILE1 > $TMP1 $CMD $FILE2 > $TMP2 diff -u $TMP1 $TMP2 rm $TMP1 $TMP2
Huzzah, we now have actual street maps with google. Recently the Dept. of Lands changed their interface, which meant a couple of hours updating my Geocoder. You can now lookup addresses and get maps at http://www.benno.id.au/map and can download my python script for querying the Lands database.
Fun in the lab this week, we got an Intel Mac Mini last Friday, and today Chuck got L4 up and booting on the Mac Mini.
This is a bit different to just getting a standard
OS up and running. For a start we a booting new development
kernels all the time, so its nice to be able to transfer an image
across without having to move media around. Also you do a lot of
printf debugging so having a decent terminal is
important. Usually for developing L4 we boot over the network and
use the serial console for terminal debugging. Unfortunately the
Mac Mini doesn't have a serial port, and getting a toolchain to compile
EFI modules that enable network booting was somewhat problematic.
Enter Firewire. This lets us just use one cable to connect to the workstation which we can use for downloading and serial deubgging. Awesome! So Chuck boots ELILO from EFI, which then loads a simple Firewire driver. From the desktop side, a simple app is used to copy the image across and act as a console. Rock on!
With any luck we will have a working Darbat up and running on the Mac Mini by the end of next week.
So watching the weather is interesting these days. Instead of making a projection they now pass the buck to computers. In the Ch 9 news they described it as "computer simulations predict tommorow will be a cloudy day". Seriously do they really think it was the weather presenter comes up with the prediction themselves?