Mail on UNIX is weird, I spent a few hours this week tracking down some bugs in my mail setup, and in the process learnt a lot more about how things interact. I'm documenting it here for Benno 5 months from now and anyone else that cares. There is this pseudo-standard of mail user agents (MUAs) not actually talking SMTP to a mail submission agent (MSA), but instead simply hoping that /usr/sbin/sendmail exists. Things are so bound this way that you really can't avoid having a /usr/sbin/sendmail on your machine, so as a result other mail transport agents (MTAs) have to provide a binary that works in the same way as sendmail. Unfortunately as far as I can tell there is no standard about what command line flags a sendmail compatible program should accept, and appears to come down to common usage.

In some ways this is quite backwards to what I would have expected, which is that an MTA would run on the localhost and programs would talk SMTP to port 25 (or the MSA port 587), then all communications is through this documented standard. On the other hand, this means every program that wants to send e-mail must have TCP and SMTP libraries compiled which is against the UNIX philosophy.

Now I am actually quite interested to find out what other programs (that I use), actually rely on sendmail. I'm guessing (hoping) that it is mostly just MUAs such as mutt and mailx, but I really wonder what else is out there relying on sendmail.

More importantly, what command lines arguments do these different programs expect /usr/sbin/sendmail to handle correctly. (In case I wanted to join the hundreds of others and say, write my own sendmail replacement). So after putting in a simple python program to log the command line arguments my great findings are: mailx uses the -i argument. This one is great, by default a line with a single '.' character is treated as the end of input, with the argument standard end of file, means end of file. mutt on the other hand uses -oi (which is the same as -i, and -oem. -oem is conveniently undocumented in Postifx's sendmail man page, but on consulting the exim sendmail man page, discovered that this basically means, the command can't fail, and any errors should be reported through email, rather than with an error return code.

mutt lets you override the command it uses for mail submission by setting the sendmail variable. This is handy to know if you want to add extra arguments to the sendmail command line. For example a handy feature is being able to control the envelope from address used. This is done with the -f command line argument.

Next up for my adventures in the wonderful world of UNIX email is to setup my on sendmail compatible program, that can set the envelope address and smarthost to use based on the e-mail From: header.

Want to help run the best open source conference in world? Want to get involved with the local open source community? Then linux.conf.au wants you!

We are currently after volunteers interested in helping run linux.conf.au 2007.

What types of things will you be helping out on?

• Conference help desk.
• Packing conference bags.
• Helping direct people to and from events.
• Running things form one end of campus to the other.
• Looking after theatres.
• Driving things (and people) from one side of Sydney to the other.

What's in it for you?

• Free entry to the conference dinner.
• Reimbursement of ticket price (up to hobbyist value). Note that reimbursement will be done on a sliding scale, you will be reimbursed 20% of ticket price, per day helping out.
• A different coloured t-shirt!!

Here's what you do

1. Register as hobbyist or student on the linux.conf.au website.
2. Email seven-contact@lca2007.linux.org.au to register your interest in volunteering, including a your registration name, description of what you are interested in doing, which days you can help out on, and a bit about yourself. (Including where you will be staying during the conference, and if you have a car.)

If you have any questions please email seven-contact@lca2007.linux.org.au

Today I was going through some old RSS feeds and ended up splurging on Amazon. The first book on the list is Beyond Java which was recommended by Joel. I was intrigued by a comment, specifically, "its starting to look like type declarations are one of those accidental difficulties that good programming languages can eliminate". Which I don't really agree with, but hey, I'll comment once I've read the book.

The second is The Hidden Pattern: A Patternist Philosophy of Mind , which I was pointed to indirectly via BoingBoing, which discusses philosphy, cognition and AI.

And going with the brain topic, Classic Case Studies in Psychology, looks like an enjoyable, and probably slightly easier read than The Hidden Pattern.

I also picked up some sci-fi, Broken Angels, as recommended by the AI that is Amazon.

I'm currently try to remerge two versions of a source file that have diverged quite a long way. This is quite a painful process, and a tool like mgdiff makes it a lot easier to see diffs than just the normal command line diff.

Unfortunately, mgdiff has one essential missing feature; the ability to update the diff as you are editing the underlying files. So with a bit of help from Erik, a cooked up this patch which adds the feature. Hopefully this will hit the upstream package soon.

So for some reason I didn't realise how cool Dinosaur Jr was until about now (hey only 20 years behind but what the hey). Anyway, Feel the Pain, has a really catchy guitar in it, which I thought I might like to learn how to play. (Not that I can play guitar mind you, Guitar Hero doesn't really count). So I went searching for some tabs (not that I know how to read guitar tabs, but I've heard they are useful for learning how to play guitar), and anyways, stumbled across Google music search. Back to work.

Just like Jdub, I'm stuck in the USA. Thanks united!

My flight from San Diego to Chicago ended up being delayed by two hours, which meant I missed my connecting flight to Zurich. Joy! So I'm stuck in Chicago over night. Not as bad as last time when my flight from SFO to SYD was turned around midflight, or as bad as getting snowed in, in Denver.

Oh well, maybe I'll check out Chicago tommorow morning.

You've really got to watch out for the BLOGOSFEAR

Reading Jeff's humorous post got me digging into exactly why Python would be running in a select loop when otherwise idle.

It basically comes down to some code in the module that wraps GNU readline. The code is:

		while (!has_input)
		{	struct timeval timeout = {0, 100000}; /* 0.1 seconds */
			FD_SET(fileno(rl_instream), &selectset);
			/* select resets selectset if no input was available */
			has_input = select(fileno(rl_instream) + 1, &selectset,
					   NULL, NULL, &timeout);
			if(PyOS_InputHook) PyOS_InputHook();
		}

So what this is basically doing, is trying to read data but 10 times a second calling out to PyOS_InputHook(), which is a hook that can be used by C extension (in particular Tk) to process something when python is otherwise idle.

Now the slightly silly thing is that it will wake up every 100 ms, even if PyOS_InputHook is not actually set. So a slight change:

		while (!has_input)
		{	struct timeval timeout = {0, 100000}; /* 0.1 seconds */
			FD_SET(fileno(rl_instream), &selectset);
			/* select resets selectset if no input was available */
			if (PyOS_InputHook) {
				has_input = select(fileno(rl_instream) + 1, &selectset,
						   NULL, NULL, &timeout);
			} else {
				has_input = select(fileno(rl_instream) + 1, &selectset,
						   NULL, NULL, NULL);
			}
			if(PyOS_InputHook) PyOS_InputHook();
		}

With this change Python is definitely ready for the enterprise!

One of the problems with pyannodex was that you could only iterate through a list of clips once. That is in something like:

anx = annodex.Reader(sys.argv[1])

for clip in anx.clips:
    print clip.start

for clip in anx.clips:
    print clip.start

Only the first loop would print anything. This is basically because clips returned an iterator, and once the iterator had run through once, it didn't reset the iterator. I had originally (in 0.7.3.1) solved this in a really stupid way, whereby I used class properties to recreate an iterator object each time a clip object was returned. This was obviously sily and I fixed it properly by reseting the file handle in the __iter__ method of my iterator object.

When reviewing this code I also found a nasty bug. The way my iterator worked relied on each read call causing it most one callback, which wasn't actually what was happening. Luckily this is also fixable by having callback functions return ANX_STOP_OK, rather than ANX_CONTINUE. Any way, there is now a new version up which fixes these problems.

Jamie asked me about using splint in his SCons based projects such as filtergen. So the basic way to use SCons and splint together is through the power of pre-actions. Basically this allows you to specify a command to be run before a particular build rule is used. So I added something like this:

for object_file in filtergen_objects:
    env.AddPreAction(object_file, env["SPLINTCOM"])

to the filtergen main SConstruct file. The worst part about this is you now need to explicitly have a list of source objects around to pass to your Program builder. This is a bit annoying but not the end of the world. the SPLINTCOM command is setup using hte SCons command interpolation like so:

env["SPLINT"] = "splint"
env["SPLINTFLAGS"] = ["-weak", "+posixlib", "-I."]
env["SPLINTCOM"] = Action("$SPLINT $CPPFLAGS $SPLINTFLAGS $SOURCES")

Currently I'm splint-ing filtergen at the weak level, and it passes, but to really use splint Jamie will need to start using splint at the standard level and fix some stuff up.

When using splint, instead of parsing the standard system headers it uses its own which are firstly splint-clean themselves, and also add annotations to describe memory usage. Unfortunately these headers don't define all the standard functions found in headers, in particular network related ones, so you end up with some ugly code in the source files to deal with this. If I have time I'll try and update the standard set of headers to be more full featured.

I also found that filtergen doesn't really compile very well on non-GNU platforms, but that I'm not about to try and fix that in a clean way. For Jamie's future reference problems involve:

1. System bison sucks, and can't handle the grammar files. Installing bison from ports fixes this, expect you need to extend SConstruct to grab $PATH from the environment.
2. Not everything has strndup.

For those that care, which is mostly Jamie, the patches can be merged from my bzr branch: http://benno.id.au/bzr/filtergen.splint