This is a rant; if you don't like rants, well just don't read it.!

Computer science is full of fuzzy terms, and cases where multiple concepts are mapped to the same term (shall we have the argument of whether an operating system is just the thing running in privileged mode or also includes any of the libraries? and whether we should call that thing Linux or GNU/Linux?). Usually I can just deal with it, there generally isn't a right answer, or if there ever was it has been lost in the mists of time. And of course, language is a living thing, so I get that the meaning of words change over time. But, never have I seen a term's meaning been so completely eviscerated as what has happened to the previously useful and meaningful term Inversion of Control, which has become completely conflated with the term Dependency Injection.

Wikitionary provides a relatively servicable definition of the term:

Abstract principle describing an aspect of some software architecture designs in which the flow of control of a system is inverted in comparison to the traditional architecture.

A trivial example would be reading some data from the console. The usual flow of control would be something like:

function read_input()
{
   input = /* some magic to get the input */
   return input
}

function menu()
{
   loop
   {
       input = read_input()
       /* Do something with the input */
   }
}

function main()
{
    menu()
}

The normal flow of control is that the application passes control to the read_input() function, the function returns with some result and then the application does something with the input. Now we can invert this by doing something like:

function input_reader(handler)
{
    loop
    {
        input = /* some magic to get the input */
        handler(input)
    }
}

function menu_handler(input)
{
     /* Do something with the input */
}

function main()
{
    input_reader(menu_handler);
}

In this example rather than the application code calling the read_input() function, the input reading code calls the application. That right there is the inversion!

So breaking it down; the control is normally passed from an application to a library (via a function call). If we invert this, i.e.: put in the opposite order, then the library passes control to the application and we can observe an Inversion of Control.

Such a pattern is also called the Hollywood Principle: Don't call us; we'll call you. (Don't ever accuse computer scientists of lacking a sense of humour!).

Of course there is nothing mind-blowing about structuring code like this. Clark discussed structuring system using upcalls [PDF] almost 30 years ago.

So, inversion of control isn't really a big concept. Pretty simple, straightforward, and probably not deserving of an 18 character phrase, but it certainly means something, and that something is a useful concept to use when talking about system design, so needs some kind of label.

A real-word example can be found in the C standard library, where qsort calls in to the application to perform a comparison between elements. Another example is the Expat XML parser which calls back in to the application to handle the various XML elements. Glib's main loop is probably another good example. I think it is interesting to note that size or scope of an inversion of control can vary. In the case of qsort, the inversion is very localised, by comparison in the Glib example, basically the whole program exhibits inversion of control.

The one quibble I would have with the Wikitionary definition is using the use of the word principle. To me a principle implies some desirable design aspect that you generally strive towards. For example, the Principle of Least Privilege or the Separation of Concerns principle.

Inversion of Control is not really something you strive towards. In fact it can often lead to more complex designs. For example, when parsing XML it is generally simpler to call a function that converts the XML data into a document object model (DOM), and then make calls on the DOM, compared to an Expat or SAX style approach. Of course, there are advantages to the Expat/SAX approach such as reduced memory footprint. The point here is that Inversion of Control is a consequence of other design goals. No one starts out with the goal of: let's design this system so that it exhibits inversion of control (well, OK, maybe node.js did!). Rather it is a case of: so that we could achieve some specific design goal, we had to design this part of the system so that it had inversion of control.

If you didn't really like my description of Inversion of Control, I can recommend reading Martin Fowler's article, which I think does a pretty good job of explaining the concept.

However, the final paragraph of the article starts to point to the focus of this rant:

There is some confusion these days over the meaning of inversion of control due to the rise of IoC containers; some people confuse the general principle here with the specific styles of inversion of control (such as dependency injection) that these containers use. The name is somewhat confusing (and ironic) since IoC containers are generally regarded as a competitor to EJB, yet EJB uses inversion of control just as much (if not more).

This starts to introduce new terms such as IoC containers (see, I told you 18 characters was too much, look how swiftly it has been reduced to a mere 3 characters!) and dependency injection. When I first read this, I had no idea what Martin was talking about. What is an IoC container? What is dependency injection? (I'm ashamed to say I did know that EJB was an Enterprise JavaBean.) And my curiosity in to this final paragraph let me on a magical journey of discovery where the more I read the more I thought that I must have been confused about what IoC is (and I'm pretty sure if been using the term off and on for over 11 years now, so this was a bit of a concern for me!). So, if you are still reading you get to come along with me on a ride of pedantry and computer science history as I try to work out why everyone else is wrong, because I know I can't be wrong!

Let's start with Wikipedia. The first paragraph of the Inversion of Control article on Wikipedia defines IoC as:

a programming technique, expressed here in terms of object-oriented programming, in which object coupling is bound at run time by an assembler object and is typically not known at compile time using static analysis.

I'm really not sure what sort of mental gymnastics are required to get from my definition to a definition such as that. They are certainly beyond my capabilities. But it's not just Wikipedia.

The top answer on StackOverflow question What is Inversion of Control? isn't much better:

The Inversion of Control (IoC) and Dependency Injection (DI) patterns are all about removing dependencies from your code.

Sorry, I don't get it. Inversion of Control is all about, you know inverting control. At least this explicitly conflates IoC with this dependency injection thing, which gives us some hope of tracing through time why these things seems so intertwined.

And just in case you thought it was just Wikipedia and Stackoverflow, you can pick just about any blog post on the subject (excepting Martin's previously linked above, and I guess this one, but that is a bit self-referential), and you'll find similar description. Just to pick a random blog post on the subject:

Inversion of Control (IoC) is an approach in software development that favors removing sealed dependencies between classes in order to make code more simple and flexible. We push control of dependency creation outside of the class, while making this dependency explicit.

Again this recurring theme of dependencies. At least all these wrong definitions are more or less consistently wrong. Somewhere in the history of software development dependencies and inversion got all mixed up in the heads of developers. It is interesting to note that most of the examples and posts on the subject are very object-oriented (mostly Java).

To get to the bottom of this I guess I need to learn me some of this dependency injection terminology (and learn how to spell dependency without a spell-checker... hint, there is no a!). Given the quality of the IoC article, I'm skipping the Wikipedia article. The canonical source (by proclamation of the Google mediated hive-mind) is Martin Fowler's article. His article on IoC was pretty good (apart from a confusing last paragraph), so this looks like a great place to start.

In the Java community there's been a rush of lightweight containers that help to assemble components from different projects into a cohesive application. Underlying these containers is a common pattern to how they perform the wiring, a concept they refer under the very generic name of "Inversion of Control". In this article I dig into how this pattern works, under the more specific name of "Dependency Injection"

Aha! I knew it was Java's fault! OK, so, about 10 years ago (2003), the Java community discovered some new pattern, and were calling it Inversion of Control, and Martin's article is all about this pattern, but he calls it by a different name Dependency Injection (DI). Well, I guess that explains the conflation of the two terms at some level. But I really want to understand why the Java community used the term Inversion of Control in the first place, when, as far as I can tell, it had a relatively clear meaning pre-2003.

So, these lightweight containers are mentioned as originators of this pattern, so they look like a good place to start investigating. One of those mentioned in Martin's article is PicoContainer. Unfortunately that goes to a deadlink. Thankfully, through to the awesomeness that is the Wayback Machine we can have a look at the PicoContainer website circa 2003. That website mentioned that it honors the Inversion of control pattern (IoC) but doesn't really provide any detail on what it considers the IoC pattern to be. Thankfully, it has a history section that attempts to shed some light on the subject: Apache's Avalon project has been selling the IoC pattern for many years now.

OK, great, so as of ~2003 this Avalon project had already been talking about the IoC pattern for many years. The Apache Avalon project has (unfortunately?) closed, but there seems to still be references and docs in various places. On bit of the project's documentation is a guide to inversion of control.

It introduces IoC as: One of the key design principles behind Avalon is the principle of Inversion of Control. Inversion of Control is a concept promoted by one of the founders of the Avalon project, Stefano Mazzocchi.

It goes on to provide a very different description of IoC to my understanding:

Chain of Command This is probably the clearest parallel to Inversion of Control. The military provides each new recruit with the basic things he needs to operate at his rank, and issues commands that recruit must obey. The same principle applies in code. Each component is given the provisions it needs to operate by the instantiating entity (i.e. Commanding Officer in this analogy). The instantiating entity then acts on that component how it needs to act.

The concrete example provided is:

class MyComponent
    implements LogEnabled
{
    Logger logger;

    public enableLogging(Logger newLogger)
    {
        this.logger = newLogger;
    }

    myMethod()
    {
        logger.info("Hello World!");
    }
}

With an explanation:

The parent of MyComponent instantiates MyComponent, sets the Logger, and calls myMethod. The component is not autonomous, and is given a logger that has been configured by the parent. The MyComponent class has no state apart from the parent, and has no way of obtaining a reference to the logger implementation without the parent giving it the implementation it needs.

OK, I guess with such a description I can start to see how this could end up being called inversion of control. The normal order of things is that a class creates its dependents in its constructor, however this pattern changes this so that the caller provides the dependents. This doesn't really feel like inversion to me, but I guess it could be considered this. And equally I don't really think there is control as such involved here, maybe control of dependencies?

I think you could claim that myMethod in the example exhibits some localised inversion of control when it calls the logger, but that isn't really what is identified by the any of the explanatory text.

Anyway, this isn't really a particularly pleasing place to stop; there must be more to this story.

Some extensive research work (OK, I just used google), led me to the aforementioned Stefano Mazzocchi's blog. On which he has an insightful post about the origins of the use of the term IoC within the Avalon community.

I introduced the concept of IoC in the Avalon community in 1998 and this later influenced all the other IoC-oriented containers. Some believed that I was the one to come up with the concept but this is not true, the first reference I ever found was on Michael Mattson’s thesis on Object Oriented Frameworks: a survey on methodological issues.

At last! The origin of the term, or at least the point at which is became popular. I'll quote the same paragraph from the thesis that Stefano did:

The major difference between an object-oriented framework and a class library is that the framework calls the application code. Normally the application code calls the class library. This inversion of control is sometimes named the Hollywood principle, “Do not call us, we call You”.

Which brings things back full-circle to the start, because this certainly matches my understanding of things. What I fail to understand is how you get from this quote to the military chain of command point of view.

Stefano continues (emphasis added):

Now it seems that IoC is receiving attention from the design pattern intelligentia (sic): Martin Fowler renames it Dependency Injection, and, in my opinion, misses the point: IoC is about enforcing isolation, not about injecting dependencies.

I think this is really the crux of the issue. I think there is a massive leap being made to get to any claim that IoC is about enforcing isolation. I don't think the previous uses of the term imply this at all. Certainly not the quoted thesis.

Doing some more digging, there may be some justification from John Vlissides column Protection, Part I: The Hollywood Principle. The column describes a C++ object-oriented design for a filesystem that has some protection and enforcement. (As a side note the idea of providing any level of protection enforcement through language mechanisms is abhorrent, but lets just accept that this is a desirable thing for the moment.) In any case the pertinent quote for me is:

The Template Method pattern leads to an inversion of control known as the "Hollywood Principle," or, "Don't call us; we'll call you."

The inversion of control is a consequence of using the Template Method pattern, which may themselves have something to say about enforcing some isolation, but IoC itself is a consequence, not an aim.

So lets recap where we are right now. We have one view of IoC which is just a simple way of what describing happens when you flip things around so that a framework calls the application, rather than an application calling the library. This isn't good or bad, it is just a way of describing some part of the a software systems design.

Secondly we have the view promoted by originally it would seem promoted by Stefan, but subsequently repeated in many places, where IoC is a design principle, with an explicit goal of enforcing isolation.

I find it hard to want to use IoC as a design principle. We already have a well established term for a design principle that is all about isolation: separation of concern. Which goes back to Dijkstra a good forty or so years ago.

Given this, I think IoC should only be used to describe an aspect of design where the flow of control is inverted when compared to the traditional architecture. If I can be so bold, this is the correct definition. Other definitions are wrong!

(And just to be clear, there is nothing wrong with dependency injection, it's a great way of structuring code to make it more reusable, just don't call it inversion of control!

If you main use of GDB is for debugging embedded devices you can't really go too long without encountering the GDB remote protocol. This is the protocol used to communicate between the GDB application and the debugger, usually over either TCP or serial.

Although this protocol is documented, it is not always clear exactly when which packets are actually used and when. Not knowing which packets to expect makes implementing the debugger side of things a little tricky. Thankfully there is a straight forward way to see what is going on:

$ set debug remote 1

If you are developing stuff on ARM Cortex M-series devices and need a reasonably priced debugger, I'd have to recommend the Black Magic Probe. For about 70 bucks you get a pretty fast debugger that directly understands the GDB remote protocol. This is kind of neat as you don't need to have run some local server (e.g.: stlink); the debugger appears as a standard CDC ACM USB serial device.

The thing about it which is pretty cool is that the debugger firmware is open source, so when you suspect a bug in the debugger (which, just like compiler bugs, does happen in the real world) you can actually go and see what is going on and fix it. It also means you can add features when they are missing.

Although the debugger has some support for ARM semihosting, unfortunately this support is not comprehensive, which means if you use an unsupported semihosting operation in your code you end up with a nasty SIGTRAP rather than the operation you were expecting.

Unfortunately one of the simplest operations, SYS_WRITEC, which simply outputs a single character was missing, which was disappointing since my code used it rather extensively for debug output! But one small commit later and the debug characters are flowing again. (As with many of these things, the two lines were the easy bit, the hardest and most time consuming bit was actually installing the necessary build pre-requisites!)

My Python EXIF parsing library is joining the slow drift of projects away from Google code to Github. If you are intertested in contributing please send pull requests and I'll attempt to merge things in a timely manner.

As you can probably tell from the commit log, I haven’t really been actively working on this code base for a while, so if anyone out there feels like maintaining, please just fork on github, let me know, and I’ll point people in your direction.

So, another fun fact about C. You'd think that a language would make it possible to express any value for its in-built types as a simple constant expression. Of course, you can't do that in C.

In C you can express both signed and unsigned integer of any of the integer types. For example 5 is a signed integer; 5L is a signed long, 5U is an unsigned integer. But, there is no way to expression a negative number as a constant! Despite what you might easily assume, -5 is not constant, rather it is the unary - operator applied to the constant 5. Now, generally this doesn't cause any problems. Unless of course, you care about types and care about having programs that are valid according to the standard.

So, assuming two's complement 32-bit integers, if you want an expression that has the type int and the minimum value (-2147483648), then you need to be careful. If you choose the straight-forward approach you'd write this as -2147483648, and you'd be wrong. Why? Well, the integer constant 2147483648 has the type long, since it can not be represented as an integer (the maximum value of a 32-bit integer is 2147483647). So, the type of an expression of the form - <long> is, of course long. So, what can we do instead? Well, there are lots of approaches, but probably the simplist is: -2147483647 - 1. In this case, 2147483647 can be represented as an integer, -2147483647, is still of type int, and then you can safely subtract 1.

Of course, this becomes even more interesting is you are dealing with the type long long. Assuming a two's complement, 64-bit type, then the minimum value is -9223372036854775808, but of course, you can't just write -9223372036854775808LL, since 9223372036854775808LL can't be represented by the type long long (and assuming the implementation doesn't define any extended integer types), this would actually be an invalid C program.

So, yesterday's article on with turned out the be a bit wrong. As lesson in reading the explanatory text and not just the example code. (Or possibly a lesson in providing better example code!).

So, I gave an example along the lines of:

@contextmanager
def chdir(path):
    # This code is wrong. Don't use it!
    cwd = os.getcwd()
    os.chdir(path)
    yield
    os.chdir(cwd)

Now, this appears to mostly work. The problem occurs when you have some code like:

with chdir('/'):
    raise Exception()

In this case the final cleanup code (os.chdir(cwd)) will not be executed, which is almost certainly not what you want. The correct was to write this is:

@contextmanager
def chdir(path):
    cwd = os.getcwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(cwd)

Writing it this way the final os.chdir(cwd) will be executed, and the exception will still be propagated. So I think this kind of sucks, because I'd really have preferred it to have this behaviour by default. To this end I've created a simplecontextmanager function decorator. Basically you can use it like so:

@simplecontextmanager
def chdir(path):
    cwd = os.getcwd()
    os.chdir(path)
    yield
    os.chdir(cwd)

Looks pretty much like the first example, however it will have the semantics of the second example. Now clearly the simplecontextmanager doesn't provide the whole range of flexibility that contextmanager does, but for simple code it is more straightforward.

My pyutil git repo has been updated to have working version of the chdir, umask and update_env context managers, and also contains the source for the new simplecontextmanager.

Moral of the story, test your code and read the docs carefully!

Update: This article has some errors, that are correct in a new article.

I'm only like 8 years late to the party on this one, but I've got to say Python's with statement is pretty awesome. Although I've used the functionality off-and-on I'd mostly avoided writing my own context managers as I thought it required classes with magic __enter__ and __exit__ methods. And besides, my other main language is C, so you know, I'm used to releasing resources when I need to.

Anyway, I finally decided that I should actually learn how to make a context manager, and it turns out to be super easy thanks to Python yield statement. So, some truly simple examples that will hopefully convince you to have a go at make your own context managers.

First up, a simple chdir context manager:

@contextmanager
def chdir(path):
    cwd = os.getcwd()
    os.chdir(path)
    yield
    os.chdir(cwd)

If you are doing a lot of shell-style programming in Python this comes in handy. For example:

with chdir('/path/to/dosomething/in'):
    os.system('doit')

A lot simpler than manually having to save and restore directories in your script. Along the same lines a similar approach for umask:

@contextmanager
def umask(new_mask):
    cur_mask = os.umask(new_mask)
    yield
    os.umask(cur_mask)

Again, this avoids having to do lots of saving and restoring yourself. And, the nice thing is that you can easily use these together:

with umask(0o002), chdir('/'):
     os.system('/your/command')

Code for these examples is available on my GitHub page.

I've only been programming C for about 15 years, so for the vast majority of that time I've been using C99. And most of that time has been spent writing low-level code (operating systems, drivers, embedded systems). In these kind of systems you care a lot about the exact sizes of types, since memory and cache are important as so is mapping variables to memory mapped registers. For this reason having a standard way of declaring variables of a certain bitsize struck me as a pretty smart thing to do, so I've almost always used stdint.h on my projects. And in the rare cases where it wasn't available used some close equivalent. However, there is an argument to be made against using them (an extension of the general argument against typedefs). The core of the argument against is that it hides information from the programmer that is vital in constructing correct programs.

Let's start with some simple C code:

unsigned int a;
unsigned int b;

The question is, what is the type of the expression a + b. In this case, things are fairly straight-forward. The results expression has the type unsigned int.

Slightly trickier:

unsigned short a;
unsigned short b;

In this case due to integer promotion the operands are promoted to int and therefore the resulting expression is also of an type int. Integer promotion is a little obscure, but relatively well known. Most of the time this isn't really a problem because if you assigned the expression back to a variable of type unsigned short the effect will be the same. Of course if you assign it to an unsigned int, then you are going to be in trouble!

So nothing really new there. There is some obscurity, but with local knowledge (i.e: just looking at that bit of code), you can tell with complete accuracy wha The problem occurs when you have something like:

uint16_t a;
uint16_t b;

So, now what is the type of the expression a + b? The simple answer is it depends. The only way to know the answer to this question is if you know what the underlying type of uint16_t is. And this is a problem, because the correctness of your program depends on being able to answer that question correctly. The unfortunate result of this is that in an attempt to make code more portable across different platforms (i.e.: by not relying on the exact sizes of types), you end up in a situation where the correctness of your program still depends on the underlying platform, although now in a nasty obscure manner. Win!

Of course, most of the time this doesn't matter, but in the times where it does matter it is certainly a problem. Unfortunately, I don't have a good suggestion on how to avoid this problem, other than careful coding (and I really hate having to rely on careful coding just to essentially get type safety correct). If you've got a good suggestion please let me know in the comments.

Of course, I think I will still continue to use stdint.h despite these problems, however it is certainly something that C programmers should be aware of when using these types.

I'm working on a project where I want to ensure that two .pyc files I have are identical. Essentially I want to verify that a .py byte-compiled by Python on one platform, matches the same .py byte-compiled on another platform. This is more challenging than you might think! (As an aside, pretty much all the challenges apply equally if you are trying to verify that a .py compiled at some specific time is indentical to a .py compiled at some later time.

To understand the first problem it is import to understand the format of a .pyc file. Basically, it is a header, followed by the module's code object serialised using the marshal library. In the first instance we are interested in just the header. It has three fields:

Magic	4 bytes
Timestamp	4 bytes
Source file size	4 bytes

These are basically verified during load to ensure that a stale .pyc file isn't inadvertently used. Anyway, if you are going to compare .pyc files you need to ensure that the .py files you use have the same mtime on both the places you are generating the .pyc files. This is not necessarily a trivial matter, if you get the source from a .tar file chances are that it will preserve mtime, but if you got it out from revision control, such as git, chances are it will just have the mtime from when you retrieved the file. The moral of the story here is you need to either ignore bytes 4 - 7, manually set the timestamp to a known value, or ensure that the .py files have the same mtime.

Now, that is just a warm, because, even after doing this, you aren't necessarily going to get the same exact bytes. Most of the time you will, but not always. I was trying this on the socket.py module. The output was almost identical except for three bytes that were transposed as seen in the diff of the hexdump below:

432,433c432,433
< 00001af0  00 00 00 75 01 00 00 00  72 75 01 00 00 00 77 75  |...u....ru....wu|
< 00001b00  01 00 00 00 62 4e 69 ff  ff ff ff 28 0c 00 00 00  |....bNi....(....|
---
> 00001af0  00 00 00 75 01 00 00 00  77 75 01 00 00 00 62 75  |...u....wu....bu|
> 00001b00  01 00 00 00 72 4e 69 ff  ff ff ff 28 0c 00 00 00  |....rNi....(....|

This is kind of weird, almost identical. So I didn't really know what the bytes 0x77, 0x72, and 0x62 were at the time, but I'll give you a clue: the are the ASCII codes for the characters w, r, and b.

Next step in the debugging is to actually decompile (disassemble) the .pyc file. Strangely there doesn't seem to be such a utility in the standard Python set of tools, however this recipe gives you something relatively usable. So after running the .pyc files through that you find the cause of the problem:

480c480
<                                  16 LOAD_CONST               9 (frozenset({'b', 'r', 'w'}))
---
>                                  16 LOAD_CONST               9 (frozenset({'w', 'r', 'b'}))

This comes from a line of code in the original Python:

            if c not in {"r", "w", "b"}:

So, basically, each .pyc has a different representation of the set {'r', 'w', 'b'}. Now of course, these are sets so they are equal and the code works the same way, no real problem there. However, I'd really like the python compile process to be deterministic. (You could say I'm determined to make it so! See what I did there!). Now if you start digging around in to the Python implementation you find that when marshalling a (frozen)set object, the library just pulls out the set elements in the order in which they are stored, so clearly, when the frozenset is created, it appears to put the elements in a different order at different times.

Next step, digging in to the set implementation. Reading through setobject.c it becomes clear that the ordering of items in a set is based on their hash, which makes sense if you think about it. So, this makes sense, but surely values same value will have the same hash across different runs? WRONG!.

Try something like:

$ python3 -c "print(hash('r'))"

You will get a different result each time! It turns out there is a good reason for this. The hash function can be used as an attack vector to DoS a webserver or something else. If the attacker can control which items are used as dictionary keys, then they have the potential to choose a set of values all with the same hash value, which can quickly lead to poor performance!

The fix to this is to randomize the hashing algorithm on each Python invocation, making it much harder for an attacker to guess values that have the same hash function.

More details on the Python bug tracker and more list thread.

The good news is that there is a simple way to disable this randomization by setting the PYTHONHASHSEED environment variable. Once that is done you get deterministic hash values, and then your sets end up being in the same order, and your compilation is deterministic. Win!

The backstory to this one is too tedious to describe, and besides, whenever I do that I get irrelevant comments about the backstory, also I'm going to try and make my posts more concise this year both for my sake and yours. See, it is working already!

Anyway, I'm using autotools to compile binutils, and for some reason when a run make install, the permissions of directories are different on different machines. I mean, it doesn't really matter other than the fact that I'm tar-ing up the result and checking the tarfiles are identical (for various regression tests and validating that builds are reproducible). And I could probably just fix this after make install has run by changing the file permissions as I add the file to the archive (since I already squelch the username and timestamp in a similar manner), but doing that wouldn't let me have fun appreciating how awesome the autotools are.

At first I assumed this was just a umask problem, and that is certainly part of it (why the Fedora default umask is 002 remains a mystery to me). Anyway, after changing my other machine to have a matching 002 umask the same problem was happening. (Fun note, if I'd experimented the other way, and changed my umask on Fedora to something sane, I would have also managed to avoid this fun.). So anyway, at this point on Fedora make install creates group-writable directories (as expected with a 002 umask), however, on OS X make install continues to create non group-writable directories, even with the modified umask.

The best thing about autotools is that it makes the output from make really simple to read and understand, so it didn't take long at all to work out that on Fedora it was directly using mkdir -p to create the directories, however on OS X it was using install-sh -c -d to create the directories. install-sh is a magical shell script that, as far as I can tell, is install during the auto* process.

Just as the output from running an autoconf Makefile is a pleasure to read, the generated Makefiles are themselves a work of beauty primarily underscored by great clarity and concision, much like my own prose./

In a snap, I found the offending line:

test -z "$(man1dir)" || $(MKDIR_P) "$(DESTDIR)$(man1dir)"

Sarcasm aside, this is pretty simple to grok; if we have a non empty $(man1dir) variable, create the directory using whatever MKDIR_P expands to. Given the name of the variable, you would have to assume that this is generally assumed to expand to mkdir -p in the common case, which is exactly what it does on my Fedora box. In any case, we are now just left with two interesting questions: Why isn't mkdir -p used on OS X? and Why does install -c -d have different semantics to mkdir -p? and Why does configure choose that if the semantics are different? and Why can't I count?

Well, to answer the first question, the magical configure script does an explicit version check on mkdir using the --version flag. If your mkdir doesn't give back one of the magically blessed strings, you are out of luck. As you can guess the BSD derived mkdir on OS X doesn't play nicely here, in fact it doesn't even support the --version flag at all. The interesting thing is that this check is described as checking for a thread-safe mkdir -p, but really it is just checking for a GNU mkdir. This check was added in 1.8.3.

Early versions of OS X mkdir certainly seem to have the race-condition (see mkdir.c), however more modern version appear to use mkpath_np underneath, which appears on my first reading to correctly address the issue (see: mkpath_np.c).

OK, so that is a pretty good answer for that first question; looks like autotools saves the day again by protecting us from broken mkdir -p implementations, of which there seem to be many. (Seriously, how do you fuck that up in the first place? Time-of-check vs time-of-use is a pretty simple thing to consider during design. More to the point how does it stay there for what, 20 years, without getting fixed?) Of course, now that various non-GNU mkdir implementations (well at least one), seem to have fixed the problem, it would be nice if configure had some way of knowing and didn't have to drop to the sub-optimal install-sh -c -d.

OK, so now on to trying to work out what is going on with install-sh. In theory this should have the same semantics as mkdir -p. Or at least I think it should if configure is going to choose it in preference to the mkdir -p. Now this script does a lot, but if we find the interesting bit is:

(umask $mkdir_umask &&
    eval "\$doit_exec \$mkdirprog $prefixes")

This is explicitly setting the umask to a variable mkdir_umask before executing mkdir (in a round-about way)

Well, this is pretty much explained in the source:

# Create intermediate dirs using mode 755 as modified by the umask.
# This is like FreeBSD 'install' as of 1997-10-28.

and then backed up by this commit message. Basically, using mode 755 as the base mode is safer than using mkdir's default of 777. On some levels I've got to agree (see my earlier outrage at Fedora's default umask), but shouldn't this kind of script obey whatever I specify as my umask? If I was collaborating on a group project, and had explicitly set my umask to make that easy, then this would be a pretty annoying. Of course, my main problem here is that the behaviour is different to that of mkdir -p.

So, the best work around for this is to just not use a umask of 002 in the first place. To be honest, I'm not really sure if this behaviour, is a bug or a feature, and if it is a bug whether the mkdir -p or the install -d behaviour is correct.

The real moral of this story is that if you are super pedantic about the little things, you get to find out really uninteresting details about software you'd rather not use while wasting time that could be better spent on just about anything else interesting things about the software you use every day, making you a more knowledgeable engineer.