Abstract: This article provides a short explanation of how I fixed a problem I was having with node. In doing so it provides an introduction in to some of the node’s internal architecture.
There are times when running a node webserver where, rather than
opening a network socket yourself, you want to use an existing
socket. For example, I’ve got a simple loader which does some
minimal setup, opening a socket and binding to a port, before
exec()-ing another server (in this case a node
server). In pseudo-code it looks something like:
s = socket(...); bind(s, ...); setuid(unprivileged_uid); exec(...);
The goal behind this is to acquire some privileged resources (such as port number less than 1024) while running as root, and then dropping privileges before executing a larger, less trusted, code base. (I’m sure there are other potential approaches for achieving a similar kind of goal, that isn’t really what this post is about).
Anyway, if you use this approach, when calling a createServer
API, rather than providing an address and port
when listening, you can provide a file descriptor. Effectively what
happens is that the code operates something like:
if (fd is null) {
fd = socket(...)
bind(fd, ...)
}
fd.listen()
This pattern worked really well for me on node around the time of
version 0.4. As time passed and I upgraded from 0.4 through to 0.8
this approach appeared to continue working real
well. Unfortunately, not appearances can be deceiving! It wasn’t until
much later when trying to log an request’s remote address that I
started to run into any problems. For some reason despite everything
mentioned in the docs al attempts to derefence the
remoteAddress property resulted in failure. It’s at this
time that open source (or at least source available) libraries really
shine, as it makes debugging really possible. A few dozen
printfs later (well, OK, a combination of
console.log and printf), and I was able
to rule out any of my code, and any of express.js and any of the
node http.js and https.js libraries.
The problem was that my socket object didn’t have a
getpeername method, which is weird, because you expect
sockets to have a getpeername method. The reason that the
socket object didn’t have a getpeername method, is that
it wasn’t actually a socket. Well, unfortuantely things
aren’t quite that simple. There are a number of layers of abstraction
in node, that make some of this more difficult to understand!
The top level of abstraction there is a javascript object that is
based on node’s Socket prototype. The Socket
prototype provides a number of userful methods; most of the things
that you would expect from a socket: connect, read
write, etc. It also defines the property we are most
interested in: remoteAddress. Now this is a really simple
property:
Socket.prototype.__defineGetter__('remoteAddress', function() {
return this._getpeername().address;
});
As you can see, this property defers all the real work to another
method _getpeername:
Socket.prototype._getpeername = function() {
if (!this._handle || !this._handle.getpeername) {
return {};
}
if (!this._peername) {
this._peername = this._handle.getpeername();
// getpeername() returns null on error
if (this._peername === null) {
return {};
}
}
return this._peername;
};
Now, ready the code for _getpeername it should be
clear that the Socket object is primarily a wrapper for
an underlying _handle. After exporing the code in
net.js some more it becomes clear that
_handle could be one of two different things: a
Pipe() or a TCP() object. These objects are
implemented in C++ rather than Javascript. Now, reading the code for
these objects (in pipe_wrap.cc and
tcp_wrap.cc) it becomes clear that these two objects
have very similar implementation, however the TCP object provides a
few more features, critically it provides the getpeername
method, whereas the Pipe object does not. This is the key to the underlying
problem: my Socket objects have a Pipe object as the handle, rather than an
a TCP object as the handle. The next question is why!
The answer lies in the createServerHandle function residing
in net.js. This is the function that eventuallly is called when
the HTTP listen method is called. This has some code like this:
if (typeof fd === 'number' && fd >= 0) {
var tty_wrap = process.binding('tty_wrap');
var type = tty_wrap.guessHandleType(fd);
switch (type) {
case 'PIPE':
debug('listen pipe fd=' + fd);
// create a PipeWrap
handle = createPipe();
default:
// Not a fd we can listen on. This will trigger an error.
debug('listen invalid fd=' + fd + ' type=' + type);
handle = null;
break;
....
The upshot of this code is that if an fd is specified, node
tries to guess what kind of thing the file descriptors refers to and creates
th handle based on this guess. The interesting thing with the switch statement
is that it only has a single valid case: PIPE.
This is interesting for a couple of reasons; it clearly isn’t hitting the default
error case, which means that node somehow guesses that my file descriptor is a
PIPE, which is strange, since it is definitely a socket. The kind of amazing thing
is that apart from the inability to retrieve the remote address, everything in
the system works perfectly. We’ll come back to this point later, but for now
the challenge is to try and find out why the the guessHandleType
method would think my socket is a pipe!
Again we get to follow the layers of abstraction
game. guessHandleType is implemented in the
tty_wrap function:
Handle<Value> TTYWrap::GuessHandleType(const Arguments& args) {
HandleScope scope;
int fd = args[0]->Int32Value();
assert(fd >= 0);
uv_handle_type t = uv_guess_handle(fd);
switch (t) {
case UV_TTY:
return scope.Close(String::New("TTY"));
case UV_NAMED_PIPE:
return scope.Close(String::New("PIPE"));
case UV_FILE:
return scope.Close(String::New("FILE"));
default:
assert(0);
return v8::Undefined();
}
}
So, this wrapper function is really just turning results from an
underlying uv_guess_handle function in to strings. Not
much interesting, although it is somewhat interesting to know that at
one layer of abstraction things are called NAMED_PIPE
but simply PIPE at another. Looking at
uv_guess_handle gets more interesting:
uv_handle_type uv_guess_handle(uv_file file) {
struct stat s;
if (file < 0) {
return UV_UNKNOWN_HANDLE;
}
if (isatty(file)) {
return UV_TTY;
}
if (fstat(file, &s)) {
return UV_UNKNOWN_HANDLE;
}
if (!S_ISSOCK(s.st_mode) && !S_ISFIFO(s.st_mode)) {
return UV_FILE;
}
return UV_NAMED_PIPE;
}
Huh, so, from this code it becomes clear that if a file-descriptor
is a FIFO or a SOCK the uv library wants to treat the file-descriptor
as a UV_NAMED_PIPE. This would seem to be the source of
the problem. Clearly named pipes and sockets are different things, and
it is strange to me that this function would force a socket to be
treated as a named pipe (especially when the upper layer treat the
two things differently). As an aside, this is the place where comments
in code are essential. At this point I have to guess why this is written
in this non-obvious manner. Some potential things come to mind: compatability
with the Windows version, expediency, a restriction on the API
to only return one of { TTY, UNKNOWN, FILE or NAMED_PIPE }. The other
odd things is that a macro S_ISREG is also provided that could
test explicitly for something that is UV_FILE, but that isn’t
used so things such as directories, character devices and block devices are
also returned as just FILE. Without comments it is difficult to
tell if this was intentional or accidental.
So the first fix on the way to solving the overall problem is to
update uv_guess_handle so that it can actually return an
appropriate value when we have a socket and not a pipe. There are two
possible defines that this could be: UV_TCP or
UV_UDP. There is likely a some way to distinguish
between the two, however for my purposes I know this is always going
to be TCP, so I’m going to return UV_TCP. Of course, I’m
sure this will cause someone else a similar problem down the track, so
if you know what the appropriate interface is, let me know!
Another approach that could be used is to leave this function alone entirely and provide some way of specifying the to listen and createServer whether the file-descriptor is a named-pipe or a socket, however deadling with all the parameter marshalling through the stack made this somewhat unattractive.
Once this is done, net.js can then be fixed up
to create a TCP() object, rather than a Pipe()
object in createServerHandle.
Aside: I find it amazing that a 43-line patch requires aroudn 1400 words of English to explain the rationale and back-story!
Now we’ve fixed enough of the underlying libraries that we can think of fixing net.js. What we’d like to do is something like this:
var type = tty_wrap.guessHandleType(fd);
switch (type) {
case 'PIPE':
debug('listen pipe fd=' + fd);
// create a PipeWrap
handle = createPipe();
break;
case 'TCP':
debug('listen socket fd=' + fd);
handle = createTCP();
break;
default:
// Not a fd we can listen on. This will trigger an error.
debug('listen invalid fd=' + fd + ' type=' + type);
global.errno = 'EINVAL'; // hack, callers expect that errno is set
handle = null;
break;
}
if (handle) {
handle.open(fd);
handle.readable = true;
handle.writable = true;
}
return handle;
This looks pretty good, but unfortunately it doesn’t work. That’s because
although the Pipe() object supports an open method,
the TCP() object (which has a very similar set of APIs) does not
support the open method! How very frustrating. Now the implementation of pipe’s
open method is relatively straight forward:
Handle<Value> PipeWrap::Open(const Arguments& args) {
HandleScope scope;
UNWRAP(PipeWrap)
int fd = args[0]->IntegerValue();
uv_pipe_open(&wrap->handle_, fd);
return scope.Close(v8::Null());
}
That looks pretty easy to implement on the TCP object, however
we hit a problem when we get to the uv_pipe_open line.
This function takes something of uv_pipe_t type, where
as the handle in a TCP object has uv_tcp_t type.
Dropping down another level of abstraction again, we need to add
a new uv_tcp_open function in that takes a uv_tcp_t
parameter. It turns out that this is pretty straight foward:
int uv_tcp_open(uv_tcp_t* tcp, uv_file fd) {
return uv__stream_open((uv_stream_t*)tcp,
fd,
UV_STREAM_READABLE | UV_STREAM_WRITABLE);
}
The only frustrating thing is that this is almost an exact
duplicate of the code in uv_pipe_open. Such egregious
duplication of code rub me the wrong way, but I don’t think there is a
good alternative.
A this point I do find it somewhat amazing that there are four levels of abstraction:
_getpeername).uv_tcp_t
and uv_pipe_t object. These have similar shaped APIs however the specific
functions names are distinct.I’m sure it is done for a good reason (most likely to provide compatability with Windows, where named pipes are sockets are probably different the kernel interface layer), however it is somewaht amusing that th eabstraction moves from the lowest kernel level, where both objects are just represetned as ints, and any function can be applied, up through two levels of abstaction that make each thing very different types, before ending up at a top-level where they are essentially recombined in to a single type.
In any case, after a lot of plumbing, we reach are able to make sure that when we listen on a socket file-descriptor, we actually get back socket objects that correclty support the remoteAddress property, and all is good with the world.