Dave has been working like a maniac to switch our database code over from SQLite to PostgreSQL. PostgreSQL has two main advantages: it is much faster, and we can open up ODBC connections to the database for other uses that don’t require a web interface. The change is now complete, however it hasn’t been without some difficulties. One problem that bit us was running out of file handles. If you ever have a similar problem, here is how to debug it.

On Linux the /proc filesystem reflects a great many kernel resources. The particularly interesting directories for our purposes are:

• The files file-nr and file-max in /proc/sys/fs.
• The per process directories keyed by process ID

The first thing to check is the value of /proc/sys/fs/file-max, which is the maximum number of file handles allowed on your system. This shouldn’t be a problem, but just ensure it isn’t something ridiculously small. On our system we get:

$ cat /proc/sys/fs/file-max
89367


That should be plenty under any reasonable usage, but we can check how many file handles are open by reading the value of/proc/sys/fs/file-nr. On our system this is:

$ cat /proc/sys/fs/file-nr
920 0 89367

This first number is the number of file handles in use. Definitely no problem there. It must be that a process is exceeding the per-process limit on file handles. In our setup this could be either PostgreSQL or MzScheme. We need the process IDs to find out how many handles each is using.

$ ps -A | grep postmaster
12936 ? 00:00:00 postmaster
12937 ? 00:00:00 postmaster
12939 ? 00:00:00 postmaster
12940 ? 00:00:00 postmaster
12941 ? 00:00:00 postmaster
$ ps -A | grep mzscheme
20382 ? 00:00:26 mzscheme

We can see how many handles are in use by looking in the directory for each process ID. For example, for the first PostgeSQL process:

$ sudo ls -l /proc/12936/fd/ | wc -l
4

So that PostgreSQL process is using 4 handles. The other processes are using similar numbers. So it must be our MzScheme process that is using up all the handles. We check that in a similar way, and the result is:

$ sudo ls -l /proc/20382/fd/ | grep socket | wc -l
193


Looks like we’ve found our culprit.

If you happen to be in Birmingham on the 18th I’m presenting our current ideas on web development as part of the School of Computer Science’s Cake Talk series. The abstract is below. If you intend to attend follow the link for location and time. My slides will go up after the talk.

Functional Programming and the Web

Continuations, functional reactive programming, and
bidirectional programming. A random walk down
Lambda the Ultimate or the next Big Thing in web
development? In the long and glorious tradition of Cake
Talks I will present some half-baked ideas that argue for
the later interpretation. Turn up and decide for yourself.

One of the recurring themes of the Dagstuhl workshop was “convenience matters”. This is clearly true. All Turing complete programming languages have equivalent power yet we prefer some over others. It’s about what they make easy.

This point came to mind when looking at the various abstraction mechanisms in PLT Scheme. The designers of PLT Scheme have analysed the types of abstractions people often use, carefully separated them into different classes, and provided separate mechanisms for each class. If you want to create a unit of functionality you can use the module system. If you want to parameterise code you can use the unit system. If you want to dispatch on type you can use the OO system. This is in contrast to Java, where you get one main abstraction mechanism, the class, which is a module system, a way of parameterising code, and a dispatch mechanism all mashed up into one. This is considered undesireable as you are forced to consider interactions with other mechanisms when you only want to use one. However it has one big advantage: convenience. I don’t often use PLT Scheme’s unit system, so when I do I have to look up the documentation. Same with PLT’s OO system. This tends to make me avoid using them, as I don’t like to spend time reading the docs or figuring out the system. However with Java you’re always writing classes, so it’s familiar and the barrier to use is lower. Convenience matters.

Here at Untyped Central Dave and I are hacking away like lumberjacks, which is a good thing as the project we’re working on is due soon. We’re constantly running tests and loading code into the web server, and we’ve noticed that these processes have been getting slower and slower. It turns out the bottleneck is the time to parse and byte compile our Scheme code. Simply byte compiling the code beforehand has made an incredible difference. Tests that used to take minutes now run in seconds. Two features of PLT Scheme make it really easy to integrate byte compilation into our development process. Firstly, mzc will follow dependencies when given the -k flag. So we just run mzc -k main.ss and all our code is compiled. Also useful is that PLT Scheme does the Right Thing and loads source code if it’s newer than byte compiled code, so we don’t have to constantly recompile our code. So we can just code away as normal, except every time we take a break we run mzc. Eventually we might write some code to recompile at regular intervals (say, every 10 minutes) but for now it isn’t worth the effort.

Dave’s on a roll. A few days ago we heard he’s on the ECMAScript committee. Now he follows up with a great post on Nancy typing(read the post to get the joke), and a PLT Scheme language that implements Javascript (available from PLaneT, of course).

There’s a good summary of lessons learned from building del.icio.us. It’s a bit telepathic at times but there’s a lot of good stuff in there.

The Yahoo! Design Pattern Library is a collection of design patterns for web applications, along with links to the also-newly-released Yahoo! User Interface Code Library. Most of the patterns should be familiar to web application developers but it is good to have them all collected in one place. The code should be good to, though I haven’t had a chance to look at it yet.