The Biographicon is a very pretty web site of user contributed biographies. It is also written in Scheme, and here’s one of the developers discussing how it was done.

We’ve just released a package called Instaservlet, which enables you to get a servlet running in two lines of code, plus the servlet code. Here’s a simple example:

  (require (planet "instaservlet.ss" ("untyped" "instaservlet.plt" 1)))

(define (servlet request)
'(html (head (title "It's working!"))
(body (h1 "Instaservlet is in the house!"))))

(go! servlet)

Try it in PLT Scheme 3.99 and see!

This package arose out of the development work I did on Smut Shorts, which made me I realise we needed to support a quick start for web development. Instaservlet is the first step in doing this. Not only does it setup the servlet, it does it in a robust manner. Continuations are managed using the LRU manager, which can be a bit difficult to setup but it much more robust than the time-out manager. Instaservlet uses some sensible defaults which should work for most small to medium size sites — they’re the setting used on Smut Shorts, so we know they can handle a reasonable load. It also sets up a nice default page to handle continuation expiry. Instaservlet is good enough to get you started with, and future versions will enable more customisations so it can scale to any website build using PLT Scheme.

For Valentine’s Day we have created a new website,<a
href=”http://smutshorts.com/”>Smut Shorts. If you have
something to say about love or lust, and can do so in 500
characters or less, then please add it to the growing number
of “shorts”. It’s anonymous and fun. Just, no
porn thanks.

If you’re reading this site, you’re probably interested
in the technical details behind Smut Shorts. It is a
collaboration between a number of people, most of whom have
chosen to be anonymous. The majority of the coding was done
by yours truly, and therefore in Scheme. It is running on
the PLT Scheme webserver (version 3.99) and uses PostgreSQL
as the back-end. I coded it up in about two days. It was a
side-project, so it was a bit of rush job and there is lots
still to do. If you break the site let me know and I’ll try
to fix it.

A few interesting lessons were learned from doing this
site. It all comes down to scalability, which is something
that has recently been on my mind a lot. In this case we
want to scale down to the low end — the guy who is
just hacking up something in his spare time and wants to get
it done in a hurry. Our frameworks, Snooze and Lylux, are
pretty good but they don’t support a fast start. You have
to create a whole bunch of files before you’ve even got your
first page up. Furthermore, we’ve always avoided creating a
templating mechanism, as we’ve said that we’d rather use
smart people who can balance parens than create this
unnecessary divide between designers and programmers. I now
recognise this is a mistake. Had we a templating mechanism
I could have pushed more design work to my collaborators.
It’s not that they’re stupid (far far from it) but they’re
busy and don’t have time to learn even Scheme basics just so
they change a few lines of text. If we’re gonna grow the
Scheme web-hacking community it has to start with dudes
messing around in their spare time, so we need to address
the low-end of scale. The high end of scale can wait till
the IPO 

Delirium is a web-browser automation toolkit, which means it’s a Scheme library that provides a bunch of functions that you can use to control a web browser. We expect the primary use will be for web testing, and Delirium can be used inside SchemeUnit like any other Scheme library.

For the Schemers Delirium isn’t anything special, but we believe the use of continuations make Delirium a major advance over similar web testing tools like Selenium. If you write your server code in Scheme you can directly test how your server side responds to web tests with tests running on the same server. That is to say a test can interleave calls to the web browser and to the server side code, which is impractical without continuations. This features makes it much easier to write reliable and comprehensive tests.

Delirium is on PLaneT. Note the documentation was translated by hand from Scribble source. Some errors may have been introduced during the translation.

We’ve just released <a
href=”http://planet.plt-scheme.org/display.ss?package=instaweb.plt&owner=schematics”>Instaweb
2.0. Instaweb is our utility that takes care of setting
up the PLT web-server and running servlets. If you have a
servlet in a file called servlet.ss with
Instaweb you just need to write the following lines to get
it running:

(require (planet "instaweb.ss" ("schematics" "instaweb.plt" 2)))
(instaweb)

The new version of Instaweb includes many new options and
works in a slightly different way to the 1.0 branch. To my
mind the best new feature is that Instaweb now configures
the web-server to pass to the servlet all requests
that don’t match a file in the htdocs
directory. This means your servlet no longer has to live
under a URL starting with /servlets. You can
<a
href=”http://planet.plt-scheme.org/package-source/schematics/instaweb.plt/2/1/doc.txt”>read
the documentation to get the full details of what’s new.
What I want to talk about here is how we implelemented this,
as it illustrates some very nice features of the web-server
that aren’t well known.

In the web-server’s terminology a dispatcher is a
function that may generate a response given a request.
Examples includes the filesystem dispatcher, which responds
to requests with the contents of a file, and the servlet
dispatcher, which invokes a servlet. Dispatchers are
arranged in a list. The first dispatcher in the list
inspects the request and, if it decides the request is
relevant, generates a response. Otherwise control is passed
to the next dispatcher in the list. For some time now the
web-server has had a configurable dispatcher pipeline, which
can be set by simply passing a value with the
#:dispatch keyword to the serve
function.

The web-server provides a number of dispatchers, all in
the <a
href="http://svn.plt-scheme.org/plt/trunk/collects/web-server/dispatchers/">dispatchers
subdirectory of the web-server collection.
They all provide a make function that does most
of the work. Here’s how to use the file, servlet, and
sequence dispatchers, the most generally useful ones:

1. The file dispatcher, in
   dispatch-files.ss, takes a single parameter, a
   function that converts a URL to a path (and another value
   that the dispatcher ignores). The path can name a file,
   which the dispatcher will serve if such a file actually
   exists, or it can name a directory, in which case the
   dispatcher will look for a file within that directory called
   index.html or index.htm.

   To use the file dispatcher you will probably want the
   handy make-url->path function in
   filesystem-map.ss. Pass this function a base
   path (the directory where your files live), and it will
   return a function suitable to pass to the file
   dispatcher.

   Here’s an example of use:

   (require
   (prefix file: (lib "dispatch-files.ss" "web-server" "dispatchers"))
   (lib "filesystem-map.ss" "web-server" "dispatchers"))
   
   (define base-path (string->path "/my/directory/of/files"))
   
   ;; htdocs-url->path : path -> (url -> path (list-of path-element))
   (define (htdocs-url->path path)
   (make-url->path (path->complete-path path)))
   
   ;; dispatch-htdocs : (connection request -> response)
   (define dispatch-htdocs
   (file:make #:url->path (htdocs-url->path base-path)))

2. The servlet dispatcher, in dispatch-servlets.ss is a bit more difficult to use as you need a function from the privatesubcollection of the web-server, suggesting the code reorganisation isn’t quite finished. The make function takes two arguments, the first being a cache-table, and the second being a function that, like for the file dispatcher, maps URLs to paths. To construct a cache-table use the following lines of code:

   (require
   (lib "cache-table.ss" "web-server" "private"))
   
   (define cache-table (box (make-cache-table)))

   If you want all URLs to go a particular servlet, as in Instaweb, the URL to path function just needs to return the path of the servlet. The function used in Instaweb is this:

   ;; serlvet-url->path : url -> path (list-of path-element)
   (define (servlet-url->path url)
   (let ([complete-servlet-path (path->complete-path servlet-path)])
   (values complete-servlet-path (explode-path* complete-servlet-path))))

   Now we can create a dispatcher as follows:

   ;; clear-servlet-cache! : -> void
   ;; dispatch-servlets:    connection request -> response
   (define-values (clear-servlet-cache! dispatch-servlets)
   (servlet:make (box (make-cache-table))
   #:url->path servlet-url->path))

3. The sequencer dispatcher couldn’t be easier to use. It just takes any numbe of dispatchers and creates new dispatcher that tries them in sequence. For example:

   ;; dispatch-all : connection request -> response
   (define dispatch-all
   (sequencer:make dispatch-htdocs
   dispatch-servlets))

With the above you should be able to create your own custom dispatchers. If you have problems just read the (very short) Instaweb code!

If you’ve looked at the ICFP 2007 preliminary program you’ll have noticed we’re presenting “Experience Report: Scheme in Commercial Web Application Development”. We submitted the final version of the paper a couple of weeks ago, and I’ve finally got around to putting itonline for your reading pleasure. The contents shouldn’t come as a surprise: a summary of our experiences developing commercial web applications in PLT Scheme over the last year. We’ve tried to be honest, including the good and bad. Hopefully the points you’ll take away are that we’ve been able to overcome initial problems with stability, and in a fairly short time we’ve developed a framework that compares well to popular alternatives such as Ruby on Rails.

The four page limit on experience reports is very tight, and unfortunately our experiences with Flapjax were cut from the final version. So let me say here that if you write Javascript code you need to check out Flapjax! Our Flapjax code is about half the size of the equivalent Javascript, and this is without using the Flapjax compiler. The only problem with Flapjax is performance in large networks. This is more a property of the poor quality of Javascript interpreters: Wolfenstein 3D on my 286 back in 1990-something was smoother than Javascript raycaster running today on my Powerbook. Luckily the new developments taking place at Mozilla will alleviate this problem in the next few years.

The Selenium code I talked about in an earlier post has beenreleased to PLaneT

Two interesting case studies on large web sites: MySpace and EBay. They are remarkably similar. Or perhaps it should be unremarkable. You don’t get many chances to get that sort of system wrong, so it is natural that different groups should converge to tried and tested solutions. Ezra Kilty offers some thoughts on EBay and the great Goog, who of course have a system way different and way cooler than anyone else’s.

In a few days we’ll be releasing our first code using Flapjax to drive the user interface. The core of the system is a grid view that runs in the browser. The user can edit data in place, and when they move to a new row changes are sent to the server. Invalid data is flagged in the grid, and mousing over shows the reason for the error. Given we’re probably the first people in the world to release commercial code using Flapjax a few notes on our experience are worthwhile.

The summary is: Flapjax worked very well for us. Flapjax code is much smaller and faster to write than the equivalent plain Javasscript, and Flapjax insulates you from most of the cross-browser issues. If your application revolves around events, and most Javscript applications do, Flapjax will probably be a good fit. However current Javascript implementations are so slow that they can limit how much you can take advantage of Flapjax.

Now, the details. We went through two design. Our first prototype rendered the TABLE that contained the grid data as one enormous Behaviour. Every field was an “edit in place” field, using more Flapjax code. (See here if you don’t know what an edit in place field is, though note that this isn’t the code we used.) This plays very nicely with the Flapjax model, as changes to any part grid automatically get updated in the browser. It worked fine for small tables, but scaled really badly; for large tables it was so slow to load and redraw that the browser would kill the script.

The main problem with our first prototype is that browsers are really slow at rendering elements inserted using the DOM. Our second design made two major changes to overcome these problems. First we dropped edit in place fields in favour of plain old INPUT elements, and then got the server to render the HTML instead of building the grid dynamically on the client side. So essentially we used Flapjax to handle events from the grid, but not to render the grid. This design is much faster; for example a grid with a thousand cells renders in about a second.

As with any new technology, the community still has to develop best practices and design patterns to make it easier to adopt Flapjax. The best design for error handling is still an open issue. Architectures for mixing OO and FRP (Flapjax) code are not well defined, though there is prior work. However we feel the benefit we got from Flapjax, in terms of shorter code and a faster development cycle, outweighed the cost.

I’m editing this post using the
Performancing extension for the new Firefox 2.0 browser. Being a web whore I simply had to download the latest version of Firefox as soon as it was out.

If anything, Firefox 2.0 is proof that the browser is now a stable product. The changes from 1.5 are incremental. There is nothing that particularly excites me on the user end.

From the point of view of a developer, it’s a different story. Javascript 1.7 continues the evolution of Javascript from a hairball to a language you wouldn’t be embarassed to take to school to meet your Professors. Array comprehensions, proper lexical scoping, and generators are the main features. Support for the WhatWG client-side session and persistent store adds more possibilities for storing continuations on the client-side. I’m interested to see that their motivating example is exactly the same as the one Shriram uses when proselytizing continuations (the “Orbitz bug”).

Of course, to use most of this you’re stuck with Firefox. However, as Firefox evolves further from the functionality offered by IE it becomes more compelling as a platform is its own right. Somebrowser stats show encouraging numbers of people are using Firefox. Give it a few years and IE support may legitimately be optional.