Read on and leave a comment
or return to our portfolio.

Posts in the ‘Scala’ category

20 Oct 2011

by Dave

The Circus is in Town

Following a couple of late caffeinated nights, we are pleased to announce that our latest project, Bigtop, is finally set for beta release!

Bigtop is a collection of libraries for web developers. Included in the first beta release are:

  • Bigtop Routes – a library for creating type-safe bidirectional mappings between URLs and Scala code;
  • Bigtop Core – a library for generating and manipulating URLs and hyperlinks in a functional style.

Routes uses the HList-based approach to bidirectional pattern matching that Dave presented at Scala Lift-Off London 2011. There are versions of the library for use with LiftScalatra, and plain old Java servlets.

Check the Bigtop web site for a getting started guide, API documentation, and links to the code and Maven repository.

Posted in Code, Functional Programming, Scala, Web development | No Comments »

10 Oct 2011

by Dave

Reading, Writing, and the REST

I’ve just finished preparing the slides for my talk at Scala Lift-Off London 2011 next week. The title of the talk is A Route to the Three ‘R’s: Reading, Writing, and the REST. Here’s the abstract:

The mappings between URLs and code form an integral part of any web application. Many web frameworks help map from URLs to code, but the reverse mapping is often neglected, leaving developers to construct URLs via haphazard string manipulation. Furthermore, many frameworks do not match URLs in a type-safe manner.

Scala provides all the tools we need to address this problem in a more comprehensive manner. In this talk we will walk through the creation of Bigtop Routes, a bidirectional mapping library that is both type-safe and developer-friendly. We will pay particular attention to the ways in which Scala language features, such as flexible syntax, implicit conversions, and a touch of type-level programming, help to simplify the task at hand.

The slides and code samples are all available from my Github page. Skills Matter have posted a video of the talk on their web site.

Posted in Code, Front page, Functional Programming, Scala, Web development | 1 Comment »

27 May 2011

by Dave

Friday fun with Scala syntax

It’s Friday – time to kick back and relax with two fun Scala programs we put together for your amusement.

Scala has a pretty flexible syntax (although not as flexible as Racket’s, of course) that makes it popular amongst proponents of DSLs. Not all DSLs have to be serious, however, as I’m about to prove.

First up is Noel’s original concept – proof of his love for DSLs, expressed in purest code:

 class A(num: Int) { def Scala = "It's s" + ("o" * num) + " much fun!" } object I { def <(num: Int) = new A(num) } I <3 Scala // ==> "It's sooo much fun!" 

Second is a simpler work that I call “The Startled Lolcat”:

 object O { def o = "What has been seen, cannot be unseen." } O.o // ==> "What has been seen, cannot be unseen."

Posted in Code, Fun, Scala | No Comments »

23 May 2011

by Noel

The Future of VoIP Phone Configuration Interfaces

We’ve recently completed a very fun and interesting job working on a new interface for managing VoIP phone systems. We have a VoIP phone, provided by Loho, who were also our client for this project. It’s great — we can forward calls to our mobiles, cart the phone around with us (plug it into a network connections and it just works), and it even emails us our voice messages. The only thing not great about our phone is the configuration interface. Luckily, that’s what this project set out to solve.

The brief was to implement an elegant online phone configuration system. Alex, Director at Loho, provided the vision. We provided two weeks of development time, which was enough to create a working prototype. Alex has asked us to not give away too many details about the system, but I can show you a few screenshots. First up, here’s the main screen:

The very stylish main menu of the VoIP administration tool we've built for Loho.

The very stylish main menu of the VoIP administration tool we’ve built for Loho.

Doesn’t give away much, does it? A bit more interesting is a detail of editing a configuration:

Also very stylish: editing the configuration of a voice menu

Also very stylish: editing the configuration of a voice menu

Here I’m editing a voice menu — one of those “Press 1 if you’re interested in giving us all your money” type things.

We think we’ve created a very nice system. Loho tell us they were overwhelmed with interest at a recent tradefair, suggesting we’re not alone in our opinion. While the interface is an important aspect of the work, the backend (which I can talk about!) is just as important. The main task was defining a data model to capture the rich feature set that Loho provide. This turned out to be very similar to designing a programming language and its intermediate representation. For example, we use a continuation-passing style representation to avoid maintaining a stack on the server side. Our representation distinguishes between tail calls and normal function calls to avoid excessive resource consumption on the VoIP side. Relational databases don’t do a very good job of storing recursive datastructures, like the AST of a programming language, so we used Mongo for the data store. In addition to its flexible data model, Mongo is web scale which has given us an immediate status boost at local programmer meetups.

The backend code is implemented in Scala and Lift. There are actually two interfaces to the service. One is the nice interface the users see, and the other is a REST interface that is called by the Asterisk AGI scripts that implement the VoIP functionality. The Asterisk system doesn’t handle all the functionality we represent internally, so the REST interface includes a small interpreter that executes intermediate steps till we arrive at something Asterisk deals with.

Posted in Business, Code, Design, Functional Programming, Scala | No Comments »

6 Mar 2011

by Dave

Javascript compilation for SBT

Over the weekend I knocked up a little SBT plugin to wrap up the Javascript resources in our Lift projects and deploy them as one big minified file. Read on to find out how it works, then grab yourself a copy and take it for a spin.

The plugin scans your webapps directory and looks for files with the extensions .jsm or .jsmanifest. These files, called Javascript Manifests, describe lists of Javascript sources that should be combined into a single file. For example:

# You can specify remote files using URLs...

http://code.jquery.com/jquery-1.5.1.js

# ...and local files using regular paths
#    (relative to the location of the manifest):

lib/foo.js
bar.js

# Blank lines and bash-style comments are also supported

Manifest compilation happens in two phases: first, the plugin downloads and caches any remote scripts specified using URLs. Second, it feeds all of the sources (remote and local) into Google’sClosure Compiler, which concatenates them and minifies everything (and provides excellent services like static type checking to boot). The output from the compiler is a .js file with the same base name and relative path as the original manifest.

There’s not a lot more to it than that. The plugin hooks into SBT’s standard compile and package phases, so your Javascript gets rebuilt automatically alongside your Scala code. If this sounds useful to you, please feel free to grab a copy and take it for a spin. Full details are available in the README on Github.

I should point out that there are other useful SBT plugins that do a similar job. For example, I plagiarised extensively from Jon Hoffman’sYUI Compressor plugin and Luke Amdor’s Coffee Script plugin when writing my code. These two particular examples don’t do file combination, though, and that was an important feature for our specific use case.

Posted in Code, Front page, Javascript, Scala, Web development | No Comments »

2 Mar 2011

by Dave

Setting the run.mode in Lift web apps

Update: You can now set the run mode easy and conveniently using our sbt-runmode plugin for SBT.

Setting the run.mode in Lift applications is the source of a surprising number of questions. The documentation recommends passing it as a parameter when the JVM is invoked. This can be hard to achieve for various reasons. In our case our deployment is automated using Chef, and scripts to start and stop the Jetty web server are installed by the package manager. We don’t really want to monkey around with these scripts, so we had to find another way. Jetty is written in Java, which means it must have a ridiculously complex XML configuration language. The Jetty developers turned it up to 11 by making their configuration language Turing complete, so we can actually set the system properties in a configuration file. The file we want to create isWEB-INF/jetty-web.xml and we want it to contain this:

 

<?xml version=”1.0″?>
<!DOCTYPE Configure PUBLIC “-//Mort Bay Consulting//DTD Configure//EN” “http://jetty.mortbay.org/configure.dtd”>
<Configure class=”org.mortbay.jetty.webapp.WebAppContext”>
  <Call class=”java.lang.System” name=”setProperty”>
    <Arg>run.mode</Arg>
    <Arg>production</Arg>
  </Call>
</Configure>

 

If we leave this around then our application will always run in production mode. We don’t want this when we’re developing as we won’t, for instance, get stack traces printed to the browser. Thus we should copy this file in when we package up the project, and remove it when the packaging step completes. Assuming you’re using SBT, store the above text in project/jetty-web.xml and add the following to your SBT project file to get this functionality:

 

  val jettyWebPath = “src” / “main” / “webapp” / “WEB-INF” / “jetty-web.xml”
  lazy val installProductionRunMode = task {
    FileUtilities.copyFile(“project” / “jetty-web.xml”,
                           jettyWebPath,
                           log)
    log.info(“Copied jetty-web.xml into place”)
    None
  } describedAs(“Install a jetty-web.xml that sets the run mode to production”)
  lazy val superPackage = super.packageAction dependsOn(installProductionRunMode)
  lazy val removeProductionRunMode = task {
    FileUtilities.clean(jettyWebPath, log)
    None
  } describedAs(“Remove jetty-web.xml and hence set run mode back to testing”)
  override def packageAction = removeProductionRunMode dependsOn(superPackage) describedAs BasicWebScalaProject.PackageWarDescription

 

This is pretty simple code. Basically it redefines the package action to first copy in the jetty-web.xml file, then it runs the original package action, and finally it deletes the jetty-web.xml. Now any WARfiles you run under Jetty will automatically be in production mode, but callingsbt jetty-run will still give you development mode.

Posted in Code, Scala, Web development | No Comments »

1 Dec 2010

by Dave

File upload using Comet Actors

We’ve been using the Lift web framework for a lot of web development work recently, and we’re very impressed some of its features. Lift’s Comet support, in particular, is a blessing for the kind of data-crunching back-end web sites we typically get involved in.

Importing data from uploaded files, for example, frequently causes trouble. An import can take from a few seconds to a few minutes depending on the size of the file and the complexity of the data processing and validation involved. If the import takes more than a few seconds there is an increasing risk that the web browser will time out. If this happens we fail, because the user won’t know whether the import succeeded or not. Lift’s Comet actors provide a simple way around this problem. But before describing how they work, let’s quickly go over Comet and actors.

Comet is a way of doing push notifications over HTTP, which on the face of it appears to only support pull. Without the jargon, this means a way of allowing the server to send information to the web browser when that information is ready, not when the web browser checks for it. This gives us a better interface, as the UI can instantly reflect new data, and better resource consumption, as the client doesn’t have to continuously poll the server.

There are two or three common ways of implementing Comet. Lift uses a mechanism called “long polling”, which implements Comet using plain old AJAX. As soon as the web page loads, the web browser sends an XMLHTTP request to the server. Instead of replying immediately the server keeps the connection around until it has information to push back. When information is available, the web server responds to the HTTP request, and the browser processes the response and immediately makes another request.  In other words, long polling uses HTTP’s pull mechanism to simulate push communication. This is all well and good, but it immediately raises two issues: how do we manage a large number of open, but idle, connections without swamping the server, and what programming model do we use to manage the additional complexity of Comet applications.

Handing many idle open connections is relatively simple. The traditional model is to use one thread per request, but this doesn’t scale when many requests are idle for long periods. All modern operating systems provide a scalable event notification system, such as epoll or kqueue, allowing a single thread to simultaneously monitor many connections for data. The JVM provides access to these systems via the Selector abstraction in the NIO package. All this is taken care of in the web framework, so the application programmer does not need to be aware of it. (Note that other languages present the same facilities in different ways. Erlang, for example, presents all IO operations as blocking, but the implementation uses the same scalable non-blocking OS services as the JVM. Erlang can do this as it doesn’t use as many resources per thread as the JVM does. This is an appealing choice as it provides a uniformity not found on the JVM, but impacts how Erlang handles multicore.)

More relevant to the application programmer is the programming model used for Comet, and this is where actors come in. An actor is basically a thread with the important restriction that it only communicates with the outside world via messages. To ask an actor to do something, you send it a message. This is rather like a method call, except that the actor queues the message and processes it asynchronously. When an actor wants to communicate with another resource, it sends that resource a message. Since actors never share state with each other, there is never a need to lock resources to avoid concurrent access. This is a great model because all the complexities of programming with locks disappear. If you are interested in more information on the actor model in Scala try here for the original papershere for the Akka framework and here for a bit on Lift’s actors.

Actors are a natural fit for Comet. On the server each Comet connection is handled by a Comet actor, whose job it is to manage communication with a connected browser. Each actor is bound to a single user’s session, but actors persist across web requests. We can asynchronously send an actor messages (whether the user is looking at the web page or not), and have the actor buffer them for transmission to the browser. This means we’ve got almost all of our file upload functionality straight out of the box, without having to do any particularly tricky development.

We put a proof-of-concept of the file uploader on Github. The basic structure of the code is:

  • When a file is uploaded it is handed off to a thread for processing, and a Comet actor is started to communicate with the client.
  • The processing thread periodically sends messages to the actor, informing it of progress on the file upload.
  • The Comet actor in turn communicates progress to the client.

The great thing about this arrangement is that the user can navigate away from the page without aborting the file upload, and if they later return to the page they will get a progress update. It makes for a very pleasant UI.

Posted in Code, Functional Programming, Scala, Web development | No Comments »