Read on and leave a comment
or return to our portfolio.

Posts in the ‘General’ category

7 Aug 2013

by Dave

Writing Documentation using Grunt and Jekyll

Running a startup is a lot of work. All manner of tasks constantly compete for the team’s attention mandating a careful dance to keep on top of everything. We haven’t nailed all the footwork to this dance yet, but we have learned that choosing the right tool for each job can drastically simplify the choreography.

We recently rewrote a large portion of Myna in the march towards version 2 of the platform. Most of the customer-facing software – the dashboard, client libraries, HTTP API, and so on – were completely new and required new documentation. Our existing help pages were in need of an overhaul, so we decided to move them out into their own project using a build system based on Grunt and Jekyll.

The new documentation is still a work-in-progress – you can watch its evolution on the Myna website and its repository on Github (yes, it’s open source – another experiment we’re trying). We’re really happy with the way it’s all working out, so we’ve published the build system as a separate project that you can use to bootstrap your own documentation. Go forth, fork, and profit!

Why Jekyll?

Our old documentation was implemented as a set of templates in the Play 2 web app that runs our marketing site and original customer dashboard. Publishing the app requires a PhD in SBT, Mongo and Redis, and if you’re writing documentation on the train (as we are prone to do) it isn’t uncommon to have your plans abruptly terminated by one of SBT’s frequent unavoidable urges to download the whole internet (unadvisable in the middle of signal-free rural England).

We considered moving to a CMS such as WordPress, but our support team are all developers with their own preferred editors and IDEs. Forcing them to write documentation (painful) in a tiny WYSIWYG editor embedded into a web site seemed like torture. Also, CMSs require internet connections… it kinda goes with the territory.

Grunt and Jekyll, in contrast, run completely offline, and have a number of other advantages too. Plugins like grunt-contrib-watch provide instant previews via Livereload, and Jekyll’s syntax highlighting (provided by Pygments) can highlight any syntax you throw at it (including, to my amazement, HTTP).
Jekyll isn’t completely perfect for the job. We had to work around a few issues. Fortunately, none of them proved insurmountable:

Versioning

We’ve run into versioning problems many times before, some requiring some pretty serious workarounds. In fact, versioning issues are pretty much endemic across all software development platforms. In this toolchain we’re relying on lots of components: Grunt, five Grunt plugins, Ruby, and Jekyll. Fortunately, versioning is pretty much a solved problem these days: NPM and Bower are great package managers for Node, and Bundler normalizes not only the version of Jekyll we’re using, but also the version of Ruby itself.

Static Assets (Say “NO” to Plain CSS)

I may catch some flack for this, but CSS is a silly language riddled with missing features and bizarre design decisions. No way are we going to battle with a new documentation project without tools like Less CSS and Twitter Bootstrap to support us. And if we’re compiling and minifying our CSS, we might do it for Javascript as well. Jekyll doesn’t support support for either process out-of-the-box.

One way of solving these issues would be to use Jekyll plugins – there are many candidates available on Github. However, we prefer to use Grunt for this kind of thing, running Jekyll via grunt-exec and Bundler, and using grunt-contrib-watch and grunt-contrib-connect for preview functionality.

Content Navigation

Jekyll has built-in support for cataloguing and paginating blog posts, but it can’t natively generate navigation for a hierarchical documentation site. Fortunately, this was easy to work around with a couple of custom plugins: one to create a table of contents for the sidebar, and one to generate next and previous buttons at the bottom of each page.

Authenticating Users

The main navigation bar on Myna changes when users log in. Ideally we want like to keep this consistent across the main web site, the blog, and the documentation. Our solution is to built a small Javascript app to monitor the user’s login details and rewrite the navbar on demand. This is a work-in-progress project and it’s not part of the Github repo above.

Conclusion

If you like the idea of writing documentation in Markdown, you can get started in two minutes by cloning our  Github repo  and following the instructions in the README. We’d love to hear from you if you find our system useful, and we welcome pull requests with improvements.

Posted in Code, Front page, Fun, General, Javascript, Myna, Web development | No Comments »

28 Aug 2012

by Dave

New site

The portfolio on untyped.com was in need of some love, so we’ve redesigned the site and brought it up to date.

The new design is a lot simpler. We’ve combined the main page with our new portfolio and given each project its own distinctive look and feel. For example, the section for Myna mimics mynaweb.com, and the section on the Urban River Survey is based on the map of survey data around London*.

We should also give another round of thanks to Si Hammond, who came out with us to photograph our beloved Royal typewriter mascot against various Birmingham-based backdrops. We’ve got such mileage out of these photographs – they lend real character!

Posted in General | No Comments »

23 May 2011

by Noel

Internship This Summer at Loho

Our friends at Loho are looking for an intern over summer. We’vepreviously blogged about the (fun and interesting) work we’ve done for Loho: using ScalaLift, and a heavy does of Javascript we’ve created a really simple interface for setting up complex VoIP services. (All credit goes to Alex for the great concept — our work is simply implementing his vision.) The internship could extend this work, or address other parts of the stack right down to mucking around with VoIP hardware. It’s an awesome chance to play with a lot of exciting technology. If you fit the bill (student, free for 8-10 weeks over summer, can relocate to Cambridge) get in touch with Alex at Loho. If you get the internship and decide to focus on the Scala/Lift parts we’ll certainly help you get to grips with the code base.

Posted in General | No Comments »

11 Feb 2011

by Noel

Stop A/B Testing and Make Out Like a Bandit

This is the blog post that led to Myna. Sign up now and help us beta test the world’s fastest A/B testing product!

Were I a betting man, I would wager this: the supermarket nearest to you is laid out with fresh fruit and vegetables near the entrance, and dairy and bread towards the back of the shop. I’m quite certain I’d win this bet enough times to make it worthwhile. This layout is, of course, no accident. By placing essentials in the corners, the store forces shoppers to traverse the entire floor to get their weekly shop. This increases the chance of an impulse purchase and hence the store’s revenue.

I don’t know who developed this layout, but at some point someone must have tested it and it obviously worked. The same idea applies online, where it is incredibly easy to change the “layout” of a store. Where the supermarket might shuffle around displays or change the lighting, the online retailer might change the navigational structure or wording of their landing page. I call this process content optimisation.

Any prospective change should be tested to ensure it has a positive effect on revenue (or some other measure, such as clickthroughs). The industry standard method for doing this is A/B testing. However, it is well known in the academic community that A/B testing is significantly suboptimal. In this post I’m going to explain why, and how you can do better.

There are several problems with A/B testing:

  • A/B testing is suboptimal. It simply doesn’t increase revenue as much as better methods.
  • A/B testing is inflexible. You can’t, for example, add a new choice to an already running test.
  • A/B testing has a tedious workflow. To do it correctly, you have to make lots of seemingly arbitrary choices (p-value, experiment size) to run an experiment.

The methods I’m going to describe, which are known as bandit algorithms, solve all these problems. But first, let’s look at the problems of A/B testing in more detail.

Suboptimal Performance

Explaining the suboptimal performance of A/B testing is tricky without getting into a bit of statistics. Instead of doing that, I’m going to describe the essence of the problem in a (hopefully) intuitive way. Let’s start by outlining the basic A/B testing scenario, so there is no confusion. In the simplest situation are two choices, A and B, under test. Normally one of them is already running on the site (let’s call that one A), and the other (B) is what we’re considering replacing A with. We run an experiment and then look for a significant difference, where I mean significance in the statistical sense. If B is significantly better we replace A with B, otherwise we keep A on the site.

The key problem with A/B testing is it doesn’t respect what the significance test is actually saying. When a test shows B is significantly better than A, it is right to throw out A. However, when there is no significant difference the test is not saying that B is no better than A, but rather that the data does not support any conclusion. A might be better than B, B might be better than A, or they might be the same. We just can’t tell with the data that is available*. It might seem we could just run the test until a significant result appears, but that runs into the problem of repeated significance testing errors. Oh dear! Whatever we do, if we stick exclusively with A/B testing we’re going to make mistakes, and probably more than we realise.

A/B testing is also suboptimal in another way — it doesn’t take advantage of information gained during the trial. Every time you display a choice you get information, such as a click, a purchase, or an indifferent user leaving your site. This information is really valuable, and you could make use of it in your test, but A/B testing simply discards it. There are good statistical reasons to not use information gained during a trial within the A/B testing framework, but if we step outside that framework we can.

* Technically, the reason for this is that the probability of a type II error increases as the probability of a type I error decreases. We control the probability of a type I error with the p-value, and this is typically set low. So if we drop option B when the test is not significant we have a high probability of making a type II error.

Inflexible

The A/B testing setup is very rigid. You can’t add new choices to the test, so you can’t, say, test the best news item to display on the front page of a site. You can’t dynamically adjust what you display based on information you have about the user — say, what they purchased last time they visited. You also can’t easily test more than two choices.

Workflow

To setup an A/B experiment you need to choose the significance level and the number of trials. These choices are often arbitrary, but they can have a major impact on results. You then need to monitor the experiment and, when it concludes, implement the results. There are a lot of manual steps in this workflow.

Make out like a Bandit

Algorithms for solving the so-called bandit problem address all the problems with A/B testing. To summarise, they give optimal results (to within constant factors), they are very flexible, and they have a fire-and-forget workflow.

So, what is the bandit problem? You have a set of choices you can make. On the web these could be different images to display, or different wordings for a button, and so on. Each time you make a choice you get a reward. For example, you might get a reward of 1 if a button is clicked, and reward of 0 otherwise. Your goal is to maximise your total reward over time. This clearly fits the content optimisation problem.

The bandit problem has been studied for over 50 years, but only in the last ten years have practical algorithms been developed. We studied one such paper in UU. The particular details of the algorithm we studied are not important (if you are interested, read the paper – it’s very simple); here I want to focus on the general principles of bandit algorithms.

The first point is that the bandit problem explicitly includes the idea that we make use of information as it arrives. This leads to what is called the exploration-exploitation dilemma: do we try many different choices to gain a better estimate of their reward (exploration) or try the choices that have worked well in the past (exploitation)?

The performance of an algorithm is typically measured by its regret, which is the average difference between its actual performance and the best possible performance. It has been shown that the best possible regret increases logarithmically with the number of choices made, and modern bandit algorithms are optimal (see the UU paper, for instance).

Bandit algorithms are very flexible. They can deal with as many choices as necessary. Variants of the basic algorithms can handle addition and removal of choices, selection of the best k choices, and exploitation of information known about the visitor.

Bandits are also simple to use. Many of the algorithms have no parameters to set, and unlike A/B testing there is no need to monitor them — they will continue working indefinitely.

Finally, we know bandits work on the web, as much of the current research on them is coming out of GoogleMicrosoftYahoo!, and other big Internet companies.

So there you have it. Stop wasting time on A/B testing and make out like a bandit!

Join Our Merry Band

Finally, you probably won’t be surprised to hear we are developing a content optimisation system based on bandit algorithms. I am giving a talk on this at the Multipack Show and Tell in Birmingham this Saturday.

We are currently building a prototype, and are looking for people to help us evaluate it. If you want more information, or would like to get involved, get in touch and we’ll let you know when we’re ready to go.

Update: In case you missed it at the top, Myna is our content optimisation system based on bandit algorithms and we’re accepting beta users right now!

Posted in Business, Code, Design, Front page, General, Myna, Web development | No Comments »

10 Jan 2011

by Noel

The University of Untyped

We’ve recently started a reading group at Untyped. As consultants we need to maintain our expertise, so every Friday we tackle something new for a few hours. Given our love of Universities (we average three degrees per Untypist) and our even greater love of grandiose corporate training (hello, Hamburger University!) we have named this program Untyped University.

Broadly, we’re covering the business of the web and the business of building the web. The online business is, from certain angles, quite simple. The vast majority of businesses can be viewed as a big pipeline, sucking in visitors from the Internet-at-large, presenting some message to the user, and then hoping they click “Buy”. At each stage of the pipeline people drop out. They drop out right at the beginning if the site isn’t ranked high enough on search terms or has poorly targetted ads. They abandon the website if the design is wrong, or the site is slow, or the offer isn’t targeted correctly. Each step of this pipeline has tools and techniques that can be used to retain users, which we’ll be covering. The flipside of this is the pipeline that delivers the site, starting with data stores, going through application servers, and finishing at the browser or other client interface. Here we’ll be looking at the technologies and patterns for building great sites.

So far we’ve run a couple of sessions. The first covered bandit algorithms, and the second Amazon’s Dynamo. We’ll blog about these soon. We’ve started a Mendeley group to store our reading (though not everything we cover in future will be in published form.) Do join in if it takes your fancy!

Posted in Business, Front page, General, Web development | No Comments »

20 Apr 2010

by Noel

Formalising Bonds with the Informal

There is an interesting move underway by the US Securities and Exchange Commission (SEC) to more precisely define the meaning of certain asset backed securities (like the now infamous mortgage-backed securities that were central to the recent crash). The NY times has covered the story from a high level, but what of particular interest to me is the proposal to specify the meaning of the bonds in Python. This is a step is the right direction but Python is not the answer.

The core problem here is to give a clear and unambiguous meaning to a bond. This requires the language in which it is written is precisely defined. Python is not precisely defined. There is only a prose definition of the language, which is inadequate in the same way that the prose definitions of bonds are inadequate, and of course there are differences between various versions and implementation of Python. Since Python is not precisely defined the only meaning one can give to a program in Python is whatever the particular implementation one uses does with that program.

In contrast there are languages that are formally defined, suchStandard ML and Scheme. These would make a sound basis for the formal definition of bonds. In turns out that functional languages also make a good (meaning expressive and convenient) basis for the formal definition of bonds. There is a great paper on expressing contracts in Haskell and at least one company has implemented this idea in a commercial system (in O’Caml, I believe). So my advice to the SEC: use an appropriate subset of Scheme or Standard ML, or hire someone to create a formally defined DSL, but don’t use a language without a formal definition if precision is your goal.

Posted in Business, Code, General | No Comments »

2 Feb 2010

by Noel

Is the iPad the beginning of the end for Intel?

Much has been written about the iPad since its launch. I’m sympathetic to the concerns about the closed nature of the iPad, and I think the iBookStore (along with the Kindle) is going to have a big effect on the book market, but I want to focus on something I haven’t seen much discussed: the A4 chip powering the iPad.

What you need to know about the A4 is this: at its core is an ARMCortex-A9 MP CPU. ARM cores also power the iPhone and about every other smartphone out there. Intel just can’t compete in this market as their chips require too much power. This weakness is, paradoxically, a result of their greatest strength: the Intel instruction set. Even the most modern Intel chip still retains the ability to execute code for the ancient 8086. This ensures you can run just about any program ever written for an Intel machine on the latest CPU, giving Intel an enormous software base to leverage. However supporting this instruction set comes at a cost. The 8086 instruction set is not a good fit for modern CPU designs, and the instruction set has accreted decades of cruft to try and wedge modern features into it. To get acceptable performance all modern Intel chips have vast amounts of silicon devoted to instruction decoding; that is, the process of turning instructions into so-called micro-ops, which are what the CPU actually executes. All this silicon takes power, which is why ARM handily beats Intel on performance-per-Watt.

Now this wasn’t an issue for Intel even a few years ago. But we’re seeing three things that ought to make them worried. The first is the iPad, showing that little devices can grow bigger and perhaps move into the PC market, in much the same way PCs took over from the scientific workstations of Apollo, DEC, Sun and others. The second is the increasing concern for performance-per-Watt from people like Google and Amazon whose huge server farms power the major Internet services. The third is open source software, and particularly GCC’s support for just about every CPU on the market. This means the software can be easily recompiled for a new architecture. Suddenly Intel’s dominance doesn’t seem so assured.

So perhaps in a few years ARM will become the dominant architecture, rather than Intel. Apple have already shown that switching architecture (twice!) isn’t so painful. And as someone who has been writing an Intel assembler for fun I can’t say I see this as a bad thing.

Posted in General | No Comments »

3 Jan 2009

by Noel

Weblog Updated

In case you were wondering, we’ve moved our web presence to a new server (but still hosted by the most excellent Bytemark). The weblog was migrated/updated, and some things have moved around.

Our Feedburner RSS feed is the preferred way to keep up with the blog.

Posted in General | No Comments »

4 Nov 2008

by Noel

Siesta time

When I went travelling in Spain I had a siesta just about every day. There are very practical reasons for doing so: it is so damn hot in the middle of the day, and, despite being very close to the Prime Meridian, Spain is on +2GMT in summer so the evenings last forever. Another benefit of siestas: I felt great!

This little anecdote is designed to entice to view
this graphic from The Boston Globe. Within you’ll find everything you ever wanted to know about napping. Now a lot of it shouldn’t come as a surprise. Everyone knows the mid-afternoon lull (at Untyped Midlands it tends to lead to a frenzy of piano playing or drumming, for reasons I don’t understand) but few of us heed the urge to sleep. Perhaps we should. Remember to plan your naps: either get a full cycle (1.5 hours) or stop your nap after about 45 minutes. If you wake in the middle of deep sleep you’ll feel terrible.

If you have problems getting to sleep, I recommend a cat as a snoozing companion. They’re always ready for a nap and purring is very relaxing. Furthermore, a good alarm cat will stop your afternoon nap extending too close to dinner time.

Posted in General | No Comments »

10 Sep 2008

by Noel

The Spectacular Scribular

Scribular is a website and iPhone app that takes the idea offolksonomy, adds in GPS data from the iPhone, and gives you access to user contributed notes about the place where you currently are. The best part about Scribular if, like me, you don’t have an iPhone, is that it is written in PLT Scheme. It uses the LeftParen framework and you know LeftParen has to be awesome because it in turn uses lots of libraries from Untyped! Woo! Congratulations to Rob on launching. It is great to see another entrepreneur using PLT Scheme in a commercial setting.

Did I really just write Folksonomy?

Posted in General | No Comments »