Read on and leave a comment
or return to our portfolio.

Archive for July, 2010

22 Jul 2010

by Noel

Epistemology and A/B Testing

A/B testing is all the rage in certain web development circles. Naturally, when something becomes popular the criticism starts. I’ve read some unconvincing attacks on A/B testing recently, as well as some good ones, so I want to lay down my thoughts on what A/B testing is and what it isn’t.

The general method of A/B testing on the web is as follows:

  • Decide on a change to make to the site. This could be as small as the wording of a title or as large as the entire navigational structure of the site.
  • Decide what outcome you want to measure. Typical examples are purchases, time spent on the site, and number of repeat visits.
  • Randomly assign each visitor one of the two (or more) versions of the site.
  • Measure how the different versions stack up against the outcome of interest.

This is a fairly simple thing. Critics of A/B testing usually claim that it is only good for small changes. It cannot, they claim, be used for business-changing disruptive innovation. The critics are wrong. They are confusing the principles underlying A/B testing with the commons implementations of the idea.

How We Acquire Knowledge

There are basically three means by which we come to acquire knowledge:

  1. By appealing to authority.
  2. By constructing statements consistent with assumed first principles.
  3. By making observations on the effects of actions.

The third method has proven to be vastly superior when studying the natural world, and is the basis of the method known as science. If you are reading this then you are validating the efficacy of this method, as the computer you are using is the result of a few hundred years of scientific developments.

The primary mechanism of science is the experiment. An experiment involves performing some action in the world and measuring it’s effect. If different actions leads to different outcomes one typically does some statistical analysis on the result, to determine if one is justified in believing the differences represent a true difference or are just the result of chance.

A/B Is Science

A/B testing is science. A/B testing is about taking an action and measuring its effects. That is, doing an experiment. One can experiment with small things, like the colour of a button on a web site. One can also experiment with large things, like business models, new technology, and other disruptive changes.

The critics see the small experiments used to market A/B testing to internet businesses and think it is the totality of the method. They are right that companies usually don’t A/B test large changes. It is unusual to run two or more different business models, for example. That doesn’t mean these experiments aren’t done, but they are typically done at the level of the market rather than the individual company. Different companies, called competitors, experiment with a particular combination of strategy, model, and implementation, and the market measures their effect. Sometimes big companies will run these experiments internally. Google, for example, is currently experimenting with both Android and Chrome OS in more or less the same space. Complex experiments like this aren’t controllable nor are they repeatable, so the methods of social science are preferred over those of the hard sciences, but they still fall within the scientific paradigm.

A/B Testing Isn’t All That

I’ve said A/B testing is science, and science is great. However I do think the current implementation of A/B testing, as used by web companies, is flawed. The reason is we’re usually interested in decision making not hypothesis testing, and with decision making we want a different setup than is currently used. Exploring this is for another post.

Posted in Business, Front page | Comments Off on Epistemology and A/B Testing

20 Jul 2010

by Noel

Birmingham Events

Birmingham doesn’t have great visibility at the intersection of software development, design, and entrepreneurship in which Untyped operates, but in the last few months there has been a surge of events that suggest this is changing.

Here’s a list of regular events, some new and some established, that we’ve found of interest:

  • fizzPOP is a hackerspace that runs fortnightly meetings. The next meeting, which will be my first, is tomorrow!
  • Tech Wednesday is a networking group for computing professionals. It has only been running for a month. The next meeting also tomorrow, and it will also be my first.
  • The first meeting of StartupMill Birmingham was yesterday. I could only make the beginning of the meeting, but Dave tells me there was an interesting group of people.
  • Likemind is a nice relaxed networking event held in the Jewellry Quarter. There is a good mix of people there representing all sorts of nearby businesses.
  • Digital Playground attracts a tonne of web and graphic design people from Fazeley Studios and surrounds.

Posted in Business | Comments Off on Birmingham Events

18 Jul 2010

by Noel

Open source libraries now on GitHub

We are excited to announce that we’ve moved all of our open source code to Github!

If you want to use our libraries in your application, we recommend getting hold of them via PLaneT as usual. If you want to live on the edge or hack on things, however, you’ll find the libraries here and build instructions here.

We’re still working on the best way of organising and documenting everything. If you have any advice, please get in touch or leave us a note in the comments.

Posted in Code | Comments Off on Open source libraries now on GitHub