At the Experimental Gameplay Project at CMU the participants were challenged to create a game about a particular theme every week. The results are impressive, but more interesting are the insightsderived from observing the process. It’s not about games. It’s about using extreme resource constraints to force innovation. It’s about accepting failure as a necessary part of creativity. It should be read by anyone who wants to turn a big idea into reality.
In chatting with Nick about hosting, I began to realize how complex the ISP game can be. These are some thoughts, all provide starting points for further brainstorming and discussion.
We’re building our new machine for ourselves, make no mistake. We took this route because it looked cost effective for the needs we had for ourselves. However, we can, in theory, provide services to others as well—as our server is more than capable of supporting more than just untyped.
But what services would we offer, and at what cost?
We currently run Apache, Exim/Courier, Postgres and MySQL, Subversion… in general, a full assortment of services that users might want to make use of. Through judicious use of LDAP, we’re hoping to expose some of these services (email, WWW) without needing to provide full (shell) accounts on the server. Likewise, we’re planning and building scripts to automate many of our maintenance tasks on the machine; we’re stopping short of CPanel, however.
So if we have the means to support other users, why would I be hesitant? Because we are not an ISP. We are people who enjoy making software. Our machine is like our workbench—it provides the essential services we need to make that software. Therefore, we have no interest in trying to compete with your run-of-the-mill ISPs… you can get a $5/month web account anywhere. So what kind of hosting will we provide?
At the least, I imagine we might offer hosting for “odd” servers and languages. By “odd”, I mean “anything not Apache.” For example, we know that there might be some interest in the PLT Scheme; likewise, I imagine that allowing servers written in Erlang may be of interest as well. Beyond that… how much bandwidth can someone expect to get hosting with us? Do we provide backup services? What kind of quota does a user’s account come with? What kind of response time to user requests are we going to try and make good on?
We will probably have 50GB of traffic per month in our colo agreement. We will be charged beyond that as per Bytemark’s policies. Therefore, I could simply pass on costs to clients. Really, though, we’re not prepared to deal with the administrative overhead of billing clients for bandwidth, etc.; we’ll barely be in a position to administer billing for basic services. Again, we’re back to the mantra “we are not an ISP.”
We’ll have bandwidth accounting in place; how you charge for bandwidth, I don’t know. Just having to administer this takes time, and time is money (“We are not an ISP…”). Perhaps there’s a simple way to handle this… perhaps not. I need to think more on this one; insights are welcome.
Backup is expensive. It takes careful scripting on the host, regular movement of data from the server to a remote site (see “Bandwidth,” above), and it takes time to archive that to some kind of storage media. I don’t have a tape jukebox anywhere near the server, nor can I afford one (“We are not an ISP” ). Therefore, backup is an issue of me having a second machine that runs a cron job, and regularly pulls backups off the server. Then, I’ll have to manually burn DVDs and/or make digital tapes archives of that data. There are time and media costs involved in this that are non-trivial.
Backup is something that typically is not offered with a shell account, period. Taking a look at Mythic Beasts, their shell accounts start at £15 per quarter, they offer 200MB of disk space, and there is no mention of backup. I found some backup services while investigating storage (below), and they charge a lot per month for 1GB of backup (£25). This says to me that most hosting facilities assume the following:
I’m not committed to either route at this point, but it’s clear that I should charge money for backup services. I’m willing to spend my time to guarantee my own data’s safety, but I’m not willing to do that for others, for free.
What kind of quotas should we provide users? What can I get shopping around the internet?
| Site | Capacity | Monthly |
| storegate.com | 50MB | £1.50 |
| box.net | 1GB | £1.70 |
| box.net | 5GB | £4.50 |
| storegate.com | 1GB | £5.00 |
| xdrive.com | 5GB | £5.60 |
| ibackup.com | 5GB | £5.60 |
| www.datafort.co.uk | 1GB (compressed) | £25 |
Epinions has more commentary on online storage than I care to duplicate here. I myself have a 1GB Box.net account, and am quite pleased with it—their pricing is competitive, and the interface to the site, as well as features (instant photo albums, passwords, RSS feeds) are great.
Point being? If I’m going to offer space, I have to charge at least £1.75/GB; and since we’re not really in this market space to compete, I’d probably have to charge more than that (due to the administrative overheads, and limited resources on our own machine). Furthermore, that’s assuming it’s a gigabyte of unarchived storage. It looks like sites that actually offer backup of a gigabyte of data charge around £25/month for a gigabyte of content that gets archived monthly. This doesn’t surprise me in the least—offering security is an expensive proposition, both in terms of tools and in terms of time.
Again, comparing, Mythic Beasts charge £30 per month for a “developer” account with 1GB of disk (no backup). The reality is (because we’re not an ISP), we only have one server, with limited disk space. I can resell storage from Bytemark, which eases the stress on my own disks… but as can be seen, having active disk space that you can serve out to the world costs money.
All these services take time to administer. Yes, we have to do it for ourselves… but it’s another thing entirely to have customers who want guarantees, and support when things go wrong (regardless of whether it is at our end or their end). What does administration cost? I can look at other ISPs and see what they charge for adding a subdomain, or configuring databases, or any of a host of things… and I don’t want to get trapped in a situation where we offer to do these things “for free,” and end up playing sysadmin to a bunch of customers without any recompense.
Again, this is a tough one.
This isn’t necessarily the kind of laundry a small business should air in public. Typically, this is the kind of uncertainty that would be kept private, resolved off-line, and then published to the world as policy. However, these are probably the kind of questions that others have wrestled with in their time, and it’s possible someone who reads untyping may have some insights into these issues that we don’t have. Given that we only have one machine (“We are not an ISP”), it may be that we should avoid hosting altogether. Or perhaps we can come up with some reasonable terms of service and pricing structures that reflect the quality and nature of services we will offer. Who knows? Either way, they’re challenging questions.
As we’ve been hinting on the blog our current project has been close to delivery. Truth is, once we got a new server set up the deadline passed without incident. We’ve been waiting a few days to ensure there are no problems, but it’s been long enough so we’re now happy to announce the project is complete.
This work has been done for the School of Biological and Chemical Sciences at Queen Mary University of London. We have implemented a preregistration system for them: a website by which students can be registered for courses and modules, replacing a tedious and error-prone manual system. Unfortunately there is no publically available front-end so we can’t show off our work. However we have a few projects that should be completed in the next months that will have publically visible components.
The whole site runs on PLT Scheme, with SQLite for the database. This is, to our knowledge, the first large site to run the PLT Scheme web server continuously for any length of time. Over the next few days we’ll provide a few more details about the more intricate parts of our setup so other intrepid pioneers can learn from our work.
Currently, untyped lives on a virtual machine provided by Bytemark. The service they provide is excellent, and we recommend them highly. While we are building our own server we hope to continue to host with Bytemark, but instead of being on a VM, we’ll be on our own hardware. We’ll see how that goes.
Today, Christian and I spent a few hours working on the most fundamental part of a server: the filesystem. Before I go into any detail about the decisions we made, I’ll give you a sense for what we’re working with:
| Untyped’s new home | |
| Chassis | Intel SR1200 |
| Processors | 2x PIII 1.4GHz |
| RAM | 2GB 133MHz ECC |
| Hard disks | 2x 250GB IDE |
This server has seen use before, but we’ve replaced all the moving parts; we’re quite pleased with its condition, and think it will provide us with a number of years of good service. And, we hope that we don’t need to do a low-level install again in the next few years.
The first thing we did was to grab a Debian 3.1r1 “testing” net installCD image. We had to boot the 2.4 kernel, as the 2.6 kernel fails to load appropriate CD drivers from the install CD; this didn’t really matter. Then, we came around to our filesystem layout; what we knew was that we wanted to partition off different parts of the directory tree (/, /boot, /var, /usr, /tmp, /home); we didn’t know exactly how much space to give each part of the filesystem, however. Do we makehome 40GB, or 60GB? What about usr? The list goes on and on.
We started by setting up a 4GB swap partition at the end of each drive, a 400MB boot partition at the front of each drive, and setting aside the remaining 245GB or so for the main parts of the filesystem. We then used the Debian installer to turn the 400MB partitions into one RAID set, and the 245GB partition into another. This way, both our boot partition and the main part of the drive are in RAID, but we’re guaranteed that our boot partition is at the front of the disk.
(What is RAID? I’ll stick with the Wikipedia on that one. It keeps our two disk drives in perfect sync; this way, if one of them fails, we might be able to replace it before the second one does, thus keeping our system running with little or no interruption. This is a Good Thing.)
Then, we dove into that big, 245GB space. We used LVM—the Logical Volume Manager–to partition the rest of the disk. LVM is great because it essentially ignores the physical layout of your disks, and allows you to dynamically resize partitions without any great gnashing of teeth. So, we laid out 40GB for /home and /data, 20GB for /usr, and 4GB each for /, /var, and /tmp.
Note that this is only 112GB of space; we had (roughly) 245GB of space in that big RAID set. Our intention is to grow those partitions as we need to. For now, we chose large, but reasonable values for each of these partitions. In time, we may decide to increase the amount of space allocated to /home—perhaps from 40GB to 80GB. The point is, we have around 116GB of space to “grow into”, and we can allocate it to any of the partitions we currently have… or we can create new partitions. In either case, these operations don’t require us to shutdown the machine, or even take it entirely offline—we only need to “stop” the logical volume that the partition is on.
While we could do this more quickly the second time around, this took us between one and two hours; we were careful to check our assumptions, and rehashed and discussed a lot of the decisions in light of how we might want to use the server in the future.
Once the filesystem was set up, the rest of the installation went quickly; things were pulled in automatically over the network, and we rebooted into our new machine. The filesystem looks great, and we expect that the decisions we made will serve us well in the future. Now, we’ll work on upgrading the kernel to a 2.6 series (with SMP support), and then begin replicating those services that currently exist on the VM to the server.
Links that came in handy:
If you are a civilian like me (an anthropologist, that is, without much technical savvy), it’s a little mystifying. Will Internet 2.0 change everything or just some things? Is this a revolutionary moment or an evolutionary one? Is everything I know wrong, or just this, that and the other thing? Is this 2.0h! or merely 2.0. …
Here are three models that sort out the possibilities for me.
Read it, it’s good stuff.
My take: these changes will take place, but it will take another generation, that grows up with the Internet, for them to be fully realised.
This post is one in a continuing series where we internally debate the merits of the over-hyped promise of AJAX…
Debugging Ajax Requests in Prototype:
How do we debug our Ajax applications? The Rails Weenie has taken the Ajax Responder feature in Prototype…
I’d like to remind Noel that the distance of the Atlantic and business of the Christmas holidays are not going to keep me from pointing out that AJAX is an immature and dangerous platform to build a business on. Yes, I’m glad that GMail is there… and I suppose the Yahoo! Mail beta. However, these are fragile technologies to build upon.
How do we debug things written using the Prototype framework?printf. I mean, I’m glad that with AJAX and Rails I can whip something up quickly that “just works.” However, it doesn’t “just have test cases,” or “just get internationalized”, or “just stand up under load.” They’re rapid-prototyping tools, certainly nothing more. AJAX breaks usability standards, pushes data and computation to an unreliable substrate (the client’s web browser), and as the post above provides some evidence for, there are no good debugging or tracing tools available for developers working in heterogeneous browser environments in Javascript.
So, Noel, riddle me this: why should developers be willing to take eight steps backwards and be shafted with printf as their primary debugging tool when working with AJAX?
A cartogram of the world by population gives some perspective. This green and pleasant isle really is quite crowded. Canada, on the other hand, is somewhat devoid of human inhabitants.
A great big Happy Birthday! to Geoff, who turns 21 again today!
A recent study of human skulls covering a period of about 600 years shows the human skull has changed significantly. Facial features, particularly the lower jaw, have decreased markedly in size, and more intriguingly the size of the cranial vault has increased “by more than a centimetre” (presumably they mean a cubic centimetre). What does this mean? I don’t think anyone knows, but I’d guess the increase in processed foods means less need for strong jaws, and the increased brain size might explain why IQ scores are increasing over time.
Update: The increase in IQ over time is known as the Flynn effect. There are some good links here. Cranial capacity is proportional to body size, as discussed here. I haven’t been able to find the publications of the original study, so I can’t tell if they controlled for increases in body size. If not, then the increase in cranial capacity is likely not significant, as people today are generally larger than in the Middle Ages.
From The Scheme Way is a link to Frag, a first-person shooter programmed in Haskell. Neat!
