July 07, 2008, 7:26 AM — Only the cloud can scale
Step a little bit into the future. Not too far. Just a little bit. A handful of years perhaps. A group of web application developers get together in a bar to discuss their current projects...
"We are building a new .com website. We are hosting it ourselves. Doing our own storage layer and our own replication/fault tolerance, managing our own operating system instances, the works.", says one of their number.
"Oh that is so cute!", says another. "I'd love to be working on something retro like that! Yup. Those where the good old days of web development..."
Retro? Cute? Que pasa?
Step a little bit into the past. Not too far. Just a little bit. A handful of decades perhaps. A group of aviators get together in a bar to discuss their current projects...
"We are building a new aeroplane. It has two engines for fail-over. 6 seats. Two thousand mile range. We are building some landing strips for it too, complete with wind speed monitors!"
I suspect the history of aviation and the history of web applications will look somewhat similar at some point in the future. The early days of aviation were full of one-off aircraft and ad-hoc infrastructure. Over time, it became apparent that in order for aviation to go mainstream, massive infrastructure would be required at massive cost. Only countries could afford to own planes and run airlines and airports.
The early days of web applications - the current era - look similar to me. Lots of "one-off aircraft" in the form of bespoke application stacks. Lots of "ad-hoc infrastructure" in the form of DIY load balancing and redundancy and fail-over and so on. However, it is becoming apparent that in order for a modern web application to go mainstream, massive infrastructure is needed.
Today, a successful website may have to cater for many thousands of hits a second. Scaling to those loads requires some serious infrastructure. Multiple data centers spread around the world. Maybe tens of thousands of worker nodes during peak usage. Ability to grow and shrink with demand....
Are you really going to try to build all that yourself? Even if you could, would your application fly - metaphorically and commercially speaking - with all that financial overhead?
I doubt it. Being a success on the Web is becoming a terrifying prospect! Overnight you might have to scale from a few hits per minute to thousands of hits per second. To be successful you need to stay upright during flash foods. You cannot predict when they will come. If your site falls over, they eyeballs move elsewhere. You only get one shot at success in the fickle world of eyeballs.
What to do? There is only one way. Rent access to huge infrastructure. The modern web is seeing the emergence of a number of infrastructure superpowers. The web-infrastructure equivalents of Boeing, of FedEx, of Amtrak, of Qantas. We can debate which ones will succeed but I don't think we can sensibly debate their necessity.
The future is shared infrastructure. The future is the cloud. Not because it is trendy but because only the cloud can scale.
Fasten your seatbelts.
