One that wont go away (under load), one that doesnt keep me up all night (worrying about scaling), one that wont make me sleep all day (when I should be adding features). If you have tried to build an Internet site or application recently that needs to work for thousands or tens of thousands of concurrent users you may share this desire.
Why is this still so hard? Why do we find ourselves worrying about locating experts in the dark art of database performance tuning? Why are we spending time haggling with Rackspace over the price for another set of servers or racking our own servers at Equinix? Why are we writing our own user management (from scratch)?
He goes on to talk about the inadequacy of the tools we're all using.
Because we are using tools that simply were not made for the job. Relational databases have gotten faster and better, but their fundamental construct of data stored in rows and neatly parceled out across tables (which frequently need to be joined) does not really match up with either rapid development using some notion of objects or with scaling horizontally using commodity hardware. Web servers started out by simply providing static content and were then forced into running applications with the result of not doing either particularly well. Yes, people have been writing new ones to address that, but even those are fundamentally designed to work on a single machine. The second you go beyond one machine you need separate load balancers, reverse proxies, caches and all sorts of other paraphernalia just to make stuff work together. Worse yet, the web server, the application and the database are connected to each other via thin straws that were bolted on after the fact with important information either not passed at all or only painfully (e.g., information about which user is requesting a particular set of data).
The post goes on to talk about a lot of specific problems that everyone runs into sooner or later. It's a really good read, including the comments.
Our question for you, the developers looking to Yahoo! for technology, is this: What should Yahoo's role be in solving these problems? What high- or low-level services would help to change the game in building the next generation of on-line applications?
Yahoo! Developer Network