All revolutionaries need a fashion line. Maybe it's a red headscarf or a black armband or a fez or maybe it's bell-bottom jeans, but there is some strange intangible need for each revolution to have it's own clothing to set it apart. The problem is that sometimes the only thing useful that comes out of the revolution IS the fashion.

Software, it seems, has also had its share of revolutions. Here are just a few (in no particular order): Client-Server, EDI, Dynamic Typing, Test Driven Development, Relational Databases, P-Code, HTML, Object Oriented Programing, CORBA, Service Oriented Architecture, XML...etc. Now only a few of these were big enough to actually achieve the status of being a revolution, most were more like movements or perhaps just trends.

Frankly, that's a lot of movements in the very short span of about 40 years and it's not even close to a complete list. Many of the so-called revolutions (like real revolutions) have actually made things worse for programmers (CASE based development for example). Others, like Relational-Databases, have really and truly made a positive difference in what we do (although not too many people can still remember the days of hierarchical databases). As a programmer with a long memory I occasionally find myself bemoaning the slow pace at which software engineering has progressed despite it's almost continuous efforts to reinvent itself. I see hundreds of thousands of programmers all over the world writing the same CRUD applications that were being written twenty years ago. It's times like these that I begin to worry that many of the software detractors may be right when they claim that software change has become nothing more than trend chasing.

Still, I am not so sanguine about our past as this. While progress has been slow it has not stopped. Recently, however, we've come to a very strange place where people address the symptoms of our software dilemmas with increasing skill and capacity while at the same time ignoring the most critical roadblocks that are keeping us back. In particular, two very artificial problems hamper the development of today's web based applications: first, HTML is wholly inadequate for a host of applications (even with DHTML and AJAX), and second relational databases have passed their prime as a persistence mechanism but there is no successor in sight.

How can you make a phone call if you can't speak?

There is a very simple reason that developing complex web applications is hard: HTML wasn't designed for complex applications. I won't tear away at HTML too much; if you've worked in it you know exactly and precisely how limited it is. If you add a bunch of DHTML and AJAX the best you can do is a reasonably sophisticated interface, but the cost to produce it is enormous. Up until recently even modest DHTML and was basically a no-no. Of course, it's not just HTML and ; it's also the notion of state in an HTTP based application. HTTP driven applications are intrinsically designed to be stateless, but many real applications actually need a substantial amount of client application state. Adding that state at the server side is possible but creates complexity that should not be necessary (and can create resource problems as well). An ideal thin client SHOULD keep UI state information to itself since it is the arbiter of that data. Of course this is really just the tip of the HTML/HTTP iceberg when it comes to application development in web land. You also get constraints on the ability of servers to communicate directly (peering) and on servers sending messages asynchronously to the client, etc. It's a hostile UI environment that we have not yet tamed adequately.

You may be tempted to ask, "So what, don't write your complex application for the web, use the older technologies." The answer is that it's not just good to be egalitarian; it's a requirement. One way or another you'll probably have to tackle the cross platform issue. Let's say, for example, that you write an in house application entirely on windows and it gets popular and everyone loves it. Now the marketing staff needs to use it but they are running Macs and also the laptop users need to use it but it's client server and IT wont let it in through the firewall and some of the users aren't allowed to have VPN access, etc. Looks like you'll need to put it up on the intranet too. Thinking about it as a software vendor, if you're selling software people may not be able to try your stuff out if they can't install it locally (IT restrictions) or if it's a complex multi-user application. You'd better put it up as a web application so people can try it out. People have gotten used to this egalitarian software model and so for multi-user applications we just expect things to work that way. Really this is a good thing. Things should be easy for the user, but that doesn't mean they need to be hard for the developer.

The matrix has you

How is it that fifteen years in to the web revolution we're just now getting used to style sheets and we can just begin to see dynamic content becoming available? One answer is that when Netscape died (or rather when Microsoft and Netscape no longer competed for mind share) all progress ended. Looking back it was practically a "pull the plug on the web" kind of moment. Eventually Microsoft stopped even pretending to try. As long as there was competition we kept getting new goodies like CSS and Cookies and IFRAMES and DOM improvements and and blah blah blah. Sure, some of these were complete hacks, but over time if progress had continued the hacks would have gotten replaced with solid alternatives. Unfortunately, the doors of innovation slammed shut when it became apparent to Microsoft that they had won (or would win). At that point any improvements were basically bug fixes and security enhancements.

Applets were one of the features that never got working before the spigot got turned off. Concept wise they were fine but try to get them working without forcing the user to download a plug in and forget it. Today, of course, you can download a reasonable plug in for most systems and get good applet support, but who can rely on people being willing (or able in corporate offices) to install the plug in? Apparently not very many people since it's pretty hard to find a mainstream web application that uses applets in a substantial way. It's that least common denominator problem again.

Can I lay all the blame for the end of web progress on Microsoft? Nope, they had (and still have) a very able ally: namely big IT departments. That's right; many of the people reading this may actually work for the enemy of the web. While Microsoft let web technologies die on the vine the instant they gained enough control to do so, many large IT departments have been actively working against them from the beginning. Of course they didn't do it on purpose, but they did it nonetheless.

Let's say there is some cool feature I need on a web site and all I have to do is download it and try it out, what do you think I'll do? I'll download it and try it out. People are curious, people like toys, people do unsafe things with their lives. IT departments are not people. People who work in IT departments get fired when bad things happen. They can't keep bad things from happening (and they know they can't) but they're willing to keep their users from doing almost anything to try to keep bad things from happening. Hence, IT in enough large organizations became all about what you can't do and not what you can. You can't install that plug-in or have local administrative privileges, or browse unapproved sites. Anything that can be prevented will be prevented. The end result is a plain vanilla looking installation that practically fell off the Microsoft applecart. Not surprisingly it isn't exactly designed to help out people building web applications.

Get up Neo

One of the amazing things about open source is that you can't kill it. A company that has 2% market share (in most markets) may as well pack it in and go home since they aren't going anywhere. Open Source projects couldn't care less. Mozilla never died, it just got better and better and better until it emerged in its Firefox incarnation as a far better browser than IE. At 10% market share it may not be the dominant browser, but there is a very real possibility that if it continues to press forward it may well become the dominant browser (a prospect that has awakened Microsoft). At very least it has created new competition in the market. The question now is, will it stop paying attention to the old standards that it already implements fully and start blazing its own trail as a standard setter?

With the exciting growth of more advanced web applications I would not be surprised to see that happen. Perhaps we will see real modal windows or multi-edit controls or tree controls or the ability to maintain client side state, or an alternative non-HTML format designed for enhanced applications. Maybe we'll see a component management framework or perhaps a simple facility that would allow client side applications access to a confined local disk space. Maybe there will be client side Ruby support or a TK implementation for web applications. Perhaps markup will be in a meta-language that lets you mix everything from wiki-syntax to HTML to pure XML to private hybrid DSL that is parsed by some meta-parser. There are a thousand things that could be done to turn the web into a more developer friendly space, and the emergence of competition makes them all possibilities just waiting for someone who feels a need to scratch one of these particular itches.

Sure, all of these things COULD happen, but will they? Here I'm not so sure. I've been doing an AJAX oriented project for the last nine months or so and it's amazing what you can do using this approach, but it's a LOT more work that writing the same application using a more traditional UI technology. For large scale applications like we have at StorePerform this adds a lot of cost to your project when compared to not using the web at all (although it's probably actually easier that not using AJAX but still using the web). There are a ton of potential applications that probably can't be done at all using this approach. For example, building a REAL word processor or spreadsheet is probably just a silly idea (I've seen AJAX versions of both of these but I wouldn't want to use them). AJAX then has two possibilities: either it reawakens our desire for something better, or it fills in as just good enough to do stuff. My hope is that people will stop feeling like they have to develop to the inadequate target platforms we have today and start re-inventing the Internet itself. The other option is that we will build upon the web and extend and extend expending thousands of person-years of effort at the edges rather than solving the problem once at the core. I hope this isn't the outcome but my psychic powers can't assure me one way or the other.

No one has ever stood up to an agent and lived

So, on the front-end we've been squished by standards that have come to a standstill, but what about the back end? Why haven't we seen tremendous progress on the server side of the equation? My answer, in a word: Oracle. Now, unlike the front end where Microsoft is directly to blame, Oracle is more of a symbol for the status quo. The one thing that almost any multi-user application has in common is that it needs persistent data (a database), and databases have changed how much since 1980? Virtually not at all! There are some cool "new" things like new index variants and materialized views and OLAP support tools but basically these go largely unused and we've still got the same SQL that we inherited from DB2 ages past. Since the big move from hierarchical to relational databases we've gone so close to nowhere that I'm not sure I could find the progress with the Hubble telescope.

Unlike the browser space, it seems like the database space has competition so why is there so little progress in basic database technology. My answer is that to make progress in this space you need to fundamentally rethink it, and that's not something that happens easily in large companies supporting mission critical data. It turns out that it costs real money to play big in the database arena.  So when Versant and Object Store and a dozen other startups went after the object database market a decade ago they ended up delivering some cool technology that almost worked but not well enough for the data you really care about. Naturally people moved on and we were left with Object Relational Mapping which (frankly) isn't nearly as good. Even object databases aren't the right answer completely, we need to rethink databases more fundamentally in a world where drive space is free, memory is abundant and huge clustered systems are easy and cheap. We need to look beyond the limitations imposed by the semi-declarative SQL language and find a way to hide the larger problem space.

Today if we want a table that manages a fully cached set of data we do it ourselves. If we want a copy on write table that is garbage collected we do it ourselves. If we need real time access to a cross-table data aggregation we do it ourselves (can you say ETL and OLAP?). If we need to do migration between versions of our application we do it ourselves. Object Relational mapping can't overcome limitations that are fundamental to the persistence model itself and so we get progress that is glacial. It's that darned least common denominator issue yet again.

While I may see light at the end of the HTML tunnel, the database problem has been around so long that I'm a little less optimistic. I fear that this may fall in to the category of software that no one can write. My suspicion is that to fix this problem someone will need to dedicate a number of years to an open source project without much in the way of apparent progress before it will come to light.

There is no spoon

If you look at these two problems together they share a common thread. They stem from people accepting the software world in which they find themselves. We accept that the web is as it will be and we shape ourselves to its design. We accept that SQL and relational databases are all that persistence is and we bend to fit it. When we do this we forget the infinite malleability of software. We ignore the fact that the constant refreshing rain of hardware progress has turned the meandering stream into the mighty Amazon. As individual programmers most of us feel (justifiably) that the problems are outside of our domain, but sooner or later someone needs to step outside of that role and start the revolution. I do not know if web 2.0 will ever mean anything or if it will remain another fad word. I do not know if I will ever be able to program persistent data without having to remember what "group by" means. I do not know, but I believe that we will find the one, and I know that if we find him he will not wear a black suit. He will wear black leather.

Most of the people reading this may never be in charge of writing standards and thus be inclined to skip this post. Don't. These rules may be for the writer, but it's up to the user to ensure they are followed. If a standard is breaking a lot of these rules it's up to the people who adopt standards to choose NOT to adopt it. Remember IBM and Sun and Microsoft may be big, but the people writing code are a lot bigger and more powerful. Maintaining intellectual integrity is the best way to keep the worst instincts of the big guys in check. In short we need not use the processed bovine nutrient they dump out just because they dump it out. So, wielding the staff of Fiat granted me by my fathers of old I hereby and henceforth declare the following five rules for software standards.

The rule of size

"Any standard that cannot be written by three people over a period of three nights with suitable caffeinated beverage support should not be a standard. There is no limit on how long it takes to figure it out, only on how long and complex its expression is."

Basically big standards suck and are useless. When people start making a standard bigger they are trying to solve more and more problems which is another way of saying that they do not clearly KNOW which problem they are trying to solve. Are there big problems out there? Yes, but it is very unlikely that trying to solve them all at once via standardization is the right answer. When a problem is really big it is probably many problems that all have something in common. Rather than spending time writing an "ultimate" standard it would be better to spend time figuring out what some of the core problems that make up the big problem are so that they can be solved one at a time.

Another reason that small is better in standards is that it deters striving for perfection. Humility is a critical aspect in building good standards. This is because standards are a form of design and trying to get everything perfect in a design before you write the code is a fruitless effort. In a standard it is suicidal because unlike a regular design which people can simply ignore, people will be FORCED to adhere to the standard. So in standards creation it's better to keep them simple and open-ended and to humbly acknowledge that we don't know what we don't know. This is YAGNI for standards.

The rule of growth

"When updating standards the total size of the standard cannot change by more than 5% per year."

We should avoid arbitrarily extending standards; this is virtually a corollary to the first rule and the same reasoning applies. Most standards could use a revision or two, but the goal should generally be clarification and not extension. It's also good to remove things that have proved confusing or that are unused (although for some reason no one ever does this). Backwards compatibility need not be a feature of a standard, standard implementors can continue to support the old features if needed even if it has been removed from the standard.

If there are good ideas that the standards need then I recommend new standards; these can be extension standards rather than extensions of the standard. The problem is that standards that are popular are powerful and as such powerful interests are inclined to manipulate them. Whatever the new version contains will have to be supported so extending standards provides opportunities for abuse.  If you assume the new version of the standard will be valuable and correctly done then extending it seems good since it will keep things moving forwad.  Unfortunately, even if we assume people writing standards do so with the most altruistic and pure motives, standards are like untested code; until they've been in place for a while it's a bit hard to know if they are good or not.

Extension standards (unlike extensions TO standards) need to earn their place in the market and taking those efforts out of the initial standards effort provides small and large companies opportunities to disagree and propose competing alternatives. In short it takes the power away from a small group of potentially powerful interest groups and places it back in the hands of the standard users (coders).

The rule of consensus

"Excessive consensus building is to be discouraged.  If two or more people (or organizations) discussing a standard are in strong disagreement let them go their own ways and write different competing standards."

This rule flows from my observation that some of the worst standards ever written are those that are the results of consensus building between standards bodies with competing perspectives. Consensus frequently means everyone agreeing to do what is wrong so they can smugly proclaim that the resulting failure was due to the other guy's bad ideas. A standard should have a relatively clean and simple vision if it does not then it will fail.

People look at HD-DVD and Blu-Ray and say, "see how expensive it is to have competing standards, we need to always look for consensus so people don't waste time and money." This may be true for hardware but it is much less true for software. The percentage of hardware standards that prove workable is extremely high, the percentage of software standards that hold up over time is extremely low. In an environment where failure is a high probability event a diversity of options is an extremely sensible alternative to putting all one's eggs in one basket.

A lot of the time I hear people grousing about how the fact that people aren't adopting standard X or Y is a horrible thing and how the world would be a better place if people would just do things the "right" way. To me this is verbal shorthand for "I'm right and if you all just get in line my life will be easier." I've felt similarly and expressed this opinion from time to time as well, but my perspective is just a little more egalitarian. Sure I think JSON is better than XML and I'll tell you why, but at the end of the day it's your call if you want to go with XML for everything from properties files to language extensions to communications. I'll even talk to your system with it if I need to. Just don't be surprised if I also choose NOT to use your great XML alternative if I think it's a pain in the hind-section.

The rule of frameworks

"Frameworks may not be standards. Ever."

This rule is harder to define clearly, but in essence it is an argument against horizontally intertwined standards AND an argument that no standard should ever position itself as the owner of your program space. Consider the IP to TCP to HTTP chain as an example of a properly related set of standards. It's very possible to use TCP without explicit and detailed knowledge of IP but you couldn't write TCP without that knowledge. Similarly you can use HTTP without too much knowledge of TCP and with virtually no understanding of IP at all and this is all good. This is a nice vertical stack.

A standard becomes a framework standard when it takes four or five standards that should (or perhaps do) stand on their own and puts them together in such a way that you need to UNDERSTAND all of them in order to use the standard.. So if, for example, you need to understand JNDI in order to obtain a database connection from JTA there is a problem. If JTA simply used JNDI without requiring your intervention, use, or understanding then it would be better. If instead it could work with any number of alternative services without your knowledge or concern for which one the implementor chose it would be best. When possible the best solution is for a standard not even to sit in a technology stack at all, but to require the implementors to pick and use an appropriate stack (or stacks); this keeps popular standard from forcing people to use other weaker ones and it lets vendors actually compete by working with various stacks. Actually JDBC and ODBC are examples of this approach. They don't mandate that any particular kinds of wire protocols are used, they let the vendors work that out for themselves. Heck, neither one even mandates much about SQL. This rule is not absolute, there are plenty of standards that have become so dominant that presuming their presence is fine. TCP and IP have this quality, it's OK to presume them these days but most other standards are not so easy to presume.

The other cardinal sin of a framework standards is to tell you how your program should run. In Java or C this is taking away your main. When the standard dictates so much about your program that your program starts to look like an adjunct to the standard and not the other way around it has definitely gone in the wrong direction and is to be avoided. J2EE did this in spades. In fact I view App Servers as a whole as an unfortunate turn in the wrong direction. The same services could easily be provided as tools but instead you get an entire massive system whether you need it or not. Fortunately there are plenty of non-standard tools these days so we can all happily go back to ignoring the J2EE non-standard but Java Land wasted a good three years or more thanks to some very awful choices and almost an entire generation of engineers got to learn how not to do things before re-learning how to do them properly.

I should point out that I am not against frameworks. Frameworks can be a great thing and to the Spring/Rails/Tapestries of the word I say "go for it" with only the mild caveat that if you can produce independent components (as Spring does for example) that's double plus good. But anything as big as a framework is too big to be a standard. Way WAY WAY too big.

The rule of utility

"It's not a standard if it doesn't help you solve your problems."

This isn't really a legit standard for standards, but it's very important in how we view them. Do you need SOAP? Maybe, if it helps your program meet its goals; if it doesn't then it's a non-standard from your perspective and your perspective is the only one that matters. Think TCP sucks and you want to use IP directly that's awesome; do your thing. You're a big girl and you can decide who you want to take to the prom without mommy and daddy corporate entity picking your prom dress and telling you how late to stay up. Of course if you find out that you need guaranteed delivery and you waste nine months making your code work then the baby is yours and you'll be the one taking responsibility for it. Of course, as it turns out, the responsibility is yours either way.

Where do you get the right

By now some people may be wondering where I get the right to decide what should and should not qualify as a standard. Actually the real question is where does anyone (ANSI, W3C, AdHoc committee no. 481K-23) get the right? They get it from you. In the end standard adoption is a democratic process (or at least a capitalistic one) and all that committees can do is to offer up candidates for us to vote on. There aren't any rules about who can propose candidates and there aren't any rules that govern who you get to choose. My standard of standards is meaningless unless it makes sense to you and that means it's just as valid as T.120 or XSLT. All I'm doing here is pointing out that you do get to vote and like in all democracies your vote makes a difference and affects other people. Use your vote wisely and I shall try to do the same with mine.