IT projects are notorious for their failure to live up to expectations. In my experience the causes are many and varied, some of which are outlined below:

Goal -     goal unclear -     goal changes -     varying interpretations -     goal too ambitious	Planning -     insufficient planning -     too much planning -     unfounded optimism -     no learning from experience	Resources -     insufficient resources -     inadequate resources -     too many resources -     unstable resources
Methodology -     lack of methodology -     too much methodology -     inappropriate methodology	Technology -     lack of understanding -     poor design -     hidden complexity	Risk -     lack of risk analysis -     inadequate contingency -     wishful thinking/groupthink
Culture -     quality -     service -     values	Tracking -     failure to heed warnings -     managing expectations -     change control	Structure -     poor structure -     poor process
People -     motivation -     delegation -     teamwork	Communication -     poor feedback -     inappropriate channels	Contracts -     win/lose contracting -     poor negotiation/conflict resolution

Many of these are interrelated and serve to compound each other. There is no such thing as a perfect project, but improvements in any of these areas will increase the likelihood of success. I will address each in some detail in a later article. There is an underlying issue however, that I would like to discuss first. My experience is that the way we perceive IT projects greatly impacts the way we approach their planning and implementation them and therefore how they unfold. So I would like to examine the view of IT as a form of "engineering".

Software and Engineering

Engineering implies building a well-defined item from clear architectural plans. Let's take as an example building a motorway. We survey the proposed route and generate a project plan that usually is not too far from the eventual reality. There will be some surprises along the project but fundamental changes are rare. With a bit of experience in building motorways we can estimate contingency that will likely address most risks. So on average we can achieve a quite good success rate in terms of time, cost and quality. Now imagine planning a motorway in which the destination is not quite clear and may even change while we are building it. We may encounter unforeseen obstacles such as rivers, mountains and towns on the route. One wouldn't be surprised at huge cost overruns and abandoned half-built roads. And this is similar to what happens in many technology projects.

The are many reasons why IT is trickier than building a motorway.

Such levels of uncertainty and complexity are challenging to cope with, so we are tempted is to act as if software is a straightforward mechanical process . But underestimation of the complexities involved can have serious consequences:

Unfortunately there are no quick and easy answers to these complexities. If there were, they would have been already identified and we would have a sound methodology that we could depend on. But the good news is that just being aware of the issues and facing up to them helps us to avoid the worst pitfalls.

Lessons from Open Source

Open Source has demonstrated that it is possible to produce high quality products that users love in a timely fashion. It is becoming increasingly evident that the Open Source community is able to deliver complex IT projects better than most, if not all, commercial organisations. So what can we learn from Open Source?

Everybody can view it. Peer review is a vital element of all scientific disciplines. Imagine a new drug being made available to the public without any independent verification of its benefits and possible side-effects. Many eyes looking at Open Source code ensures that overly complex or problematic implementation gets sorted out quickly. Intellectual Property concerns of most businesses preclude them from making the inner workings of their technology open to scrutiny, so product quality and quality improvement are usually far inferior to Open Source equivalents.

The model is iterative and highly interactive. The idea is to get customers involved from as early as possible and to refine and add features in response to feedback from the market. New updates are regularly provided so feedback between producers and users is efficient. This results in higher quality and responsiveness and in products that address people's needs - because the users have a big influence throughout the process. By contrast, traditional software products are released at long intervals with the result that customers have less influence in what is delivered, when it gets done and what problems get fixed.

The developers are extremely smart. Developing complex technology is hard. "Natural selection" by peers and the user community weeds out inferior technology. By contrast, most managers in commercial enterprise cannot tell which developers are smart. Adding more people to projects in jeopardy often just compounds existing problems. In technology projects a small team of highly talented people will outperform a large group of competent people most of the time.

The developers are highly committed. Open Source is a community of people who share common interests and often common values. They do what they do because it makes them happy. They work on projects that interest them. This is not often the case in commercial organisations where the common interest, it any exists, is likely to be financial.

The developers take pride in what they create. They encourage feedback and criticism because they want to produce the best possible product. They take responsibility for the quality of what they produce. This is a far cry from many companies where quality seems to be divorced from development, hived off to a separate QA department, who typically don't have the power or expertise to do anything much about improving quality.

Developers work when and how they want to. They don't feel pressure to "look busy" when they are thinking. They take a break when they want to. They mostly work from home, don't have to commute, to attend meetings which don't interest them, or to be in the office for fixed hours. But when they work they are extremely productive. The engineering model can have the effect of stifling creativity, productivity, quality and commitment.

Open Source is about serving people's needs. It doesn't have to try very hard to sell its products or convince users of its benefits. This removes one of the biggest structural tensions in commercial technology organisations - The Sales Versus Engineering Divide.

Smart organisations can learn from Open Source and have the opportunity to reproduce some of the conditions that have made it so satisfying for producers and users alike. But this can challenge many of our assumptions about how our organisations should be structured and operate.