Category Archives: project failure

The Best Research On Project Management

Some really insightful data backed research on project management comes from Bent Flyvbjerg. He’s originally from Denmark, but now at Oxford University’s business school. You can see a good summary of his research here, and he has a name that’s pretty easy to find on Google.

His recent article in Harvard Business review explains why projects can be subject to the black swan problem.

And a number of his publications discuss the problems of inaccurate cost estimates, especially in public projects, but here is a good summary of reference class forecasting.

Project management is often a fairly practical subject, and so it’s interesting to see the level of deep, theoretical rigor that Flyvbjerg brings to the topic.

Winning The Lottery And Speed Skating – Outcomes vs. Process

It’s important, but hard, to separate good process from good outcomes. Often it’s assumed that any good outcome, must reflect a good process and vice versa. But in risky situations this approach could lead you to make major mistakes. Assume you win the lottery. Now, you now have a very large amount of money, but that does not change the fact that lotteries are, as Adam Smith said, “a tax on idiots” and the expected return on any lottery ticket is negative – there are much better ways to spend your money. So just because people win the lottery each week does not mean playing lottery is a good idea (unless you like losing money). So playing the lottery is a bad process, but there’s a chance you hit a good outcome. In fact, it happens every week.

I’m just using the lottery as an example to show that in many cases closer to home, we might be making the same mistake. For example, your project finished ahead of schedule, but how much of that is due to good process that can be repeated? And how much is due to luck? The answer comes down to how good your process is.

Of course, there’s a more positive side to this too, just because you didn’t get the outcome you wanted didn’t mean the process was bad. Speed skating at the Winter Olympics is a good example of this, it takes many years of dedicated training to enter the Olympics, but in a speed skating race you can easily get pushed over by a competitor, and it might be totally out of your control. It doesn’t mean you shouldn’t have won gold, but it means you didn’t win goal. Good process, bad outcome.

So what to do in situations where risk means that outcome and process aren’t totally tied together?

Two things can help:

  • Repetition – over time processes and outcomes will converge where risk is present. You might get lucky on one project, but across ten it’s far less likely. Look for multiple instances of a situation before forming a judgment.
  • Analysis – good process can be supported by analysis. If something went wrong or poorly look at why it happened. Luck can often be identified with logical analysis – a good process should make sense and be robust.
Bad Outcome Good Outcome
Good Process Changes could make things worse Ideal situation
Bad Process Process improvement needed Unsustainable luck

Visualization Of A Troubled Project

In case you haven’t seen it previously, this is a clever visualization of the problems of project management communications.

Analysis Of Government IT Project Failure

Old, but still relevant analysis of government IT project failure here on HBR. The six themes are familiar:

  • Poor governance
  • Lump sum funding
  • Waterfall development
  • Scope creep
  • Automating too early
  • Doing data cleanup too late.

It’s not a bad list, although I’m surprised to see waterfall development on there. There’s nothing inherently wrong with waterfall, and I suspect the issues relate more to scope creep rather than the methodology choice.

Project Management at the BBC

Continuing the theme of UK based projects such as Wembley and Scottish Parliament, the BBC is a state funded provider of television and radio in the UK. The UK’s National Audit Office reviewed 3 recent construction projects by the BBC, 2 of which exceed cost and budget and 1 came in below budget the full report is here.

The key findings are as follows:

  • The projects did not identify benefits clearly from the outset, making it hard to assess success objectively
  • The importance of financial contingencies – the reason the one project (Salford Quays) came under budget was because it had financial contingencies at just over 10% of the initial budget. The projects which went over budget had much smaller contingencies.
  • Change management, all projects had material changes during their lifetime, an average of 41 per project, which generally lead increased costs and in some cases could have been avoided through more thorough planning efforts.
  • There is a need to better assess skills required to execute the project relative to the organization’s existing capabilities.

Project Failure – Scottish Parliament

source: Asif Musthafa (via Flickr)

The Scottish Parliament overran initial costs by a factor of ten and was delayed by 3 years.  It is clear that similar to the Sydney Opera House the design was not finalized before construction and estimates were not backed by a credible cost estimation process. For example, here are the 5 finalists chosen from a shortlist of 12 in May 1998. Only one of these finalists adheres to the budget, every other proposal fails to meet the brief in terms of both cost and size. It is absurd to specific minimal criteria for a brief and then fail to shortlist finalists based on those criteria. When a project starts with this level of disregard for process, it is unlikely to ever get back on course.

Source: House of Commons Briefing (for reference EMBT/RMJM was selected)

From there, the project continues to unravel, for example the architect then added 4,000 square meters (+14%) to his design area. It is perhaps not surprising that the project manager resigned just over a year into the project because there was clear tension between the sponsors’ desire to have the Parliament ready as soon as possible and the architects’ desire for a “gestation period” to really flesh out his design together with a need to be engaged in all decision making. The lead project sponsor (Donald Dewar) and architect (Enric Miralles) sadly both died during the construction period, which further complicated the project.

Detailed reports on the project can be found in the Hollyrood Enquiry and Parliamentary Briefing and this article from Max Wideman.

Quality

It is worth noting that the Scottish Parliament has won many national and international architectural awards. This is similar to many delayed projects in that they fail specularly on time and budget constraints, but the quality of what is delivered can ultimately be extremely high, even if the apparent process in creating it was not.

Conclusions

The process of selecting and managing the new Scottish Parliament had no regard for credible estimates from the outset. As a result it is unsurprising that the final result bore no relation to the initial estimates. It also appears that the sponsors of the project wanted quality above all, and in that context it is not surprising that cost and time gave way to that objective.

Project Failure – Wembley Stadium

Wembley Stadium (photo: Martin Pettitte via Flickr)

Wembley stadium is the home of English football (or English soccer if you’re American) and was rebuilt in the 2000s replacing the original structure from 1923. The project took 5 years longer than first estimated and costs were more than double initial estimates. The stadium uses an innovative steel arch that adds aesthetic appeal, but is also load bearing and minimizes the need for internal support that could have obstructed views within the stadium. As a result the arch improves the quality of the seating. The design wasn’t quite a novel as the Sydney Opera House or Guggenheim Bilbao but nonetheless included a design element in the arch that was unprecedented, making best practice techniques such as reference class forecasting impossible because there are no useful historic estimates to draw on – it had not been done before. This lack of historical precedent is often a red flag in accurate project planning.

photo: Kol Tregaskes via Flickr

There appear to be several reasons for delay in the case of Wembley stadium:

Bidding Process and Winner’s Curse

The contract was bid out and awarded to one of the lowest cost bids. This creates a winner’s curse situation, where it’s likely that the winning bid is too aggressive in estimating the actual costs of the project. The cost of the project rose 36% between the bid being accepted and the contract being signed.

Implementation Of An Unprecedented Design

The arch implementation was problematic, ultimately the sub-contractor for the arch was replaced midway through the project, and the delay caused further problems. It appears that the the fundamental issue was attempting a stadium design using a load bearing arch that was novel and untested in previous stadium designs. This is typical of projects that are too innovative, and is one of the reasons that the Denver Airport Baggage System failed. Projects with formal budgets and timelines are not the place to be prototyping unproven techniques and processes. At least not if you’re hoping for a credible initial estimate of how long the project will take.

Source: Martin Pettitt (via Flickr)

Information Flow and Incentives

Information flow around the project was never straightforward and incentives were not well aligned. The contractor was conscious of disclosure to their shareholders and their relationship with the sponsor of the project became so tense as to ultimately end in legal action. In part, this appears to be related to the fixed price nature of the contract – any delay had immediate implications for profitability. This may have lead to two interesting situations, in which it appears more junior employees were better informed about the project than senior management, perhaps because the implications of delay for so serious for profitability that information was not eagerly shared, note than around this time senior management was making statements that the project was on track:

  • Firstly, a whistleblower within the accounting department claimed to know of project delays months before they were disclosed.
  • Secondly, in the UK it was possible to place bets on potential delays on the project. These bets were stopped after the observation of “men in hard hats placing big bets in the Wembley area”.

It is also interesting that after the reviews were disclosed, management then instituted a “peer review” process to better assess the performance of in flight projects.

Trust, Drugs and Scope Changes

As with any project, there are many factors at play.

After the first delays, the sponsor and contractor became less willing to conduct work in parallel due to mistrust of completion dates, this may have added a few months to completion, but in the context of years of delay doesn’t appear to be a primary factor. It is interesting though that on delayed projects, further delays can be self-fulfilling as trust in the critical path diminishes.

There was press speculation that workers on site were using drugs. This claim is hard to substantiate and was never proven.

There were some scope changes, though again, it appears that the construction of the arch (part of the initial design) was a key factor in the delay. Unlike other projects such as the FBI’s Virtual Case File where scope change was a key contributor to delay.

Conclusions

Fundamentally, when attempting a unique work item, such as a novel load bearing steel arch as fundamental part of a stadium, it is very hard to estimate cost and duration with precision. Awarding the work via a bidding process with a fixed price contract exacerbates this problem, because the winning bid will be more likely to underestimate the required work due to the winner’s curse. In addition, it appears information flow could have been improved on this project – if junior employees were aware of potential delays and senior management was not, information was clearly not being shared effectively.

You can see more of these sort of case studies at www.projectcasestudies.com or follow me on Twitter here for blog updates, or consider reading my book.

Arch Detail - Jesse Loughborough (via Flickr)

Project Failure – Channel Tunnel

The Channel Tunnel Rail Link source: Akanekal (via Flickr)

The Channel Tunnel or Chunnel is a 31  mile tunnel running underneath the English Channel to carry Eurostar trains and freight trains between the UK and France.

Construction of the tunnel started in 1988, the project took approximately 20% longer than planned (at 6 years vs. 5 years) and came in 80% overbudget (at 4.6 billion pounds vs. a 2.6 billion pound forecast).

The tunnel wasn’t completely unprecedented. The Seikan Tunnel in Japan had similar length and depth. Nonetheless, like projects such as NASA’s missions and the Sydney Opera House, it seems part of the reason for cost overrun was the absence of many precedents and associated experience to base sound estimates off. In fact, subsequently, the Channel Tunnel has been listed as one of the engineering wonders of the world, which emphasizes its uniqueness.

The issues that caused delay resulted from three factors:

  • Changed specifications for the tunnel, there was need for air conditioning systems to improve safety that were not included the initial design.
  • The communication between the British and French teams who were essentially tunneling from the two different sides and meeting in the middle could have been improved. Theses sorts of communication issues are relatively common in delayed projects when tensions rise, Wembley Stadium is an interesting example, where poor communication meant that junior employees where often more informed about project status than senior managers.
  • The contract was bid on by competing firms, this framework will necessarily encourage the ‘winner’s curse’ of the successful bidders having the lowest and most optimistic price estimates, again the Wembley Stadium project offers another example of the winner’s curse.

Another interesting aspect of the Channel Tunnel’s forecasts were that a lot of revenue was projected to come from driving the existing ferry operators out of business. Of course, these ferry operators were the main way to cross the English Channel before the Channel Tunnel existed. However, this analysis ignored the possibility that the ferries would react to the Channel Tunnel with improved pricing and service, leading to them retaining market share. In addition the creation of budget airlines providing cheap air travel between UK and France was not foreseen. It is a good reminder that in making strategic forecasts of benefits or results you should bear in mind how competitors will react to the project you are envisioning.

Whilst it is not a project management issue per se, it should be noted that a great deal of the financial problems with the Channel Tunnel were caused by overly optimistic revenue projections, on top of the construction cost overruns, and those projections failed to anticipate that the set of options for getting from Paris to London might change, both in reaction to the tunnel and because of innovation in other areas such as the development of the budget airline business model.

See more project management case studies at www.projectcasestudies.com or follow me on Twitter for blog updates.

Channel Tunnel Drilling Equipment (source: Tony Bradbury)

How Good Is NASA At Project Management?

Source: NASA Ares Project Office

NASA too experiences schedule and cost overruns. Of the 10 NASA projects that have been in implementation phase for several years, those 10 projects have experienced cost overruns of 18.7% and launch delays of 15 months. In 2005 Congress required NASA to provide cost and schedule baselines, so no long term data is available. NASA’s projects are consistently one of a kind and pioneering, therefore uncertainty is likely to be higher than for other sorts of projects.

These cost and schedule overruns are largely due to the following factors:

External dependencies

The primary external dependencies that cause problem for NASA are weather issues causing launch delay and issues with partners on projects. NASA projects with partners experienced longer delays of 18 months relative to 11 months for those projects without partners.

Technological feasibility

As the Government Accountability Office assessment states: “Commitments were made to deliver capability without knowing whether the technologies needed could really work as intended.” This is so often a cause of project failure, see my articles on the Sydney Opera House, Denver Airport Baggage System and many
others for examples of how common this cause of failure is.

Failure to achieve stable designs at Critical Design Reviews

90% stability at Critical Design Review is cited by the Government Audit Office as a goal for successful projects, which is consistent with NASA’s System Engineering Handbook. Without this, designs are not sufficiently robust to execute against. It’s clear that NASA takes Critical Design Reviews seriously, but doesn’t always achieve 90% stability. The exact value varies across projects, but appears to be in the 70-90% range for most NASA projects. Raising the stability level at theseCritical Design Reviews would reduce project risk.

Though long, the Government Audit Office report contains lots interesting of further detail and can be found here.

Better Forecasting

Projects fail often, most studies find failure rates above 30% depending on the exact definition of failure, and since budget overrun often cause failure, it seems obvious that better cost forecasts would reduce project failure. Bent Flyvbjerg demonstrates a sound method for improving cost forecasts here in the 2006 Project Management Journal, it’s a fairly long and academic article, so I’ll provide a brief summary.

The approach Flyvbjerg suggests is called reference class forecasting. It’s a simple and elegant solution to the problem of overconfidence in forecasting. In essence, for any project, you estimate how much it would cost using normal methods, of course this requires putting in all the effort and process you would normally invest to develop a sound cost forecast. You then find a set of comparable projects (a “reference class”), with enough projects in the group to be statistically significant, but small enough that the projects are similar to the one that your undertaking. In practise, getting such data is relatively hard unless your organization conducts rigorous post-mortems consistently, but the article cites some distributions for certain project classes. For example rail projects exceed budget by 40% on average. You then gross up your cost estimate by this number and your estimate will be much more reliable.

Reference class forecasting may seem fatalistic or too simple, but it is far more reliable than existing methods. Just as most car owners believe they are better drivers than average, so project managers have excessive confidence in their own estimates, as the literature on project failure rates shows. Flyvbjerg describes this approach as taking the “outside view” looking across projects, rather than dwelling on the “inside view”, the details of the specific project. It is more useful to compare a project with a broad set of similar projects than obsess on the details of your own project.

Of course, there are some caveats to this view, if you believe overruns result from poor forecasting, then this is an effective solution, but if overruns stem from low-balling costs in order to get a project up and running, then the reference class forecasting approach won’t solve that problem.