Tag Archives: bent flyvbjerg

The Best Research On Project Management

Some really insightful data backed research on project management comes from Bent Flyvbjerg. He’s originally from Denmark, but now at Oxford University’s business school. You can see a good summary of his research here, and he has a name that’s pretty easy to find on Google.

His recent article in Harvard Business review explains why projects can be subject to the black swan problem.

And a number of his publications discuss the problems of inaccurate cost estimates, especially in public projects, but here is a good summary of reference class forecasting.

Project management is often a fairly practical subject, and so it’s interesting to see the level of deep, theoretical rigor that Flyvbjerg brings to the topic.

Better Forecasting

Projects fail often, most studies find failure rates above 30% depending on the exact definition of failure, and since budget overrun often cause failure, it seems obvious that better cost forecasts would reduce project failure. Bent Flyvbjerg demonstrates a sound method for improving cost forecasts here in the 2006 Project Management Journal, it’s a fairly long and academic article, so I’ll provide a brief summary.

The approach Flyvbjerg suggests is called reference class forecasting. It’s a simple and elegant solution to the problem of overconfidence in forecasting. In essence, for any project, you estimate how much it would cost using normal methods, of course this requires putting in all the effort and process you would normally invest to develop a sound cost forecast. You then find a set of comparable projects (a “reference class”), with enough projects in the group to be statistically significant, but small enough that the projects are similar to the one that your undertaking. In practise, getting such data is relatively hard unless your organization conducts rigorous post-mortems consistently, but the article cites some distributions for certain project classes. For example rail projects exceed budget by 40% on average. You then gross up your cost estimate by this number and your estimate will be much more reliable.

Reference class forecasting may seem fatalistic or too simple, but it is far more reliable than existing methods. Just as most car owners believe they are better drivers than average, so project managers have excessive confidence in their own estimates, as the literature on project failure rates shows. Flyvbjerg describes this approach as taking the “outside view” looking across projects, rather than dwelling on the “inside view”, the details of the specific project. It is more useful to compare a project with a broad set of similar projects than obsess on the details of your own project.

Of course, there are some caveats to this view, if you believe overruns result from poor forecasting, then this is an effective solution, but if overruns stem from low-balling costs in order to get a project up and running, then the reference class forecasting approach won’t solve that problem.

Why Are Only 32% of Projects Successful?

One of the classic reports on project failure is the CHAOS Report from the Standish Group. You can read the press release from their last report (April 2009) here.

Summary:

  • 24% of projects fail completely
  • 44% of project fail partially (i.e. fail to meet their defined criteria on at least one of time, budget and scope)
  • 32% succeed (i.e. deliver on time, on budget and on scope)

It is amazing that only 1 in 3 projects is truly successful. If project estimates were unbiased i.e. just as likely to be overly cautious as overly optimistic, then for every project that fails to deliver, one should overdeliver, and we’d see 1/3 outperform, 1/3 meet expectations and 1/3 underperform (fail). That clearly isn’t happening, as the majority (more than 2/3) of projects skew towards failure, so about twice the number of projects are failing than you would expect.

This does lead to a pretty compelling question. Why can’t we learn from this? Why can’t we get slightly better at every estimation after every failure and thereby slowly converge on effective estimates for projects? Of couse, this assumes that incorrect estimates are the only reason for failure, and, of course, there are many others such as project team coordination and stakeholder management that can easily derail a project with decent estimates.

But aren’t unreliable estimates a good place to start, since they are relatively easy to correct? If you were too low last time, increase your estimate for this time, and vice versa. Project are somewhat unique, but elements of the work in most projects has been done before and can be leveraged for effective estimates.

Further research in this area, published in 2002, and summarized here in the Seattle PI, shows that public works project costs typically exceed estimates by 28%, this is argued to be, in part, due to low-balling estimates to secure project funding. If true, this is very worrying, it’s not that we can’t estimate correctly, it’s that there is no incentive to do so. In an application of the Winner’s Curse, any project that has correct estimates will lose out to an alternative project that has purposefully low-balled its estimates and so appears cheaper, more attractive and hence will receive funding at the expense of the correctly estimated project.

There is an attempted rebuttal to these arguments around estimation  here from US Transit, though the rebuttal focuses on the problems and costs of the estimation process, as doesn’t sufficiently address the assersion by researchers such as Bent Flyvbjerg that cost estimates are far more likely to be too low than too high.

There is an interesting analysis of different types of project failure here from Michiko Dilby, drawing on the UK National Audit Office’s Perspective on Why IT Project Fail (PDF). There is also a more software development centric analysis here. There is an interesting analysis of methods to prevent IT project failure here referencing a lecture by Maurice Perks. Whilst project failure continues, there is no shortage of analysis and suggestions around the topic.

It is also interesting to note that IT and large scale engineering projects tend to receive the greatest analysis when it comes to project failure. hese are types of projects where conventional project management techniques are perhaps most applicable, however projects in other industries receive less attention.

Learn more about avoiding project failure, by reading my book on Strategic Project Portfolio Management.