Three myths of project management

“It ain’t what you don’t know that gets you into trouble. It’s
what you know for sure that just ain’t so.” Mark Twain

Here are three ‘truths’ of project management that need to be examined more carefully:

1. The triple constraint is a rule to live by.

No, the triple constraint has limitations and it breaks down at certain times. Adding more people can slow a project down because of training and network effects. Cutting scope isn’t linear, it’s binary, at some point scope is cut so much that the project can never succeed.

2. A delayed project is a bad project.

It’s critical not the miss the bigger picture, admittedly harder to measure than cost and budget but just as important, Wembley Stadium, The Scottish Parliament and Sydney Opera House were all late and over budget, but architecturally valuable. Most unimaginative structures get completed on time, but are they really more successful? Of course, the gold standard is a project such as the Guggenheim Bilbao that came in on time and received architectural awards.

3. For good estimates you need experts.

No, reference class forecasting is the answer. Using historical data is much better than using experts. Of course, if historical data isn’t available, then using experts can be a second best approach.

Project Failure – Scottish Parliament

source: Asif Musthafa (via Flickr)

The Scottish Parliament overran initial costs by a factor of ten and was delayed by 3 years.  It is clear that similar to the Sydney Opera House the design was not finalized before construction and estimates were not backed by a credible cost estimation process. For example, here are the 5 finalists chosen from a shortlist of 12 in May 1998. Only one of these finalists adheres to the budget, every other proposal fails to meet the brief in terms of both cost and size. It is absurd to specific minimal criteria for a brief and then fail to shortlist finalists based on those criteria. When a project starts with this level of disregard for process, it is unlikely to ever get back on course.

Source: House of Commons Briefing (for reference EMBT/RMJM was selected)

From there, the project continues to unravel, for example the architect then added 4,000 square meters (+14%) to his design area. It is perhaps not surprising that the project manager resigned just over a year into the project because there was clear tension between the sponsors’ desire to have the Parliament ready as soon as possible and the architects’ desire for a “gestation period” to really flesh out his design together with a need to be engaged in all decision making. The lead project sponsor (Donald Dewar) and architect (Enric Miralles) sadly both died during the construction period, which further complicated the project.

Detailed reports on the project can be found in the Hollyrood Enquiry and Parliamentary Briefing and this article from Max Wideman.

Quality

It is worth noting that the Scottish Parliament has won many national and international architectural awards. This is similar to many delayed projects in that they fail specularly on time and budget constraints, but the quality of what is delivered can ultimately be extremely high, even if the apparent process in creating it was not.

Conclusions

The process of selecting and managing the new Scottish Parliament had no regard for credible estimates from the outset. As a result it is unsurprising that the final result bore no relation to the initial estimates. It also appears that the sponsors of the project wanted quality above all, and in that context it is not surprising that cost and time gave way to that objective.

Project Failure – Channel Tunnel

The Channel Tunnel Rail Link source: Akanekal (via Flickr)

The Channel Tunnel or Chunnel is a 31  mile tunnel running underneath the English Channel to carry Eurostar trains and freight trains between the UK and France.

Construction of the tunnel started in 1988, the project took approximately 20% longer than planned (at 6 years vs. 5 years) and came in 80% overbudget (at 4.6 billion pounds vs. a 2.6 billion pound forecast).

The tunnel wasn’t completely unprecedented. The Seikan Tunnel in Japan had similar length and depth. Nonetheless, like projects such as NASA’s missions and the Sydney Opera House, it seems part of the reason for cost overrun was the absence of many precedents and associated experience to base sound estimates off. In fact, subsequently, the Channel Tunnel has been listed as one of the engineering wonders of the world, which emphasizes its uniqueness.

The issues that caused delay resulted from three factors:

• Changed specifications for the tunnel, there was need for air conditioning systems to improve safety that were not included the initial design.
• The communication between the British and French teams who were essentially tunneling from the two different sides and meeting in the middle could have been improved. Theses sorts of communication issues are relatively common in delayed projects when tensions rise, Wembley Stadium is an interesting example, where poor communication meant that junior employees where often more informed about project status than senior managers.
• The contract was bid on by competing firms, this framework will necessarily encourage the ‘winner’s curse’ of the successful bidders having the lowest and most optimistic price estimates, again the Wembley Stadium project offers another example of the winner’s curse.

Another interesting aspect of the Channel Tunnel’s forecasts were that a lot of revenue was projected to come from driving the existing ferry operators out of business. Of course, these ferry operators were the main way to cross the English Channel before the Channel Tunnel existed. However, this analysis ignored the possibility that the ferries would react to the Channel Tunnel with improved pricing and service, leading to them retaining market share. In addition the creation of budget airlines providing cheap air travel between UK and France was not foreseen. It is a good reminder that in making strategic forecasts of benefits or results you should bear in mind how competitors will react to the project you are envisioning.

Whilst it is not a project management issue per se, it should be noted that a great deal of the financial problems with the Channel Tunnel were caused by overly optimistic revenue projections, on top of the construction cost overruns, and those projections failed to anticipate that the set of options for getting from Paris to London might change, both in reaction to the tunnel and because of innovation in other areas such as the development of the budget airline business model.

See more project management case studies at www.projectcasestudies.com or follow me on Twitter for blog updates.

Channel Tunnel Drilling Equipment (source: Tony Bradbury)

The Failure of Denver International Airport’s Automated Baggage System

Denver Airport - source: Gathering (via Flickr)

Denver Airport had ambitious plans to route passenger’s bags to and from aircraft without significant human intervention. The system was called the Denver International Airport Baggage System (DIA ABS). It ran over budget by almost 30%, with an actual cost of $250M vs. $195M planned, and completion was delayed 18 months. These delays themselves are bad, but not disastrous. The problem was that the system did not function as intended. The system itself was not a trivial undertaking with 4,000 vehicles, 5.5 miles of conveyors and 22 miles of track. The design failed in several respects – the carts were often unable to cope with sharp corners in the track and loading bags directly from the aircraft failed. The sensors to determine where bags were in the system were not reliable. The design used a number of technologies that were untested. Whereas the Sydney Opera House is an example of a project with tremendously ambitious goals that simply ran over time and budget until those goals were met, the Denver Airport baggage system stayed much closer to duration and budget estimates, but the goals of the system were not met. And unlike the FBI’s virtual case file project there was no issue with vague goals, it’s just that the baggage system’s goals were clear but unrealistic.

The baggage system simply was poorly designed and poorly tested, more recent, simple computer simulations have found problems with the system, that the project itself was not able to catch until implementation.

See my book for strategies to avoid project failure.

What Failed Projects Have In Common

I’ve written about a number of failed projects (see the Project Failure link at the top of this blog). There is a common thread across all of them. Lack of clear requirements. It is the lack of clarity that is the key point.

The US Census devices had “thousands” of “unreconciled” requirements, the Sydney Opera House had an attractive picture, but no hard factual underpinnings for how the design could ever work, the FBI’s virtual case file project were “ill defined and evolved as the project progressed” and during the H1N1 vaccine project the scope of work changed materially. All of the projects I’ve just described failed.

Now this is a short post, so I won’t detail strategies for project success, but it does seem that a sure fire way to fail is to have vague, changing or conflicting requirements. Interestingly, bad estimates don’t lead to massive delays, vague requirements do. Equally, poor budgeting may not cause massive cost overruns, vague requirements do.

Of course, to run a successful project you need to succeed in a host of areas. It’s not just about the triple constraint of cost, schedules and resources. You need to get the people and communications aspects right too, but to fail you only need to keep the requirements vague.

The Failure Of The FBI’s Virtual Case File Project

Between 2001 and 2005, the FBI’s Virtual Case File project failed. The Virtual Case File project was part of a larger initiative called Trilogy. Costs overran by 89% or just over $200M. A project that should have taken 3 years, failed after 4 years with requirements still not met. In some ways this project is similar to the London Stock Exchange’s Taurus Project, where scope creep lead to the project (at least as initially envisioned) being cancelled after massive cost overruns.  This analysis draws on a very detailed report from the Office of the Inspector General, but here we focus explicitly on the project management failings.

The FBI’s Trilogy project contained three parts:

1. Upgrading software and hardware for FBI agents
2. Upgrading the FBI’s communications network
3. Significantly upgrading the FBI’s case management system (Virtual Case File) to enable better access to, and sharing of, case-related information across the FBI

The first two elements of the project were completed, not perfectly or all that impressively, but roughly as planned. However, the Virtual Case File project experienced major cost and schedule overruns and never achieved its objectives. It is widely used as an example of a failed IT project.

Vague and inappropriate requirements

As with all the cases of project failure that this blog has explored (the Sydney Opera House, the 2010 Census, the Channel Tunnel, Denver Airport and the H1N1 vaccination program) the heart of the problem is vague requirements. “Trilogy’s design requirements were ill-defined and evolving as the project progressed.”

Scope creep

It’s perhaps inappropriate to use the term scope creep for a project where scope is never clearly defined. But the political climate created by the September 11 attacks and reports into the Oklahoma City bombings in March 2002 increased pressure on the project to produce results faster (regardless of what was feasible). In addition, the Hanssen espionage case, one of the most significant in the FBI’s history as portrayed in the movie Breach, raised the bar for security requirements for the Virtual Case File system. According to one report on the project. “Trilogy’s scope grew by about 80 percent since initiation of the project.” Of course, if scope is increasing faster than work is being complete, it will be impossible for any project to finish.

Scheduling by desired outcomes, not resource based estimates

In my book, I describe this as this as the King Canute approach to project management, just because you’re a powerful executive and you want something to be complete faster than 3 years, does not mean that that is possible given the scope of work and resources allocated. However, the scheduling for the Virtual Case File project focused on what was desired, not what was possible. Two quotes from the report illustrate this:

“The FBI also developed plans to accelerate completion of Trilogy because at the time the project’s 3-year modernization timeframe was considered too long.”

“The contractor disagreed with the resulting schedules, because the schedules showed that the full implementation of the project exceeded the proposed completion date of the project.”

It is easy to see these errors with the benefit of hindsight, but in the pressure of the moment it is all to do easy to massage the timeline to make it look as if you might hit the desired outcome, on paper at least.

Cost plus contracting

The interests of the FBI and the contractors working on the project were not aligned.“The cost-plus-award-fee type of contract used for Trilogy…did not require specific completion  milestones…and..did not provide for penalties if the milestones were not met.” Arguably, this is a result of vague scope, but in addition to vague language the incentives created by cost plus pricing, made it hard for the FBI to hold their contractors accountable.

Lack of clear ownership

Tellingly, a project manager was appointed only late into the project, the FBI cycled through 5 people in the CIO role in 4 years and accountability appears generally absent due to “decision making by committees instead of knowledgeable individuals.” In addition to a lack of accountability at the contractor level, there was a need for clearer accountability within the FBI. Lack of clear accountability structures is a clear theme in failed projects as the Eurofighter project shows.

See more analysis of project failure here on this blog and at www.projectcasestudies.com.

2010 US Census and Project Management

We know that many projects fail, perhaps the more interesting follow-up question is why this happens. Following on from investigations of project failure at the Sydney Opera House and the H1N1 Vaccination Program this post examines the 2010 US Census which has experienced several project management problems in recent years. The US Census occurs every ten years to count the number of people in America so as to inform policy discussions, allocate budget and apportion seats to the House of Representatives.

Whilst it’s true that the US Census will happen (a delay to the program would be illegal) elements of the project are a failure. One of the goals of this census was to use automation to “improve coverage accuracy and efficiency”, in part by providing 500,000 handheld computing device to eliminate the need for those walking the streets collecting census data to print out questionnaires and maps. This process was “relying as never before on contractor provided technology”. $3 billion of the $11.5 billion census budget was to be spent on automation including the devices, which cost approximately $600 million. However in early testing in certain locations these devices experienced 27% downtime as well as other issues.

Reports from the US Government Accountability Office showed “an increased probability that the system would not be delivered on time and within budget or perform as expected.”

This situation occurred for the following reasons:

Lack of clear requirements

“The contractor is overwhelmed by a substantial increase in requirements having thousands of unreconciled (that is, not validated) requirements.”

Poor risk management

Some risks have been identified some risks, but there is no formal process or resource allocation to mitigate them should they occur. “Because they did not develop complete mitigation plans, the project team could not ensure that for a given risk, techniques and methods would be invoked to avoid, reduce and control the probability of the occurrence.”

Partial cost management plan

The program had initiated Earned Value, but not selected detailed performance measures to implement full cost management.

Executive reporting and communication gaps

The review calls for reporting on a “periodic and event driven basis”, but there was no evidence to document that risks were discussed with executives.

Of these issues the primary one appears to have been requirements management, because of the reliance on a vendor to provide the technology, this was a critical part of the process. It is not enough to give the vendor requirements, this was happening. Indeed the vendor received thousands of requirements, the need is for consolidation and validation of requirements.

It is likely that imprecise requirements is what caused cost overruns. Precise requirements could also have supported risk management because the timing of requirements and the mitigation strategies if they were not met could have been part of the process.

Whilst project management is a complex and interconnected discipline, it appears that similar to the example of the Sydney Opera House, imprecise requirements and scope management were a significant part of the problem.

Three Tips for ‘lightweight’ projects

Large or expensive projects often have project managers, but smaller, shorter projects with just a handful of people on the team may not have anyone with formal project management ability. Here are a few tips for effectiveness in that situation:

1. Failing to plan is planning to fail.

24% of projects fail outright, but given a new project people typically like to get stuck in believing they’ll be successful. Perhaps they will but stats suggest that you should spend 20% of the duration of the project planning it. Seems counter-intuitive, but many mistakes are avoidable by thinking things through in advance. For a 3 month project, that means the necessary planning time is 12 days, that might feel like an unwelcome delay, but the impact on project effectiveness will be dramatic.

2. Pre-mortems

People love to be optimistic, a pre-mortem does the opposite. A post-mortem is looking at things after they have happened for success and failure. A pre-mortem considers that the project has failed (an assumption) and invites participants to look at why that’s the case. The assumption of failure frees people up to poke holes in the plan and think about a better outcome.

3. Communication

Most project failures are due to lack of communication, people don’t communicate enough and assume that just because an email has been sent it has been read. Just as in consumer marketing people need to see a message 3 times before they remember it, make sure you are repeating your message.

Swine Flu Vaccines and Project Management

Earlier this year, the CDC committed to providing H1N1 vaccines on a broad scale as Time documented. However, the Wall Street Journal reported today, swine flu vaccine production has fallen far short of expectations. This is at a time when cases of swine flu are rising sharply (see graph below). It appears much of this shortfall will be recovered in the next several months. Still a shortfall remains, this post looks at why that’s happened from a project management perspective and what we can learn from it.

The government forecast 30 million doses of swine flu vaccine would be ready by the end of October 2009. With a week to go we are at 16 million, so 46% below target, and even that target was revised down last week weeks ago for 40 million doses, so really we are 60% below initial target, of course some of that deficit will be made back in the next 7 days, but not all of it.

This appears to be a classic example of project failing to deliver on time as expected, even in this case where resources are relatively abundent. That’s not surprising, a project failing to meet expectations is the most common outcome (see related post on project failure). The more interesting question is why, because over time our project management skills do not seem to be improving

The reasons for the delay are as follows:

• Inaccurate estimates – yield lower than expected from seed virus at at least 2 producers
• Regulatory risk – FDA approval for Glaxo SmithKlein’s vaccine taking longer than expected
• Negative events not captured in plan – Some production lines experience ‘brief interruptions’ as larger scale production ramps up
• Scope change – Government changed request to relatively more single dose syringes vs. multi-dose vials

Now of these the only one which is ‘legitimate’ in that it’s not under the producer’s control or ability to forecast, was the scope change from the government to single dose syringes, because that changed the scope of project. All the others events are forecastable. Of course, no estimate is perfect and things will deviate from it. But, crucially, if there are these reasons for things to be behind schedule, shouldn’t be there a similar number of factors driving things to be ahead of schedule? Basically, shouldn’t the ‘good luck’ and the ‘bad luck’ cancel out if the estimates used in the first place were fair guesses? 

Apparently, not

• Medimmune could produce more vaccine (delivered via nasal spray), but are limited by the number of sprayers they have. So they remain on target, rather than ahead of it.

Net – the majorty of firms are behind and none are ahead. It appears there was too much optimism baked into estimates. A classic problem when formulating project plans.

 Image source (Center for Disease Control Site)

Also, if the chart above worries you, note there’s a good chance, the swine flu infection numbers are somewhat overestimated based on historical data.

If these sort of project post-mortems interest you, see this related post on the delay and cost overruns of the Sydney Opera House.

Why Are Only 32% of Projects Successful?

One of the classic reports on project failure is the CHAOS Report from the Standish Group. You can read the press release from their last report (April 2009) here.

Summary:

• 24% of projects fail completely
• 44% of project fail partially (i.e. fail to meet their defined criteria on at least one of time, budget and scope)
• 32% succeed (i.e. deliver on time, on budget and on scope)

It is amazing that only 1 in 3 projects is truly successful. If project estimates were unbiased i.e. just as likely to be overly cautious as overly optimistic, then for every project that fails to deliver, one should overdeliver, and we’d see 1/3 outperform, 1/3 meet expectations and 1/3 underperform (fail). That clearly isn’t happening, as the majority (more than 2/3) of projects skew towards failure, so about twice the number of projects are failing than you would expect.

This does lead to a pretty compelling question. Why can’t we learn from this? Why can’t we get slightly better at every estimation after every failure and thereby slowly converge on effective estimates for projects? Of couse, this assumes that incorrect estimates are the only reason for failure, and, of course, there are many others such as project team coordination and stakeholder management that can easily derail a project with decent estimates.

But aren’t unreliable estimates a good place to start, since they are relatively easy to correct? If you were too low last time, increase your estimate for this time, and vice versa. Project are somewhat unique, but elements of the work in most projects has been done before and can be leveraged for effective estimates.

Further research in this area, published in 2002, and summarized here in the Seattle PI, shows that public works project costs typically exceed estimates by 28%, this is argued to be, in part, due to low-balling estimates to secure project funding. If true, this is very worrying, it’s not that we can’t estimate correctly, it’s that there is no incentive to do so. In an application of the Winner’s Curse, any project that has correct estimates will lose out to an alternative project that has purposefully low-balled its estimates and so appears cheaper, more attractive and hence will receive funding at the expense of the correctly estimated project.

There is an attempted rebuttal to these arguments around estimation  here from US Transit, though the rebuttal focuses on the problems and costs of the estimation process, as doesn’t sufficiently address the assersion by researchers such as Bent Flyvbjerg that cost estimates are far more likely to be too low than too high.

There is an interesting analysis of different types of project failure here from Michiko Dilby, drawing on the UK National Audit Office’s Perspective on Why IT Project Fail (PDF). There is also a more software development centric analysis here. There is an interesting analysis of methods to prevent IT project failure here referencing a lecture by Maurice Perks. Whilst project failure continues, there is no shortage of analysis and suggestions around the topic.

It is also interesting to note that IT and large scale engineering projects tend to receive the greatest analysis when it comes to project failure. hese are types of projects where conventional project management techniques are perhaps most applicable, however projects in other industries receive less attention.

Learn more about avoiding project failure, by reading my book on Strategic Project Portfolio Management.