The Winner’s Curse And How To Solve It

The winner’s curse comes out of economic theory, but is very relevant to any project manager outsourcing work to vendors.

Essentially if you are bidding out work where:

1. The costs are uncertain
2. You are going to pick the lowest bidder

Then there’s a good chance the vendor you pick won’t make any money out of the contract. This is because if the cost of the work is uncertain, all vendors are essentially guessing at the cost, and that’s fine. The problem is that in picking the cheapest vendor you are choosing the one who’s guessed lowest. However, based on the wisdom of crowds the most likely actual cost of the work is the average of all the bids you’ve received, but you’re not paying the average, you’re paying the lowest and the problem there is that the vendor is likely to lose money (the difference between the average of all bids and their bid),

Now, on the face of it that might sound like a good thing, but in practice, if you look at the delays that results on the Wembley Stadium project and Scottish Parliament project look can see that having a vendor who’s not making money working for you can cause problems for the overall project. It’s important to note that this outcome is a result of the structure of the bidding process, and is not because anyone is necessarily trying to rig the bidding.

So, how can this be solved, here are a few ideas (and thanks to the audience at a presentation I gave in Vancouver last year for helping out with these)

1. Declare in advance you will pick the second lowest bidder. This removes the incentive to come as the lowest and win the bid.

2. Ask for a detailed proposal, not just a price. Read the detailed proposals first, pick the best, then look at the price and see if your budget can cover it. If not, either move to the next best proposal, or ask for more funding.

Fixing Mistakes Early

Nice chart based on Barry Boehm’s book on Software Economics, shows us the value of fixing problems early in projects, the costs of fixing problems at least doubles at each stage of the process, and become even larger when the results are in operation. The data is based on software projects. This is intuitive, but it’s worth remembering just how much the costs can rise.

The Best Or The Cheapest

Michael Porter describes strategies that successful businesses should pursue. Cost leadership is one example – being the cheapest. Differentiation is another -building something that is sufficiently different, and hard to copy, that people will pay for more it.

Like most solid reasoning, Porter’s approach sounds fairly obvious – implementation is the challenge. With that in mind two recent posts in HBR and TechCrunch highlight what these approaches look like.

Differentiation

• 1111 Lincoln Road – is a parking garage with such striking architecture that people want to get married there.
• The Henry Ford Hospital – is a hospital so well laid out that it feels like a hotel.

Cost Leadership

• The Raspberry Pi Initiative – is a plan to build a PC for $25

Will your next project driving to one of these outcomes? It’s easy to end up with average cost and limited differentiation, it’s much harder to be extreme on one or the other, but extremities are where the real value lies.

1111 Lincoln Road Parking Garage, Joe Vare via Flickr

Henry Ford Hospital via Flickr

The Raspberry Pi, A $25 PC

Project Management at the BBC

Continuing the theme of UK based projects such as Wembley and Scottish Parliament, the BBC is a state funded provider of television and radio in the UK. The UK’s National Audit Office reviewed 3 recent construction projects by the BBC, 2 of which exceed cost and budget and 1 came in below budget the full report is here.

The key findings are as follows:

• The projects did not identify benefits clearly from the outset, making it hard to assess success objectively
• The importance of financial contingencies – the reason the one project (Salford Quays) came under budget was because it had financial contingencies at just over 10% of the initial budget. The projects which went over budget had much smaller contingencies.
• Change management, all projects had material changes during their lifetime, an average of 41 per project, which generally lead increased costs and in some cases could have been avoided through more thorough planning efforts.
• There is a need to better assess skills required to execute the project relative to the organization’s existing capabilities.

Improving Cost Estimation

photo: Thomas Hawk (via Flickr)

The Government Accountability Office provides a reasonable useful framework for cost estimates, they should be:

Credible – this means that sensitivity analysis should be performed on cost estimates to see how much they move in relation to changing assumptions. In addition, cost estimates should be verified by least one other party.

Well-documented – this simply involves making sure all the steps and assumptions in determining an estimate are recorded.

Accurate – this sounds obvious, but the goal is to ensure the estimate is based on the costs that are most likely to be incurred.

Comprehensive – the goal here is that the estimate includes all the costs throughout the life cycle of the work performed.

Not radical by means, but a good cross-check on your current cost estimation process.

Project Failure – Wembley Stadium

Wembley Stadium (photo: Martin Pettitte via Flickr)

Wembley stadium is the home of English football (or English soccer if you’re American) and was rebuilt in the 2000s replacing the original structure from 1923. The project took 5 years longer than first estimated and costs were more than double initial estimates. The stadium uses an innovative steel arch that adds aesthetic appeal, but is also load bearing and minimizes the need for internal support that could have obstructed views within the stadium. As a result the arch improves the quality of the seating. The design wasn’t quite a novel as the Sydney Opera House or Guggenheim Bilbao but nonetheless included a design element in the arch that was unprecedented, making best practice techniques such as reference class forecasting impossible because there are no useful historic estimates to draw on – it had not been done before. This lack of historical precedent is often a red flag in accurate project planning.

photo: Kol Tregaskes via Flickr

There appear to be several reasons for delay in the case of Wembley stadium:

Bidding Process and Winner’s Curse

The contract was bid out and awarded to one of the lowest cost bids. This creates a winner’s curse situation, where it’s likely that the winning bid is too aggressive in estimating the actual costs of the project. The cost of the project rose 36% between the bid being accepted and the contract being signed.

Implementation Of An Unprecedented Design

The arch implementation was problematic, ultimately the sub-contractor for the arch was replaced midway through the project, and the delay caused further problems. It appears that the the fundamental issue was attempting a stadium design using a load bearing arch that was novel and untested in previous stadium designs. This is typical of projects that are too innovative, and is one of the reasons that the Denver Airport Baggage System failed. Projects with formal budgets and timelines are not the place to be prototyping unproven techniques and processes. At least not if you’re hoping for a credible initial estimate of how long the project will take.

Source: Martin Pettitt (via Flickr)

Information Flow and Incentives

Information flow around the project was never straightforward and incentives were not well aligned. The contractor was conscious of disclosure to their shareholders and their relationship with the sponsor of the project became so tense as to ultimately end in legal action. In part, this appears to be related to the fixed price nature of the contract – any delay had immediate implications for profitability. This may have lead to two interesting situations, in which it appears more junior employees were better informed about the project than senior management, perhaps because the implications of delay for so serious for profitability that information was not eagerly shared, note than around this time senior management was making statements that the project was on track:

• Firstly, a whistleblower within the accounting department claimed to know of project delays months before they were disclosed.
• Secondly, in the UK it was possible to place bets on potential delays on the project. These bets were stopped after the observation of “men in hard hats placing big bets in the Wembley area”.

It is also interesting that after the reviews were disclosed, management then instituted a “peer review” process to better assess the performance of in flight projects.

Trust, Drugs and Scope Changes

As with any project, there are many factors at play.

After the first delays, the sponsor and contractor became less willing to conduct work in parallel due to mistrust of completion dates, this may have added a few months to completion, but in the context of years of delay doesn’t appear to be a primary factor. It is interesting though that on delayed projects, further delays can be self-fulfilling as trust in the critical path diminishes.

There was press speculation that workers on site were using drugs. This claim is hard to substantiate and was never proven.

There were some scope changes, though again, it appears that the construction of the arch (part of the initial design) was a key factor in the delay. Unlike other projects such as the FBI’s Virtual Case File where scope change was a key contributor to delay.

Conclusions

Fundamentally, when attempting a unique work item, such as a novel load bearing steel arch as fundamental part of a stadium, it is very hard to estimate cost and duration with precision. Awarding the work via a bidding process with a fixed price contract exacerbates this problem, because the winning bid will be more likely to underestimate the required work due to the winner’s curse. In addition, it appears information flow could have been improved on this project – if junior employees were aware of potential delays and senior management was not, information was clearly not being shared effectively.

You can see more of these sort of case studies at www.projectcasestudies.com or follow me on Twitter here for blog updates, or consider reading my book.

Arch Detail - Jesse Loughborough (via Flickr)

Google Reader Bundle For PPM

There are many good blogs out there, the hard thing is finding them. I really like Google Reader’s ability to share links with others and I’ve created a bundle of the 13 blogs I like best.

If you use Google Reader, you can access the bundle here. If you don’t and don’t have an effective way of reading multiple blogs, then go here to get started.

Building a Project Plan – Key Activities Checklist

In the appendix of a recent report regarding the Department of Energy, the Government Accountability Office use the following checklist for assessing project plans, which I’ve added two broad grouping to

1. Build an accurate plan that reflects the project

• Capturing key activities
• Sequencing key activities
• Establishing the duration of key activities
• Assigning resources to key activities
• Integrating key activities horizontally and vertically

2. Manage project risks

• Establishing a critical path for key activities
• Identifying the float time between key activities
• Performing a schedule risk analysis
• Distributing reserves to high risk activities

Of course,  this is a very schedule-centric checklist. There is no mention of talking with your stakeholders, managing partner relationships or assessing feasibility of the work to be undertaken. However, as a checklist for building a project plan, I think it’s a good list, and the risk management section is particularly useful, because it demonstrates what can be achieved when a solid plan is in place.

Project Failure – Channel Tunnel

The Channel Tunnel Rail Link source: Akanekal (via Flickr)

The Channel Tunnel or Chunnel is a 31  mile tunnel running underneath the English Channel to carry Eurostar trains and freight trains between the UK and France.

Construction of the tunnel started in 1988, the project took approximately 20% longer than planned (at 6 years vs. 5 years) and came in 80% overbudget (at 4.6 billion pounds vs. a 2.6 billion pound forecast).

The tunnel wasn’t completely unprecedented. The Seikan Tunnel in Japan had similar length and depth. Nonetheless, like projects such as NASA’s missions and the Sydney Opera House, it seems part of the reason for cost overrun was the absence of many precedents and associated experience to base sound estimates off. In fact, subsequently, the Channel Tunnel has been listed as one of the engineering wonders of the world, which emphasizes its uniqueness.

The issues that caused delay resulted from three factors:

• Changed specifications for the tunnel, there was need for air conditioning systems to improve safety that were not included the initial design.
• The communication between the British and French teams who were essentially tunneling from the two different sides and meeting in the middle could have been improved. Theses sorts of communication issues are relatively common in delayed projects when tensions rise, Wembley Stadium is an interesting example, where poor communication meant that junior employees where often more informed about project status than senior managers.
• The contract was bid on by competing firms, this framework will necessarily encourage the ‘winner’s curse’ of the successful bidders having the lowest and most optimistic price estimates, again the Wembley Stadium project offers another example of the winner’s curse.

Another interesting aspect of the Channel Tunnel’s forecasts were that a lot of revenue was projected to come from driving the existing ferry operators out of business. Of course, these ferry operators were the main way to cross the English Channel before the Channel Tunnel existed. However, this analysis ignored the possibility that the ferries would react to the Channel Tunnel with improved pricing and service, leading to them retaining market share. In addition the creation of budget airlines providing cheap air travel between UK and France was not foreseen. It is a good reminder that in making strategic forecasts of benefits or results you should bear in mind how competitors will react to the project you are envisioning.

Whilst it is not a project management issue per se, it should be noted that a great deal of the financial problems with the Channel Tunnel were caused by overly optimistic revenue projections, on top of the construction cost overruns, and those projections failed to anticipate that the set of options for getting from Paris to London might change, both in reaction to the tunnel and because of innovation in other areas such as the development of the budget airline business model.

See more project management case studies at www.projectcasestudies.com or follow me on Twitter for blog updates.

Channel Tunnel Drilling Equipment (source: Tony Bradbury)

How Good Is NASA At Project Management?

Source: NASA Ares Project Office

NASA too experiences schedule and cost overruns. Of the 10 NASA projects that have been in implementation phase for several years, those 10 projects have experienced cost overruns of 18.7% and launch delays of 15 months. In 2005 Congress required NASA to provide cost and schedule baselines, so no long term data is available. NASA’s projects are consistently one of a kind and pioneering, therefore uncertainty is likely to be higher than for other sorts of projects.

These cost and schedule overruns are largely due to the following factors:

External dependencies

The primary external dependencies that cause problem for NASA are weather issues causing launch delay and issues with partners on projects. NASA projects with partners experienced longer delays of 18 months relative to 11 months for those projects without partners.

Technological feasibility

As the Government Accountability Office assessment states: “Commitments were made to deliver capability without knowing whether the technologies needed could really work as intended.” This is so often a cause of project failure, see my articles on the Sydney Opera House, Denver Airport Baggage System and many
others for examples of how common this cause of failure is.

Failure to achieve stable designs at Critical Design Reviews

90% stability at Critical Design Review is cited by the Government Audit Office as a goal for successful projects, which is consistent with NASA’s System Engineering Handbook. Without this, designs are not sufficiently robust to execute against. It’s clear that NASA takes Critical Design Reviews seriously, but doesn’t always achieve 90% stability. The exact value varies across projects, but appears to be in the 70-90% range for most NASA projects. Raising the stability level at theseCritical Design Reviews would reduce project risk.

Though long, the Government Audit Office report contains lots interesting of further detail and can be found here.