36 Industrial Megaprojects

First, companies that subject themselves to systematic benchmark-ing are generally more capable project companies than those that do not. We can infer this in many ways. For example, we know that the companies that benchmark their projects with IPA suffer about one-tenth the number of construction accidents as the overall construction industry norms. We know that companies rst starting benchmark-ing have project outcomes that are signi cantly poorer than those that have benchmarked for more than a few years.

Second, we know that even the companies with which we have worked for many years will sometimes nd excuses not to close out a project that had particularly horrid outcomes. The excuses range from the specious That project really isn t representative! to the lame We red all the people on the team, so there is no one to talk to. Conversely, we are aware of no megaprojects whose nal data were withheld from us because the projects went abnormally well!

The fact that we have a known positive bias in the data should make the outcomes presented in the next chapter all the more sober-ing because, even with the assistance of the bias, we are not doing very well.

37

CHAPTER 3

Project Outcomes

In the world of normal-sized projects, how well the projects turn out distributes in the usual way: there is a big group of mediocre projects in the middle, a smaller group of good projects, and a slightly larger group of poor projects. When we rst started examining very large projects as a separate group, we almost immediately noticed an odd phenomenon. The projects seem to fall naturally into exceptionally good projects and exceptionally poor projects, with only a very few in the middle, where the bulk of smaller projects would be found. For this reason we separate our mega-projects into two groups: successes and failures.

To separate the wheat from the chaff, we use the ve dimensions of project effectiveness, shown in Table 3.1 . Also included are the thresh-olds to establish what would constitute failure.

If a project fared more poorly than our threshold on any one of these ve dimensions, we classi ed the project as a failure. However, very few projects failed on only one dimension. If they did fail on only one, it was overwhelmingly likely that it was the most economically impor-tant criterion: production versus plan. 1

If a project does not experience one of the serious problems listed in Table 3.1 , we call it a success. By success, we mean that the project basically performed as promised at the time of authorization or bet-ter. It was built close to budget and close to on time, it was reasonably competitive on cost and schedule, and it made on-speci cation prod-uct as intended.

Success, then, is de ned as a lack of failure. Although this may seem rather unsatisfying, it works for these projects because very few were merely mediocre; overwhelmingly the successes are outstanding proj-ects in every respect. The very small mediocre group could easily have

38 Industrial Megaprojects

been great projects: they were planned and executed well. However, they set conservative targets and then executed on lump-sum contracts that locked themselves into mediocre results. Most of these projects were sponsored by government-owned companies in regions where failure to be predictable is heavily penalized.

We were sometimes missing one or two of the measures on any particular project. For example, many of the projects do not yet have enough operational history to judge whether they have failed or suc-ceeded in that dimension. Some of the projects are so peculiar in scope that we could not measure cost competitiveness. Whenever we are missing a measure, we assume the project succeeded on that mea-sure. What this means is that a few more projects will end up in the failure category when the nal data become available.

Cost The thresholds for success and failure are less arbitrary than their round numbers would suggest. If you overrun a project by more than 25 percent in real terms, the project manager and project team will likely be considered a failure (and may well receive a career-limiting

Table 3.1 Thresholds for Failure

Type of Outcome Threshold for Failure

Cost overruns 1 > 25 percent Cost competitiveness 2 > 25 percent Slip in execution schedules 3 > 25 percent Schedule competitiveness 4 > 50 percent Production versus plan Signi cantly reduced production into year 2

1 Cost overruns are measured as the ratio of the actual nal costs of the project to the estimate made at the full-funds authorization (sanction) measured in escalation-adjusted terms. 2 Competitiveness measures how much the project spent (in constant dollars adjusted to a common location) relative to other projects with similar scopes. 3 Execution schedule is measured from the start of production (sometimes called detailed engineering) until mechanical completion of facilities. Slip is de ned as the actual schedule divided by the schedule forecast at full-funds authorization. 4 Schedule competitiveness is the length of the execution relative to similar projects.

Project Outcomes 39

ogging). * A 25 percent overrun substantially damages the econom-ics of many megaprojects. If you overspend in competitive terms by 25 percent or more, the business is spending itself into oblivion. As we look back over the past 23 years at IPA customers that have disap-peared, all but one of them grossly overspent for their capital assets. (The exception was a large oil company that could never seem to nd any oil.)

One of the ironies of large projects around the world is that most of them are executed on xed-price (lump-sum) contracts. And yet many of those so-called xed-price contracts end up anything but xed. (I discuss the merits and demerits of lump-sum contracting in Chapter 11 .) The reason that lump-sum contracts offer only the weakest insur-ance against cost overruns is that changes are often made after the con-tract is agreed on. It doesn t make much difference where the change comes from; if there is substantial change, the cost of the project will go up. Sometimes, we can point to new technology as the source of change, but that just begs the question of why the technology was not ready when the project was sanctioned. And many large overruns, some up to 350 percent, are in projects with entirely standard tech-nology. Sometimes the source of change is lack of sponsor familiarity with the region, although again we will have to ask why. Some of the large overrun projects were caught up in a downward labor produc-tivity spiral. (I discuss how and why these collapses of progress in the eld occur in Chapter 12 .)

* It is also true that the original business director of the project will probably not suffer adverse conse-quences of that overrun. It is very likely he or she will have moved on to another, usually higher, position and any connection with the project will have been forgotten. The tendency in most companies to move business directors rather quickly (three years is a long stay) fosters a lack of business accountability for project results. This is especially true for megaprojects because their gestation and execution periods are so long. This anticipates an issue that we take up in Chapter 5: the relative importance of cost versus schedule for the typical large project. Too many senior business decision makers in commodity industrial sectors fail to realize the disastrous effects of overspending on projects. When a company overspends on its capital projects relative to the competition, even by a relatively small amount, it is systematically disadvantaging itself. The effects show up when a serious downturn hits their industry. In commodity businesses, cycles are a fact of life. Part of the reason for business indifference to capital effectiveness is that the effects are insidious and long-term rather than dramatic and immediate. Some, I fear, understand the effects but simply do not carebecause the effects are long term and their personal horizons are not.

40 Industrial Megaprojects

Schedule The execution schedule is the forecast of the time that will be required from full-funds authorization until the project is ready to start up. If the schedule was developed properly, the time requirement is intimately linked to the scope and therefore the cost of the project. If you slip your schedule by 25 percent, you are usually a year late, which is very damaging to the economic value of the project. Being late in execution is associated with stretching out the period in which you are heavily invested in the project (most of the money is already spent) with no revenue stream to show for it. We set the threshold on being slow at a higher level (50 percent) because if you plan to be slow, the economic damage is less because the spending is back-end loaded and you tend not to plan to be slow in the construction por-tion of the project. However, there are limits to that strategy, and we set them at 50 percent over industry average time.

Large schedule slips are a little less common than large overruns and some of the projects with large cost overruns had only modest sched-ule slippage. The underlying reason that schedule performance looks somewhat better than cost is also one of the core pathologies of very large projects: they are frequently highly schedule-driven from the earliest point right through to startup. Because we end the execution time when the facility is, in principle, ready to operate, we are not capturing with this measure whether the project operated successfully. Executing a project rapidly that also operates is much more compel-ling than merely executing a project rapidly. Startup and operability are captured separately.

Some very extended construction schedules had their proximate cause in engineering. As mentioned in the previous chapter, industrial megaprojects are very engineering-intensive. For a typical project, 25 percent of the total cost will go to engineering and project man-agement services. A good many of the projects with severe slips and cost growth in construction suffered from late and/or shoddy engineer-ing. Late and poor-quality engineering make the construction in the eld or fabrication in the yard much less ef cient. Late engineering and poor-quality engineering tend to happen together; as engineering gets late, it gets progressively rushed and quality control is bypassed to get

Project Outcomes 41

engineering to the constructors. When engineering work is slow to mobilize early in the project, it drives late and out of order equipment and engineered bulk materials, such as pipe spools. Construction/fabrication then waits until the right materials arrive. Engineering problems were common in the failed projects, but they were usually proximate causes, not root causes of failure.

Some of the large schedule slips occurred when the projects got crosswise with the host government. One of the time-honored ways politicians have of intervening in large projects is to withdraw a per-mit on a technicality. For example, the Russian government withdrew the construction permit for the Sakhalin-2 Project when it was deep into construction knowing that the project was very vulnerable eco-nomically at that point because the investment is large and irreversibleyou can t get your money back!

A few of the large slips in schedule occurred when the sponsors acknowledged that the project was going so poorly that it needed to be slowed down while errors were corrected. That is a very painful, dif cult, and courageous decision to make, which is why it is made so rarely. Ironically, two of the successful projects were rescued by running into government trouble. Both projects were about to be authorized but were nowhere near ready for execution; their failure was ordained when government delays in approvals held them up for 18 months in one case and two years in another. They used the time to get ready for execution. Both project directors said they could never have succeeded otherwise. Their good fortune was to encounter government trouble early rather than during the middle of execution.

Production Finally, production versus plan is the most economically leveraging outcome. When we assessed operability, we did not penalize a proj-ect if there was insuf cient demand. (One could justi ably argue that is the worst outcome of all. I built a great project that nobody wanted! Lack of demand, however, is not something on which project management per se can have much effect.) We classi ed a project as a failure in this dimension only if it was experiencing severe techni-cal problems that continued well into the second year after startup.

42 Industrial Megaprojects

We did not set an absolute production threshold because the impor-tance of poor operability is different for different projects. For exam-ple, for an oil production project, production of 70 percent of plan is not terriblethe industry s average is just over 80 percent of plan. * By contrast, 70 percent of plan for lique ed natural gas (LNG) is abys-mal. LNG is very capital-intensive, which exacerbates the effects of poor operability. LNG is almost always forward-sold, and if you cannot produce the LNG, you have to go into the merchant market and buy it from competitors to ful ll your contractual commitments. The result is that you hemorrhage money. The actual production of the projects that failed in this dimension averaged a miserable 41 percent of plan in the second six months after startup.

Even worse, however, when a project suffers signi cant production shortfalls, a great deal of money is spent trying to rectify the problems. Although we lack the systematic data we would like, my guesstimate based on limited data is that 25 to 50 percent added cost over the ini-tial capital is common. Most of this is not actually capitalized, and in many cases there are no reliable records of the amount spent at allbecause nobody actually wants to know.

Production shortfalls in petroleum production megaprojects were common. Almost half (47 percent) of all the oil and gas produc-tion projects in our database had signi cant operability problems. A few were caused by facilities that did not operate properly or facilities that were incorrectly matched to the characteristics of the oil and gas being produced. The most common problems were associated with res-ervoir surprises that limited the production from the intended wells. This almost always resulted in additional wells that were often more complex than the original wells planned. We almost never were able to capture the cost of these additional wells. So once again, we are per-force underestimating the true economic costs of production shortfalls.

The effects of production problems can be debilitating for a busi-ness. One global chemical company client had a $9 billion per year

As we discuss in Chapter 7 , the reservoir problems often could have been anticipated and addressed before the start of the project if the basic technical data development had been better.

* Yes, that does raise the question about why the industry as a whole is so optimistic, but that is not an issue for us here.

Project Outcomes 43

business earning 22 percent return on capital employed (ROCE)a very nice commodity business. A single megaproject that failed to produce as planned reduced the ROCE from 22 percent to 16 per-cent for the ve years after the project was supposed to have started up. They ended up divesting the business. Another example is a met-als project that was to debottleneck and expand a major processing complex by 90 percent. After a 39 percent overrun (76 percent nomi-nal) and an 85 percent schedule slip, the complex actually produces 10 percent less than before the project. This project managed to achieve the unachievable: negative production!

When megaprojects fail, the results are rarely publicized unless the failure is spectacular. When the failures do make the press, they are damaging to a company s reputation. Large overruns and delays in cash ow due to schedule slippage or production shortfalls jeop-ardize the sponsor s ability to fund other projects in its portfolio. Megaprojects are by nature lumpy investments. Only a handful of companies in the world are large enough to be able to support a genu-ine portfolio of these projects to spread the risk internally, which is why most industrial megaprojects are joint ventures.

Trading Outcomes Making trade-offs among cost, schedule, and quality (measured by operational performance in our case) is as old as projects and a fea-ture of everyday life. How trade-offs are made tells us a lot about how decisions around major projects are made. I nd the patterns in mega-projects interesting.

First, cost-effectiveness and schedule effectiveness are not being traded overall, although they clearly are for some projects. In fact, lower cost and faster schedules correlate very strongly ( P > | t | < .0001). Projects that are slow relative to industry standards also are expensive relative to industry standards and vice versa. This means that cost and schedule in megaprojects need not compete. Expensive projects were projects that slipped their schedules and overran their costsno surprise there ( P > | t |< .0001). Expensive projects corre-late, but less strongly, with operational failures ( P >| t | < .02).

44 Industrial Megaprojects

There is no relationship between being schedule competitive and overrunning costs. One strategy for megaproject execution is to stuff additional contingency into the authorization estimate. That money then helps keep the projects on schedule and renders them relatively fast. This is a reasonable strategy for projects with very robust returns, but it does, of course, lead to projects that are at least somewhat more expensive.

There is also little relationship between cost growth and schedule slippage ( P | t |< .08). What this tells us is that we can have projects that trade cost growth for adherence to schedule (19 percent of projects), projects that slip in schedule to prevent costs from overrunning (22 percent), projects that grow in cost and experience schedule slippage (40 percent), and projects that meet or underrun both cost and sched-ule targets (19 percent).

Projects that slip in schedule, however, have a strong tendency ( P | z |< .003) to be operational failures. This result is important. Projects slip for two kinds of reasons, which often end up being related: (1) the schedules were set far too aggressively at the outset, and (2) a large num-ber of major changes occur during execution. When the schedules are set too fast at the beginning of the projectit almost always happens long before authorizationcorners are cut leading up to execution. Those cut corners ultimately translate into production failure.

Projects that grow in cost also have a tendency to suffer produc-tion failure ( P | z |< .04), but when schedule slippage is controlled, the effect of cost growth on operational failure disappears entirely ( P | z |< .51).

What about Construction Safety? Normally, when IPA evaluates a project, the rst outcome discussed is whether the project was completed without seriously hurting any-one. We have three basic safety statistics for about 70 percent of the mega projects:

1. The recordable incident rate, which measures the number of injuries and work-related illnesses that require assistance beyond rst aid

Project Outcomes 45

2. The number of DART casesinjuries and work-related illnesses that require D ays A way from work, R estricted work duties, or job T ransfer

3. The number of fatalities

I have elected not to incorporate the safety results in the assessment of success and failure for a very simple reason: Too many of the safety numbers for megaprojects are simply not credible. Some of them are ridiculous on their face: we had one project with 36 recordable injuries in over 20 million hours . . . and every one of those record-able injuries was a fatality. And we obtained the fatality number only because we had monitored the local press.

Injury rates in construction follow a reasonably predictable pattern: There are many more recordables than DART cases, and many more DART cases than fatalities. This is referred to as the safety pyramid. For megaprojects, that relationship appears not to work. In particular, there are a lot of fatalities and too few recordables. This leads us to believe there is a reporting problem; fatalities are generally very hard to hide.

The third reason to doubt the data makes us sure that underreport-ing is going on: There is a very strong regional bias to the numbers and that regional bias is completely counterintuitive. Projects with higher skilled, more experienced, and higher productivity workforces will on average experience fewer accidents. The higher productivity effect alone helps greatly because it reduces the number of laborers on site. Congestion is a known safety hazard. However, as one can see in Table 3.2 , for megaprojects none of this appears to hold.

Table 3.2 Safety by High-Wage versus Low-Wage Regions

Incident Type Europe and

North America Middle East, Central Asia,

Asia, and Africa

Recordables 1 1.20 0.43 DART rate 0.50 0.12 Fatal incident rate 0.009 0.006 Projects with fatalities 18% 39% Field hours per million dollars 7,300 21,500

1 The rates in the rst three rows are calculated per 200,000 eld hours. For readers familiar with the per million hour formulation, multiply by 5.

46 Industrial Megaprojects

If we believe the numbers, which we do not, megaprojects in Europe and North America, where skill levels and productivity are high, are about three times more dangerous than those in areas where the average skill level is much lower. They even have more fatalities per 200,000 hours worked, even though the probability of having at least one fatality is much higher in the second group. What we also know about Europe and North America is that there are substantial penalties for underreporting injuries and that the legal systems are geared to ensuring that reporting occurs. Those safeguards are much weaker in many other areas.

There is one nal clue that the safety numbers are not legitimate. For North American and European projects, the DART rate is corre-lated with the thoroughness of the project execution planning, just as it is for projects generally. (This is a subject we take up in Chapter 10 .) For projects in the regions in the second column in the table, there is no relationship whatsoever.

In low-productivity environments, keeping track of safety and get-ting reporting done accurately are much more dif cult. Each million dollars of investment (based on 2003 U.S. dollars) in North America and Europe consumes about 7,300 hours in the eld. In the low-productivity regions, about three times as many hours are needed and the projects in those regions are among the largest in our database. Furthermore, the labor force is often drawn from numerous nationali-ties and cultures, so language and custom must be overcome as well.

Nonetheless, the underreporting is systematic. The problem starts at the craft level. The workers know that when injuries are reported, they will get into trouble. Sponsors try to ght against the problem (although sometimes not too much), but the problem is deeply endemic from the craft labor and foreman level up. The pattern of safety results from the regional difference paints a clear picture of underreporting.

It is very unfortunate that construction safety numbers in some parts of the world are underreported. Honest reporting is necessary for creating genuinely safe projects because it provides the incentive structure for strong safety systems to be viewed as good business. It also must be disheartening to those project professionals who have driven genuinely safe projects in places where the numbers are sus-pect. Their accomplishments tend to get denigrated along with the liars and the cheats.