CBO Cost Estimating Methodology
What’s past is prologue.
> — Shakespeare, The Tempest, Act 2 Scene 1
Introduction
This post continues the theme that, with some adjustments, cost estimates for newly designed ships can be more accurately estimated using actual construction costs of similar warships than bottom-up planning for a specific case. Since no warship construction schedule survives contact with dry docks (and material and manning shortages, and testing), cost estimates based on a specific instance are almost always wrong (Kahneman&Tversky, 1977).
I began the series with a brief introduction to the Planning Fallacy (Kahneman&Tversky, 1977). The Planning Fallacy can be summed up with a paraphrase of Hamlet, “There are more things in heaven and earth, Horatio, than are dreamt of in your [acquisition plan]” (Hamlet, Act 1, Scene 5). The way to reduce prediction bias is to anchor schedule and cost on distributional data. “Distributional information, or base-rate data, consists of knowledge about the distribution of outcomes in similar situations” (p.2-1, Kahneman&Tversky, 1977).
In the second and third posts of the series, I provided specific applications of reference class forecasting. In their description of the Planning Fallacy, Kahneman and Tversky (1977) suggested that a detailed knowledge of your project’s technical sequences (the “inside view”) doesn’t improve the quality of your schedule predictions. This doesn’t sound right, but historical data strongly suggests it is right.
In this post, I describe the Congressional Budget Office’s (CBO) new warship cost estimating methodology. It closely follows Kahneman and Tversky’s recommendations for using distributional data (Kahneman&Tversky, 1977). I conclude with two quotes from CBO analyses that don’t inspire confidence in U.S. Navy’s new warship cost estimates.
All references are located at the end of the first post in the series, here.
CBO Cost Estimating Method-Overview
There are three steps in the CBO methodology for warship cost estimating (Labs, 2018):
Estimate the ship’s size and dimensions, if Navy data isn’t available.
Estimate the ship’s cost per thousand tons based on historical data from an analogous class(es) of ship(s), the reference class (base-rate data).
Adjust the cost per thousand tons for the entire class based on judgments of similarity to the reference class, production efficiencies, inflation, worker learning, and competition among shipbuilders, if any.
CBO Cost Estimating Process-Details
Start by estimating the ship's size based on Navy descriptions of its mission and other qualitative factors. For ships starting construction, this isn’t necessary because the dimensions and displacement are known.
Comment: Estimates of the cost of a ship "projected to be built in 20 years," (CBO, 2018, p.1), even when required by Congress, are Cost Estimating Theater. Legislation and theatrical stories aren’t subject to real-world limitations. Shipbuilding is. Just as no one expects Elvis to show up at their house (well, not many people) no one acts like they believe 30-year shipbuilding plans either.
The "agency then uses historical data from an analogous class of ship (or analogous classes) to calculate the new ship’s cost per thousand tons" (CBO, 2018, p.1). The historical data constitutes the reference class recommended by Kahneman and Tversky for improving prediction accuracy by basing the prediction on the distribution of outcomes in similar cases (1977).
Comment: Selecting the reference class is not like consulting a look-up table. The CBO's process requires adjustments based on judgments of how similar the reference class is to the Navy's proposed design and its mission. Someone seeking to apply distributional information might need to choose among several alternatives. The choice of a reference class will likely involve tradeoffs between competing criteria. If so, the organization using the Kahneman and Tversky method will need to justify its choice. The CBO does just that.
In "How CBO Estimates the Cost of New Ships” (Labs, 2018), there is a detailed justification for why the Virginia class submarine was chosen as the reference for the Columbia class over the Ohio class. TL;DR summary: the first ship class has been building since the late 1990s and is an instance of modern shipbuilding. The other was built in the 1970s when shipbuilding, submarine design practices, and the vendor base were much different.
Similarly, in "The Cost of the Navy’s New Frigate" (Labs, 2020), there is qualitative explanation for why the CBO considered DDG 51 Flight III construction costs as the reference class for the Constellation (FFG 62) and not the Littoral Combat Ship.
Class adjustments: additional factors are considered in CBO estimates. The CBO projects rate of production efficiencies. Making more ships gives shipbuilders the opportunity to get volume discounts on material and accumulate experience faster. Both can lower costs, which builders can factor into contract bids. The CBO also estimates inflation (good luck with that), worker learning, and whether there will be competition among multiple shipbuilders that could influence shipbuilding costs. Except for inflation, the CBO doesn't specify the magnitudes of these adjustments (Labs, 2018). It is hard to imagine that they would amount to more than 20% of the total price.
The CBO methodology is not the same as pricing a ship by its weight (displacement). The comparison with the known cost of similar ships deliberately ignores details of system design, weapons systems, crew size, hopes and dreams of senior Defense Department officials and contractors, and how many conversations people have had about doing better. I don’t know how anyone can calculate savings based on claims that “We’ve had a lot of conversations.”
The core assumption of the Kahneman and Tversky approach to prediction as applied by the CBO is this: there isn’t any reason to expect that a new ship and its systems will be significantly less costly to design, build, and test than recent, similarly configured ships with similar missions. As I suggested in two previous posts documenting my experience with reference classes, the details of the designs don’t matter. That’s what makes reference classes such a powerful prediction tool.
What about the impact of beginning ship construction before the design is complete? Sometimes the Navy does this. “Concurrent design and construction” sounds more professional than “We’ll be welding stuff before we know what will bump into it.” The only possible result is increased design and construction costs. Designing and building simultaneously can increase testing costs too if new systems don’t work the way we’re supposed to (spoiler alert: they never do).
Concurrent design and testing is either sheer lunacy (man in the street view) or a rational approach when the shipbuilder is considering releasing a large fraction of its experienced workforce due to a lack of work. The reasons for a warship design not being complete before a layoff deadline are beyond the scope of this post. When the Navy authorizes concurrent design and construction, the CBO’s estimates should be considered lower bounds for the final costs. No one can predict the cost to build something when its design isn’t finished. Distributional data won’t help.
Conclusion
Prior to the announcement of significant delays to all major U.S. Navy warship construction programs, the two most important quotes from recent CBO analyses were:
“If the Navy’s estimate turns out to be accurate, the FFG(X) [which later became FFG 62] would be the least expensive surface combatant program of the past 50 years” (Labs, 2020, p.1).
“[The] Navy’s estimate, the cost per thousand tons of displacement, for the first Columbia class ship [“the largest, most technologically complex submarine the United States has ever built”] would be 13 percent less than that of the first Virginia class attack submarine.” (Labs, 2023, p.25).
In my next post, I’ll discuss some implications of using reference-class cost estimating (CBO) vs. bottom-up approaches (U.S. Navy program offices, public infrastructure projects).
Notes
TL,DR (or TLDR): This is a modern version of Bottom Line Up Front (BLUF). It stands for Too Long; Didn’t Read. The common practice is to provide a summary of a more detailed argument or description that follows.
