Close Encounters with Reference Classes Part1
Introduction
In my last post, I identified recent Navy shipbuilding contracts that far exceeded planned durations and costs. I noted that the problem with estimating project durations isn’t confined to the U.S. Department of Defense. I finished with an introduction to the Planning Fallacy (Kahneman and Tversky, 1977). The Planning Fallacy suggests that a detailed knowledge of your project’s technical sequences doesn’t improve the quality of your schedule predictions. A second implication is that it is unwise to expect that your project will do better than every previous one like it no matter how badly you hope to “bend the cost curve.”
Note: details of references are at the end of the first post in this series.
In future posts, I intend to describe an approach to estimating project duration that reduces the impact of the Planning Fallacy. Since choosing a reference class of similar projects is the core as well as likely the biggest source of resistance to the approach (“My project is special!”), this post and the one that follows are focused on two of several experiences that I had with reference classes during my Navy career.
Case One: “You don’t have enough money.”
My first encounter with reference classes occurred during an overhaul. In the description that follows, I omit my command, the budget, and the ships of the reference class. Those details aren’t important.
I was the Navy project representative for a warship overhaul of an extended duration. That means it was expensive too. Money for new work was extremely tight. Audience: “How tight was it?” It was so tight that I had been approving new work using a weekly spending limit for six months. Why we were in this position is a long story that you can read about in a RAND Corporation report.
I had accumulated a significant backlog of discovered, but not authorized, new work. I had to stop approving new work each week when I reached the limit (and I always reached the limit early in the week). I will write more about the situation and my strategy for managing the backlog when I return to ship overhaul topics.
Three important points about this situation: If you are rationing change or new work funds during a warship industrial repair period, you should be thinking:
• First, somebody didn’t plan it well.
• Second, the technical requirements of warships don’t care about your budget. Technical requirements for warfighting and survival in the U.S. Navy are a very high bar.
• Third, the consequence of the second item is that “It’s game over, man.”
Approximately a year before the delivery date, a senior financial manager at the command told me, “You don’t have enough money to finish the overhaul.” I dismissed this warning as alarmism. Impossible. I wasn’t exceeding the new work budget. I had lots of money left to fund the authorized Availability Work Package. I was funding the important stuff. None of the senior officers I worked for thought there was a problem. After all, I still had my job. Take that!
Then he showed me a spreadsheet. Darn! It was an analysis of how much was spent on ships of the same class in the last year of their maintenance periods as a percentage of the total work package. It was roughly the same for all of them. My shields were weakened, but still intact. Those other ships were in different kinds of maintenance periods. I still dismissed his conclusion.
The overhaul I was managing could not run out of money. I wouldn’t let it. “Those were different work packages that aren’t comparable.” “My overhaul has a different abbreviation.” I didn’t see how the analysis said anything about my situation.
The financial manager continued to insist that I was going to need the money even if I wasn’t spending it now. I started thinking, “For me to be wrong, what would have to be true?” This is a good method to open yourself to new ways of thinking. So is asking, “What am I missing?”
I noticed that the maintenance periods in the spreadsheet weren’t the same, but the extent of work performed was similar. The man-days of work performed in the final twelve months were comparable to the overhaul I was managing. My shields were seriously degraded now.
As I noted in previous articles on ship maintenance, the last third of any availability focuses on the completion of habitability work, final assembly of systems, integrated system testing, and crew training. All of the ships in the spreadsheet were going through those things.
This led to my first reference class “Aha!” moment. It didn’t matter that the industrial work periods had different abbreviations (ex: Newcon [new construction], EDSRA [Extended Docking Selected Restricted Availability], Complex Overhaul [COH]). The kinds and amounts of work being done in the last twelve months of these industrial periods were comparable. Uh oh. The shields protecting my denial had failed.
Convinced that the overhaul was in peril, I used the spreadsheet to explain the problem to my superiors. The financial manager stayed in the background when I did so as a safety measure except to nod their head periodically. That made sense since there was no reason to sacrifice two people for the inevitable blowback.
Many beatings (with me as the punching bag) and lots of yelling ensued. In works of literature, what I was doing is called bringing “the wrath of God upon you.” Beatings are an essential part of the ship maintenance stages of grief: anger, denial, beatings, acceptance, and rewards for the uninvolved. Just like there’s no crying in baseball, there’s no “bargaining” after explaining with really good data that your ship will deliver late from maintenance.
Big, unexpected bills are never received well in the Navy, but especially in the year of budget execution when people have other plans for the money. In truth, people always have plans for the money besides maintenance. If you say more money is needed out loud like I did and the bill is very big like it was, multiple flag officers completely rearrange their schedules to meet at the command on very short notice. A bonus is that these are the most easily scheduled senior officer meetings in the Navy because you don’t have to do any of the coordinating. The downside is that they are also coming to fire you.
I didn’t get fired and we eventually received a financial infusion. Sure, I could have continued to ration new work approval, but the reference class assembled by the financial manager convinced me that the result would have been an unfinished, un-deployable ship that couldn’t start its training period until after the work I was deferring was done. This is what happened on CVN 78 because of work that was deferred out of the new construction period as a result of the new construction “cost cap.” As the GAO noted (GAO-15-22, 2014), the cost wasn’t capped so much as moved later to a period of much lower visibility. Deferring work from a warship maintenance period is Cost Analysis Theater because you aren’t saving money. It’s not like taking items out of your cart at the grocery checkout counter. You are only spending the money differently under conditions of lower press attention.
I’m not seeking to portray these aspects of Cost Analysis Theater as nefarious or deceitful. Most of government public action is theater. It is just the way things are and have been at least since the time of the Romans, if not before. Why else did they build coliseums and circuses? It wasn’t just for gladiator combat. Historians are very clear about this.
If you want high-performance warships that can do things, you have to spend the money it takes to install, test, repair, test again, and train operators to use their systems. If the ship class is important enough to the National Maritime Strategy and influential members of Congress agree, senior Navy leaders will align their priorities, budgets, and multi-platform training schedules to accommodate the impact of finishing the ship (newcon or repair) later. This includes what gets announced to the press (“Beautiful plumage!”).
Conclusion
We didn’t have to receive more money to finish the project. When I indirectly invited the flag officers for a command visit to assess my performance, there must have been other conversations along the lines of, “This is what that idiot Soule says it will cost now, this is the impact of fixing it later, and this is how much pain it will cause if we don’t fund it now.” I just wasn’t in those meetings. In my next post, I will describe a second encounter with duration predictions and reference classes.
