Close Encounters with Reference Classes Part2

Introduction

In this post, I describe my second close encounter with a cattle prod reference classes to predict project duration. Like the case described in my previous post, I didn’t produce the reference data. Unlike in that case, I recognized immediately what reference class data meant.

Case Two: “That’s not what the shipyard says”

My second experience with reference classes was predicting the delivery of a large ship that people in the Navy care a lot about. It was a classic application of the Planning Fallacy. Having sufficient funds to complete the ship wasn’t an issue.

My staff and I had been telling senior Navy leaders for over a year that the warship would not finish construction on time. This did not go over well. Each time I thought I had good data (condition of ship compartments, start dates for test programs, etc.), I was slapped down, sometimes hard, by leaders from a remote geographic location.

One set of strong negative reactions came from the “your data isn’t good enough to convince us” group. They didn’t have better data. They just didn’t like mine. The other group told me, “You don’t know what we know about what can be waived.” The slaps always finished with “… and that’s not what the shipyard says.” While it was my job to provide analysis independent of the shipyard, neither group appeared interested in it.

About four months from the shipyard’s advertised delivery date, members of the test group came to me and said, “The ship won’t complete testing on these essential systems by the delivery date.” This is why I love testers even though everyone else treats them like people who drop candy bars in swimming pools (3:40 into the video). Shipyard test engineers can’t keep their jobs if they don’t face facts about testing.

Instead of throwing the test engineers out of my office, a normal reaction for some people, I asked, “How do you know?” They showed me a comparison between the ship’s testing progress and the last several ships (the reference class). Some of the equipment being installed was different, but it was close enough to be representative (more about this in future posts). After my encounter with reference classes described in the last post, I had no trouble seeing their logic.

My second question was, “Using this data, can you estimate when the test program will finish?” Asking testers “When?” questions always makes them nervous. Their expertise is in preparing for and doing tests, not estimating their duration. After all, stuff breaks. They looked at their charts and said with some trepidation, “About four months after the advertised delivery date.” Senior leaders had to be informed.

Providing bad news about high visibility Navy projects is the evil twin of “Build it and they will come.” The Navy version is “Tell the bad news and the gold braid will come.” Like old-time baseball players emerging from cornfields, they also bring baseball bats, but they aren’t for hitting balls. At least not baseballs.

The ship delivered when the testers forecasted and I kept my job. The flag officers, senior DoD civilians, Public Affairs Officers, and staffs closed ranks and managed external communications. My staff and I focused on our assigned duties. I didn’t get my picture on the cover of Navy Times. No one thanked me nor did they apologize for the previous beatings or acknowledge that I had been right. I’ve learned to expect this. As a messenger of bad news, you’re beating the averages if you don’t get shot. Expecting a reward is to believe in fairies.

Be the Bearer of Bad News (aka “Embrace the horror.”)

An important note that doesn’t fit anywhere in the discussion of reference classes, but is exemplified by both I encountered, is: tell the truth as you know it even when you know it is going to be painful. One of the important features of warship construction and repair is that you can’t hide bad news for long. Steel doesn’t keep secrets. Eventually, you run out of money, miss a milestone, or members of the crew start suffering from highly-publicized psychological problems. All of those except missing milestones get negative press coverage. You need to be ahead of the news cycle.

To generalize from my experience as an Engineering Duty Officer responsible for several multi-billion dollar projects, you don’t make friends telling people what they don’t want to hear. You do earn the respect of (some) people who work for them, however. You don’t get thanked for doing your JOB even when you’re right. As former Navy football Coach Ken Niumatalolo would tell his players in practice, “Nobody cares!" (Cregge, 2022).

You are nearly certain to get fired for surprising people with bad news. You can still get fired for providing the information early. Part of every “What’s that f!@#$-ing Ralph done now?” reaction I generated in my career was “Is this his fault? Should we fire him?” This has to be true because I saw other people get fired after bad news surfaced when they were behind the news cycle.

You have no control over whether other people “lose confidence” in your ability to command. You can only control what you do and not what anyone thinks about it. Hiding bad news is NEVER the right thing to do. If people lose confidence in your leadership ability after telling the truth, at least you’ll walk out the door knowing that you did the right thing.

Operators and members of fleet staffs regularly tell maintainers, “We just want you to give us the delivery date bad news earlier so we can plan.” Yes, but they neglect to say that this gives their bosses and their boss’s bosses, and their boss’s boss’s bosses … you get the idea. It gives many layers of bosses more time for beatings, which may be why some maintainers are just a teeny bit reluctant to disclose delays until they have to.

What About Earned Value?

Project managers reading this and the previous post might be asking themselves, “Why didn’t the Earned Value Management System show that the ship in Case 1 was going to run out of money or that the ship in Case 2 would finish late?” For readers who don’t know about Earned Value Management (EVM), it supports analysis of project performance to its plan. In Case 1, the plan was missing work (long story) and new work funds weren’t sufficient to fund what was being encountered. EVM can’t account for that. In Case 2, the contract didn’t require EVM (another long story).

Conclusion

Using the reference class data in this case helped me convince people in a remote location that what the shipyard leaders had been telling them about the ship’s delivery date was wrong. I kept my job either because I informed senior leaders early enough to avoid bringing discredit (and bad press) upon the Navy OR they didn’t have anyone available who could manage the mess do the job better than me. I never learned which.