How the cost of a famous failure gets calculated

Four different quantities

When a figure is attached to a software failure it is usually one of four things, and which one is rarely stated.

What it cost to find and correct the defect. The value of the thing destroyed. The cost of the whole programme the thing belonged to. Or an estimate of value that was never realised afterwards. These differ from each other by factors of ten or more, which is why the same incident carries several different famous numbers.

1996The clearest example of the confusion

A launch vehicle was destroyed shortly after lift-off because software carried over from a previous vehicle performed a conversion that overflowed under the new flight profile, in a component that was not even needed after lift-off.

The inquiry report is public, technically excellent, and does not state a cost. The figures in circulation are press estimates of the vehicle and its payload, which is the second quantity above, and they are quoted as though they were the first.

2012An example where the number is documented

A trading firm deployed new code to some of its servers and not others, leaving old code active with a repurposed configuration flag. It transacted uncontrollably for about three quarters of an hour.

Here a figure is available from a regulatory filing rather than from reporting: a specific loss, stated by the party that incurred it, to a regulator, with consequences for misstating it. That is a different class of evidence from a press estimate and is worth preferring wherever it exists.

1980sThe failures that are not counted in money

Some are counted in harm. A radiation therapy machine of the mid-nineteen-eighties delivered massive overdoses to patients because of a race condition and a removed hardware interlock.

The definitive account is an investigation published in the engineering literature, and it is the standard reference precisely because it examines the whole system, including the assumption that software could replace a physical safety mechanism. No monetary figure would describe it and none is offered.

Why the marginal cost is almost never what is reported

The question a reader usually has is what the defect cost, meaning what would have been saved had it not existed.

That number is almost never available, because it requires knowing what would have happened otherwise. What is available is the total value of something that was lost, which includes everything that was working correctly.

What the good sources look like

Official inquiry reports, which are technical, slow and usually excellent. Regulatory filings, which are specific and legally constrained. Post-incident accounts published by the organisation involved, which are useful and interested.

What to treat carefully: press figures, consultancy estimates of aggregate cost, and any number that appears in a presentation advocating a product.

Why the numbers get repeated anyway

A large figure makes an argument vivid, and the incidents are used in teaching for exactly that reason, including in this document.

The honest handling is to name the incident, describe the mechanism, cite the inquiry, and either give a figure with its source and what it counts, or give none. The mechanism is what transfers to the reader's own work; the figure is decoration.

1999The unit conversion that is always cited

Another mission lost because two teams worked in different units and the interface between their software did not state which. The investigation is public and describes a process failure rather than a coding one: the discrepancy was noticed, raised informally, and not escalated.

The figure attached to it in retellings is the cost of the spacecraft, which is the second of the four quantities again, and the report is far more interesting than the number.

What to do with these stories in an argument

Use them for the mechanism and drop the money. The reason a carried-over component failed in a new context is transferable to anybody's work. The value of a launch vehicle is not.

An argument that needs the large number is usually an argument about attention rather than about engineering, and it will be defeated by the next person who checks what the number counts.

Why the reports are worth reading in full

They are the only place in this field where a failure is examined by people with the authority to obtain documents and no product to sell. Their conclusions are usually about organisation and process rather than about a line of code.

Which is also why they are rarely quoted: the finding does not compress into a sentence, and the number does.

What these failures have in common

Reading the reports rather than the headlines, the recurring features are not exotic. Reused components in a context their assumptions did not cover. Deployment that left two versions running. A safety property moved from hardware to software without the analysis being redone.

None of those requires a large budget to encounter, which is the reason the reports are worth reading and the figures are not.

What we cannot verify

The inquiry reports and the regulatory filing are public and can be read in full. Press cost estimates are not traceable to a stated method. Where this entry describes what a report contains it is checkable; where it characterises what circulates, that is our impression of a literature rather than a survey of it.