The cost-of-a-defect curve

The claim

A defect found in requirements costs a fraction of what the same defect costs in design, which costs a fraction of coding, and so on, with a defect reaching production costing something like a hundred times the original.

It appears with specific multipliers, on a rising curve, in presentations arguing for more testing, more review, more specification, and more of whatever the presenter sells.

1981Where it comes from

The curve traces to a book on the economics of software development published at the start of the nineteen-eighties, drawing on data collected from a small number of very large projects at a defence contractor and a few other organisations during the nineteen-seventies.

The original presentation is more careful than its descendants: it reports observed ratios for those projects, notes wide variation, and distinguishes between project types.

What happened to it afterwards

The careful version was cited, then the citation was cited, then a presentation cited the citation with a rounder number, and the provenance dissolved.

An investigation published in the twenty-tens followed the citation chains for several such figures and found repeatedly that the trail leads to secondary sources, that the numbers change between retellings, and that some frequently cited studies do not contain the figures attributed to them.

Why the direction is probably right anyway

Worth conceding clearly. A misunderstanding about what was wanted, discovered after a system is built around it, does cost more to correct than the same misunderstanding discovered in conversation. That is not controversial and does not need a study.

What is unsupported is the multiplier, its universality across defect types, and the smooth exponential shape of the drawn curve.

Why the multipliers cannot transfer

The projects behind the numbers had a particular shape: large, sequential, with formal handovers, on hardware where a release was a physical event and a correction reached users months later.

For a service deployed continuously, the cost of correcting a defect in production may be minutes of work and one deployment. For firmware in a shipped device it may still be a hundred times, or unbounded. The ratio is a property of the delivery mechanism, not of defects.

What the curve is used to justify

Almost always, moving effort earlier: more specification before building, more review before coding, more gates. That is sometimes correct and it is being argued from a number that does not support it for the case at hand.

The honest version of the argument is available without the multiplier: find out what you are building wrong as early and as cheaply as you can, by whatever means suits the work, and notice that for many systems the cheapest way to discover it is to ship something small.

The pattern this part is about

A defensible observation, measured once in a specific setting, acquires a number. The number is easier to carry than the setting. Within a decade it is a law, and within two the setting is unavailable to anybody repeating it.

Every entry in this part has that shape, and this one is the clearest because the original is still in print and says something narrower than its own reputation.

1981What the original actually reported

Worth being specific about, since the criticism here is about transmission rather than about the work. The book reports ratios observed on particular projects, gives ranges rather than single values, and distinguishes between projects with different characteristics.

It is a careful piece of empirical work about a population it describes. Everything wrong with the popular version was introduced by people quoting it.

The one place the curve is not controversial

Where a correction requires physical action: firmware in shipped hardware, software in a vehicle, anything embedded in a device that must be recalled or visited.

There the ratio between finding a defect before manufacture and after is not a hundred to one but effectively unbounded, and the argument for spending heavily on verification needs no citation at all.

How to spot the next one

Three signs, all visible without tracing anything. A round multiplier. A smooth curve through points that were never measured between. And a citation to a source that is itself a presentation rather than a study.

Any one of them is enough to ask for the original, and asking for the original is usually the end of the discussion, which is the finding this part keeps arriving at.

The counter-curve nobody draws

Effort spent finding defects early also has a cost, and it rises with how early it is spent, because early in a project the least is known about what will matter.

Specification effort aimed at parts that are later removed is wasted entirely, and there is no published curve for that because nobody selling early effort has an interest in producing one.

What to do when somebody quotes it

Ask which defect type, which delivery mechanism, and what the correction actually involves for the system under discussion. Those three questions answer the real problem and do not require the figure.

If the answer is that a correction takes an hour and reaches users the same day, the curve has nothing to contribute and the argument should be made on other grounds.

What we cannot verify

The original book is published and its tables can be read. The investigation of citation chains is published and its method can be checked. We have not independently traced every citation, and the claim here is that the provenance of the popular version is weak rather than that the underlying observation is false.