Testing, from an afterthought to a discipline

1969The sentence everybody quotes

Testing shows the presence of defects, not their absence. It is quoted constantly, usually as a caution against complacency, and its actual content is stronger than that.

A test exercises one path with one input. Passing establishes that that combination produced the expected result, and nothing whatever about any other combination. The set of untested combinations is not large; for most programs it is unbounded.

1950s and 1960sWhat testing was before it was a discipline

Indistinguishable from debugging. A programmer ran the program, watched what happened, and corrected it, and the activity had no separate name, no separate people and no separate artefacts.

The prevailing intent was demonstration: run it to show that it works. That intent produces exactly the inputs that will work, which is why the reframing that followed mattered more than any technique.

1979The reframing

A textbook defined testing as executing a program with the intention of finding errors, and drew the consequence: a test that finds nothing is an unsuccessful test.

That inverts the psychology completely. Under the old intent a tester wants to pass; under the new one a tester wants to break, and the same person cannot easily do both to their own work, which is the argument for somebody else doing it.

1990s onwardsWhat made it cheap

The change that actually altered practice was not an argument. It was that running a large suite of tests went from an overnight job on shared hardware to something that happens on every change within minutes.

Once verification is nearly free, running it constantly is obvious, and everything that follows, including writing tests before the code and building on every commit, becomes affordable rather than virtuous.

1990 and 2000Generating the inputs instead of choosing them

Two developments removed the tester's imagination as the limiting factor. One study fed random input to a collection of standard utilities and found that a substantial fraction of them crashed, which was a straightforward and uncomfortable result.

The other described stating properties that should hold for all inputs and having the tool generate cases attempting to violate them, shrinking any failure to a minimal example.

Both find defects that no example-based suite would, because both explore the region a person would not think to write down.

The measure that became the target

Coverage counts which lines a suite executed. It is easy to compute, easy to report, and once it is required it is easy to satisfy with tests that execute code without asserting anything about it.

Which is the pattern this document's final part is about, arriving here: a proxy chosen because it is measurable, attached to a consequence, and optimised directly.

What testing still cannot do

Establish that a program is correct. For that there are methods that prove properties rather than sampling them, they are expensive, and they are used where the cost of failure justifies it.

The ordinary position is therefore permanent and worth stating plainly: tests raise confidence, at a cost, without ever reaching certainty, and every argument about how many are enough is an argument about where on that curve to stop.

1990s onwardsWriting the test first, and what it is actually for

The practice is usually defended as producing better coverage, which is a weak argument, and it is more defensible as a design technique: writing the call before the implementation forces a decision about the interface from the position of somebody using it.

Tests written afterwards tend to describe what the code does. Tests written first tend to describe what was wanted, and the difference shows up when the implementation is later replaced.

The suite that has to be maintained

Every test is code, and code that is asserted about the internals of an implementation breaks whenever the implementation changes, whether or not the behaviour did.

A large suite of such tests actively resists change, which is the opposite of what it was bought for, and it is the commonest reason teams eventually delete large parts of one. Testing behaviour at interfaces rather than internals is the whole remedy and is easier to state than to hold to.

The failure that tests are worst at

Everything about how a system behaves under conditions that cannot be arranged in a test: real load, real data volumes, partial failure of a dependency, clock skew, and the combination of all of them at once.

Which is why observation in production became a discipline of its own rather than an admission of defeat. The two are complementary and the boundary between them is roughly the boundary between what can be arranged and what can only be watched.

1990s onwardsWhat changed about who does it

Separate testing departments were normal and have largely gone, absorbed into development teams. The argument for separation was the psychological one above; the argument against was that defects found far from their author are expensive to fix and slow to feed back.

The current arrangement keeps the second and tries to recover the first through review, generated inputs and the discipline of writing the test before the code, none of which fully replaces somebody whose job is to break it.

What we cannot verify

The quoted remark, the textbook, the random-input study and the property-based method are all published and checkable. Claims about how much testing reduces defects in production are organisation-specific and depend on what is counted; none is reproduced here. Coverage figures are not comparable between projects.