The ten times programmer

1968Sackman, Erikson and Grant, Communications of the ACM

The claim that some programmers are ten times more productive than others is the most durable number in this field. It has a single ancestor, and the ancestor is a study about something else.

The paper compared online and offline programming: whether working at a terminal beat submitting batch jobs. Individual variation was a secondary observation, reported because it was large enough to swamp the effect the authors were measuring.

1968The study's own description of its method

Twelve programmers took part. They worked on two problems. Some used a high-level language and some used assembly, and the comparison that produced the famous spread includes both groups together.

The widest ratio reported, around twenty-eight to one, is the gap between the single fastest and the single slowest measurement, not between typical performers. The ten to one figure that circulates is a rounded, tidied descendant of it.

1999Prechelt, on the same data

The clearest re-examination went back to the original numbers and pointed out what comparing extremes does to a small sample: with twelve people and a skewed distribution, the ratio between the best and worst observation is close to meaningless as a description of a population.

Comparing typical against typical, rather than extreme against extreme, produces a much smaller figure, and one that varies with the task.

1968The paper's actual question

It is worth stating what the study was for, because the number that escaped it was not the answer to its question.

The subject was whether programmers working at a terminal, with a response in seconds, outperformed programmers submitting a deck and waiting hours. That was a live and expensive question at the time, since terminals cost money and batch machines were already paid for. Individual variation appears in the paper as a complication: the differences between people were so large that they threatened to swamp the effect being measured.

A finding reported as an obstacle became the most quoted result in the field, and the finding it obstructed is largely forgotten.

What a small sample does to a ratio

The mechanism is worth spelling out because it recurs throughout the part this entry belongs to.

Take any skewed distribution and draw twelve values from it. The highest and lowest of those twelve are the two least stable numbers in the sample: draw again and both move a long way, while the middle barely shifts. A ratio built from exactly those two numbers is therefore the least reproducible statistic the data can produce.

Quoting it as a property of programmers is quoting the noisiest available summary of one afternoon's work by twelve people, two of whom were using a different language from the others.

1968 to the presentWhat survives the criticism

It matters that the underlying observation is not being dismissed. Studies since have repeatedly found individual differences in programming larger than in most measured occupations, using different tasks and different populations.

What none of them supports is a fixed multiplier. The size of the difference depends heavily on the task: highly constrained work compresses it, open-ended work expands it, and familiarity with the specific system involved explains a great deal of what is often attributed to the person.

Why the number is used anyway

A figure with no conditions attached is portable, and a figure with conditions is not. The version that travels is the one that fits in a sentence during a salary discussion or a hiring argument, which is exactly the setting where the conditions mattered most.

The same shape appears in every entry in this part: the finding is narrower than the claim, the claim is more useful in an argument, and the argument is where the number ends up.

The version worth using instead

If a figure is needed for planning, the defensible statement is that differences between individuals are large, unstable across tasks, and substantially explained by familiarity with the specific system in front of them.

That is less quotable and it supports the decisions people actually make: keeping teams together, giving people time on a codebase before judging them on it, and treating a single measurement of one person on one task as what it is.

1968 to the presentWhat is not in dispute

Individual variation in programming is real, is large by the standards of most occupations, and has been observed repeatedly since. That part of the claim survives the criticism intact.

What does not survive is the specific number, the implication that it describes a stable property of people rather than a measurement of one afternoon, and the use of it to justify a hiring or pay decision about an individual.

The pattern this belongs to

A striking ratio from a small study becomes a rule of thumb, the rule of thumb loses its conditions, and the conditions are where the meaning was. The other six articles in this section follow the same shape, and one of them ends with no ancestor at all.

What we cannot verify

Whether the participants were representative of programmers then, let alone now, cannot be established from the paper. Later studies reporting variation use different tasks and different measures and are not directly comparable with each other. No figure on this page should be carried into an argument about a particular person.