Buffer overflows, and the functions that could not be made safe

The consequence of a decision made earlier

This entry is the bill for another one. A string that carries no length gives a copy nothing to check against, so a write that is larger than its destination proceeds.

What sits immediately after a local buffer is not more of the buffer. It is the saved state of the calling function, including the value that determines where execution continues, and that value is inside the range the write covers.

1989 to 2011The function that could not be fixed

One library function read a line of input into a buffer and had no parameter for the buffer's size. No implementation of it could be safe, because the information required for safety was not in the interface.

It was deprecated by one revision of the language standard and removed entirely by the next, which is a rare event: a function withdrawn from a standard library not because it was slow or ugly but because it could not be used correctly under any circumstances.

The replacements that were not replacements

The bounded versions of the copying functions were adopted widely and are not equivalent. One of them, given a source longer than the limit, writes exactly the limit and does not terminate the result, so the safe-looking call produces a string with no end.

It was never designed as a safe copy; it was designed for a fixed-width field format from the early file systems. Using it as a safety measure is a misreading that has produced its own family of defects.

1988The event that made it public

A program released onto the early internet spread by, among other means, overflowing a buffer in a widely deployed service. It disabled a significant fraction of the connected machines within a day.

Systematic public explanation followed in the middle nineties, in an underground technical magazine, after which the technique was general knowledge rather than folklore.

1990s onwardsThirty years of partial mitigation

A value placed before the saved state and checked before returning, so that an overwriting write is detected. Memory marked as not executable, so injected instructions cannot run. Addresses randomised, so the attacker does not know where anything is.

Each raised the cost and none closed the class. Techniques that reuse existing code rather than injecting new code defeat the second, and information disclosure defeats the third. The pattern is an arms race in which the defences are cheap and general and the attacks are specific and adaptive.

2019 onwardsWhat the vendors report

Two of the largest publishers of software in this class have published analyses of their own defect histories stating that around two thirds to three quarters of their serious vulnerabilities are memory safety problems.

Those figures come from the organisations concerned, analysing their own products with their own criteria, and are cited here as what they say rather than as an independent measurement. They agree with each other, which is worth something and is not the same as verification.

2010s onwardsThe repair that works

Not a better discipline. A language in which the length travels with the data and the compiler refuses writes it cannot prove are in bounds.

Which returns to where this entry began: the class exists because a representation chosen in 1972 for a machine with a few kilobytes of memory does not carry its own size, and every remedy since has been an attempt to reconstruct that missing number from outside.

Why the compiler could not warn

A reasonable question is why a compiler seeing a copy into a sixty-four byte buffer does not object. Frequently it does now, for the cases where both sizes are visible at the point of the call.

It cannot in general, because the source is usually a pointer arriving from somewhere else, and the language provides no way to state how much it points at. The missing number is missing from the type system, so the check has nothing to be performed against, which is the same sentence as the rest of this entry from the compiler's point of view.

The heap version, and why it is worse

Overwriting past a dynamically allocated block corrupts the bookkeeping the allocator keeps beside it, and that bookkeeping contains addresses the allocator will later write through.

The consequence is a write to an address of the attacker's choosing, arriving at some later unrelated moment, which makes the failure far harder to trace and considerably more useful to whoever caused it.

present dayWhy rewriting is slower than it sounds

The obvious conclusion is to rewrite the affected software, and that is happening at the margins: new components in memory-safe languages, interfaces to existing code, gradual replacement of the riskiest parts.

The obstacle is volume. The bodies of code involved are measured in tens of millions of lines, they encode decades of undocumented behaviour, and a rewrite trades a known class of defect for an unknown set of new ones. Which is why the serious proposals are all about boundaries rather than about starting again.

What we cannot verify

The standard revisions, the function signatures and the mitigation designs are documented and checkable. The vendor analyses are self-reported. Claims about the proportion of attacks that use this class in the wild come from telemetry belonging to security companies and are not independently reproducible.