1963 onwardsSeven bits, and then an eighth with no owner
The character standard everything descends from defines one hundred and twenty-eight characters in seven bits, which covers English and very little else.
Machines stored eight bits, so there was a spare bit and therefore a spare hundred and twenty-eight positions. Nobody owned that space, and every language community, manufacturer and country filled it differently.
1980s and 1990sThe same byte, several letters
The result was dozens of incompatible arrangements. A byte value in the upper range is an accented Latin letter under one arrangement, a Cyrillic letter under another, a Greek one under a third, and half of a character under a fourth.
Nothing in a plain text file records which was intended, because there was nowhere in the format to put it. A file therefore has no encoding; it has an encoding somebody remembers.
Guessing as the standard practice
Software responded by inferring the encoding from statistics of the bytes present, which works often enough to be adopted and fails silently when it does not.
The garbled text everybody recognises is the visible form. The invisible form is text that decodes plausibly under the wrong arrangement, which is worse, because nothing looks broken.
1980sThe languages that never fitted at all
Writing systems with thousands of characters needed several bytes per character, and the encodings devised for them reused byte values that also mean something in the seven-bit range.
Which produced a durable class of defects where the second byte of a character is also a meaningful punctuation mark, so software that scans for that mark cuts characters in half. Path separators, quotes and escapes have all been damaged this way for decades.
1991One number per character
The repair assigns every character in every writing system a single number, independent of how those numbers are stored.
Separating the number from its storage is the important part, and the part most often missed. There is one catalogue of characters and several encodings of it, and confusing the two produces most of the remaining confusion in this area.
1991 to 1996Sixteen bits, obviously enough
The first design assumed sixteen bits would hold every character anyone would need, and several major systems built that assumption into their string types permanently.
It was not enough. The repair encodes characters beyond the sixteen-bit range as a pair of units, so in those systems a string's length in units is not its length in characters, and cutting a string at an arbitrary position can split a character in half.
That is the same error as two digits for a year and thirty-two bits for a count of seconds, committed by people who had watched both.
present dayWhat length even means
Four answers, all correct for different purposes. Bytes. Units of the encoding. Characters as catalogued. And what a reader would call one character, which may be several catalogued ones combined: a letter with marks, a flag, a family.
Most programming interfaces offer one of the first two and call it the length, which is why truncating a display name is one of the more reliable ways to produce something broken.
The mark that means it went wrong
The catalogue includes a character whose entire purpose is to occupy the place of something that could not be decoded, so that software has something valid to produce rather than failing.
It is a well-designed feature and it is also an admission: the system expects to be handed bytes whose meaning is unrecoverable, often enough that the standard reserves a character for the occasion.
present dayNormalisation, the part still catching people
Some characters can be written more than one way: as a single catalogued character, or as a base letter followed by a combining mark. Both display identically and neither is wrong.
Comparing them as sequences says they differ, which breaks searches, deduplication, file names moved between systems, and any identifier a person typed. The repair is to convert to a canonical form before comparing, and forgetting to is one of the most common remaining encoding defects.
present dayWhere the catalogue is not neutral
Deciding what counts as a distinct character is not a technical question with a technical answer. Whether two forms used by different communities are the same character, how a script that nobody writes today should be represented, and which symbols merit inclusion are decided by a committee whose members are largely funded by corporations.
The unification of certain East Asian characters across languages remains contested by some of the communities affected. That is worth knowing before treating the catalogue as though it were a description of the world rather than a set of decisions.
1992What made the transition possible
An encoding designed so that existing seven-bit text remains valid unchanged, no part of a multi-byte character can be mistaken for a single-byte one, and software searching for a plain character cannot match inside a longer one.
Those are compatibility properties rather than elegance, and they are why a change affecting every text file in the world could be made gradually instead of all at once. Its origin is described in the entry on the system it was designed for.
What we cannot verify
The standards, the code tables and the revision histories are published and can be checked directly. Accounts of why particular systems committed to sixteen bits are partly documented and partly recollection. Estimates of how much text remains in legacy encodings are not measurable in any general way.
In short
- Seven bits covered English; the eighth bit's hundred and twenty-eight slots had no owner.
- The same byte is four different letters, and the file does not record which.
- Guessing works often enough to be adopted and fails silently when it does not.
- Multi-byte encodings reused values that are also punctuation, cutting characters in half.
- One number per character, stored separately from the number, is the repair.
- Sixteen bits was assumed sufficient by people who had already watched two digits fail.