ASCII vs. UTF-8

Open a plain-text file written on an ASCII machine forty years ago, tell your editor it’s UTF-8, and nothing changes. There’s no conversion step and no flag to set. Every byte means what it meant before.

That isn’t luck. UTF-8 was designed so that the 128 ASCII codes keep their values, byte for byte, and so that no other character can ever be mistaken for one of them. The work happens in the top bit of each byte, the one ASCII never used. Above 0x7F, the rules change completely.

The problem in 1992

By the early 1990s, Unicode offered one character set for every major script. What it didn’t offer was a comfortable way to store it. Unicode at the time defined every character as 16 bits wide, which broke the assumption, built into decades of software, that one character fits in one byte. Rob Pike and Ken Thompson, who were converting the Plan 9 operating system at Bell Labs, summed it up in a 1993 paper: Unicode “defines an adequate character set but an unreasonable representation.”

A byte-based alternative already existed. The draft ISO 10646 standard included an encoding called UTF that kept bytes 0 to 127 as ASCII. But its longer sequences were built from printable Latin-1 bytes, so ordinary characters could turn up inside other characters. Decoding it took multiplication and division, and it was “not self-synchronizing”: you couldn’t find where a character started without reading from the beginning.

The proposal they sent back, dated September 8, 1992, lists the criteria the design had to meet. Two of them explain most of this article. The first: “ASCII does not occur anywhere in a multibyte encoding.” The second: “It should be possible to find the start of a character efficiently starting from an arbitrary location in a byte stream.”

Seven bits and a spare

ASCII is a seven-bit code. Its 128 characters run from 0x00 (NUL) to 0x7F (DEL). Stored in an eight-bit byte, as it almost always is, that leaves the top bit of every ASCII byte set to 0.

UTF-8’s first rule uses that. A byte whose top bit is 0 is a complete character by itself, and it’s the ASCII character with the same number. The letter A is 0x41 in ASCII and 0x41 in UTF-8. A newline is 0x0A in both.

So an ASCII file doesn’t need converting to become UTF-8. It already is UTF-8. RFC 3629, the current Internet standard for the encoding, puts it plainly: “a plain ASCII string is also a valid UTF-8 string.” The reverse holds too. A UTF-8 file that uses only those 128 characters is plain ASCII, and an old program that knows nothing about Unicode can read it.

What changes above 0x7F

Every other character, from é to Chinese to emoji, takes two to four bytes, and every one of those bytes has its top bit set. The leading bits of each byte say what kind of byte it is:

• 0xxxxxxx is a single ASCII character.
• 110xxxxx, 1110xxxx and 11110xxx start a character two, three or four bytes long. The number of leading 1s is the number of bytes.
• 10xxxxxx continues a character that has already started.

The remaining bits, marked x, hold the character’s number in Unicode, called its code point. Take é, which is U+00E9. In binary, E9 is 11101001. That’s too big for one byte’s seven spare bits, so it goes in a two-byte sequence. The first five bits (00011) go after the 110 marker, and the last six (101001) go after the 10 marker. That gives 11000011 10101001, or C3 A9 in hex.

AU+00410100000141éU+00E911000011C310101001A9€U+20AC11100010E2100000108210101100AC😀U+1F60011110000F0100111119F10011000981000000080

One result of these rules is that some byte values can never appear in UTF-8 at all. C0 and C1 could only start a two-byte copy of an ASCII character (more on that below), and F5 to FF could only start numbers beyond the end of Unicode, or no valid sequence at all. A decoder that finds one knows the data is damaged or isn’t UTF-8.

Why old tools kept working

The first design rule, that ASCII never shows up inside a longer character, is what let UTF-8 slip into existing systems. Every byte in a multibyte character is 0x80 or higher. So a byte with the value 0x0A is always a newline, 0x2F is always a slash, and 0x00 is always NUL.

That means a lot of software works on UTF-8 without knowing it exists. A program that splits text into lines by looking for 0x0A still splits it correctly. A file system that separates path names at / can’t be fooled by a character in a file name. C’s string functions, which stop at the first zero byte, still find the end of the string. Searching for an ASCII word with grep can’t produce a false match inside a Greek or Japanese character. There’s a smaller bonus as well. RFC 3629 notes that sorting UTF-8 strings byte by byte gives the same order as sorting by code point, so a plain byte comparison like C’s strcmp still sorts them sensibly.

The second rule is self-synchronization. Because continuation bytes always start with 10, a program dropped into the middle of a UTF-8 stream can find its footing. It skips any bytes that start with 10, and the next byte is the start of a character. At most it loses part of one character. The ISO UTF couldn’t do this, and neither could X/Open’s proposal. Pike recalled that this gap was why he and Thompson “felt free … to roll our own.”

“Extended ASCII” isn’t UTF-8

The trouble starts with files that use the top bit in a different way. Before UTF-8 caught on, the common way to get accented letters was a single-byte code such as ISO 8859-1 (Latin-1) or Windows-1252. These are often called “extended ASCII.” They keep 0x00 to 0x7F as ASCII and put one more character on each byte from 0x80 to 0xFF.

That makes them incompatible with UTF-8 above 0x7F. In Latin-1, “café” ends with the single byte E9. A UTF-8 decoder reads E9 as the start of a three-byte character, finds no continuation bytes after it, and shows the replacement character instead: “caf�”.

The opposite mistake is just as common. Read the UTF-8 bytes for é (C3 A9) as Windows-1252, and you get two characters, “é”. A curly apostrophe, three bytes in UTF-8, becomes “’”. Garbled text like this, often called mojibake, almost always means UTF-8 was read as a single-byte code. Our extended sets guide shows how to recognize it.

Labels add to the confusion. The WHATWG Encoding Standard, which browsers follow, treats “latin1”, “iso-8859-1” and even “ascii” as names for Windows-1252. So a browser asked to decode byte 0x80 as “ASCII” returns €. The standard explains that the original ASCII documents never assigned anything to bytes 0x80 to 0xFF, so different software filled the gap in different ways. Only 0x00 to 0x7F mean the same thing in all of them.

One right way to write each character

The bit patterns leave a loophole. In principle, the slash (0x2F) could also be packed into two bytes: 110 00000 10 101111, or C0 AF. That’s called an overlong encoding. RFC 3629 rules it out: there is “only one valid way to encode a given character.” That’s why C0 and C1 can never appear.

The ban isn’t just tidiness. Software that checks input for dangerous ASCII sequences can be tricked if a decoder later accepts overlong forms. The RFC gives an example: a filter that blocks /../ in a path might let through the bytes 2F C0 AE 2E 2F, which a careless decoder turns back into /../. The RFC adds that “this last exploit has actually been used in a widespread virus attacking Web servers in 2001.” It doesn’t name the virus, but Nimda, a worm that hit Microsoft web servers in September 2001, used the same trick. One of the paths it tried was /scripts/..%c0%af.., with the overlong slash C0 AF written in URL form. The standard now says decoders “MUST protect against decoding invalid sequences.”

RFC 3629, published in 2003, also tightened the encoding in another way. The original design allowed sequences up to six bytes long. The new standard stops at four bytes and code point U+10FFFF, the end of Unicode’s range.

From exception to default

For years, ASCII and the single-byte codes outnumbered UTF-8 on the web. That changed around the end of 2007. In May 2008, Google’s Mark Davis wrote that for the first time, Unicode “was the most frequent encoding found on web pages, overtaking both ASCII and Western European encodings,” the two milestones arriving within ten days of each other. Google counted pages in plain ASCII separately, even though every one of them was already valid UTF-8.

Today the contest is over. As of October 2026, W3Techs reports UTF-8 on about 99% of the websites whose encoding it can identify. The WHATWG standard requires it: new protocols and formats “must use the UTF-8 encoding exclusively.”

ASCII didn’t go anywhere in that switch. Its 128 characters are the first 128 code points of Unicode and the only single-byte characters in UTF-8, with the same numbers they’ve had since 1967. Every UTF-8 file starts from the same table.