ConceptDBISAI-written · not yet proofread

UTF-8

UTF-8 is a variable-length, ASCII-compatible encoding of Unicode code points in 1 to 4 bytes, where the leading bits of the first byte give the length and continuation bytes start with 10.

Know these firstPrefix-Free Code

Formula#

Bytes	Code point range	Payload bits	UTF-8 code
1	$[0, 127]$ (`0`–`7F`)	7	`0xxxxxxx`
2	$[128, 2047]$ (`80`–`7FF`)	11	`110xxxxx 10xxxxxx`
3	$[2048, 65535]$ (`800`–`FFFF`)	16	`1110xxxx 10xxxxxx 10xxxxxx`
4	$[65536, 2^{21}-1]$ (`10000`–)	21	`11110xxx 10xxxxxx 10xxxxxx 10xxxxxx`

Procedure: write the code point in binary, pick the row by its range, pad with leading zeroes to the payload size, fill the x positions from left to right (the lowest bits go into the last byte), convert to hex if asked.

Properties#

ASCII compatible: characters $< 128$ have the identical single byte, so any such string is valid UTF-8.
ISO-8859-1 compatible in a simple way: code 1xyyyyyy becomes 1100001x 10yyyyyy; ISO-8859-1 252 (ü) is UTF-8 195 188 (C3 BC), URL-escaped %C3%BC (DBIS Exam WS1516 (not published), Task 3.4).
Self-synchronizing: continuation bytes start with 10; decoding can start anywhere by skipping to the next byte not starting with 10. The number of leading 1s of a first byte equals the number of bytes. All bytes of a multi-byte character are $\geq 128$ .
Overlong encodings are invalid: a code point that fits a shorter form must use it. E.g. 1100000x 10xxxxxx is invalid; C1 BF (code point 127) is invalid, CF BF (code point 1023) is valid; 1100 0001 1000 0000 (code point 64) is invalid (DBIS Exam WS2122 (not published) Task 2.4, DBIS Exam WS2324-2 (not published) Task 4.2). Not every byte sequence is valid UTF-8.
Valid 2-byte sequences: first byte C2–DF (C0, C1 would be overlong), 30 values, second byte 64 values: $30 \cdot 64 = 1920$ (DBIS Exam WS1516 (not published), Task 3.3).
Valid 2-byte sequence check: first byte 110xxxxx with value $\geq$ C2, second byte 10xxxxxx.
The code is prefix-free (Prefix-Free Code) at the byte level.

Example#

$\alpha$ $=$ U+03B1 $= 945$ : 11 bits 01110 110001, template 110xxxxx 10xxxxxx gives 11001110 10110001 $=$ CE B1. $A$ $= 913$ : 11001110 10010001; they differ in one bit (bit 5 of the second byte), so lowercasing Greek letters means setting it (DBIS Exam WS1819 (not published), Task 2.3–2.4).
Euro $= 8364 = 32 \cdot 256 + 172$ $=$ 0010 0000 1010 1100: 3 bytes, 1110 0010 10 000010 10 101100 $=$ E2 82 AC (DBIS Exam WS1617 (not published), Task 3.3).
U+1F60E (sunglasses emoji): 1 1111 0110 0000 1110, 17 bits, 4 bytes, padded to 21: 000 011111 011000 001110, giving 11110000 10011111 10011000 10001110 $=$ F0 9F 98 8E (DBIS Exam WS1920 (not published), Task 2.3).
U+08F0 $=$ 0000 1000 1111 0000 $\to$ 1110 0000 1010 0011 1011 0000 (DBIS Exam WS2324-2 (not published), Task 4.2).

Sources#

Slides 10, p. 27, Slides 10, p. 28, Slides 10, p. 29
DBIS_VL10 > UTF-32, UTF-16, UTF-8, DBIS_VL10 > UTF-8 in detail, DBIS_VL10 > Properties of UTF-8; bytes vs. strings in DBIS_VL09 > Bytes vs. strings
Past exams: DBIS Exam WS1314 (not published) Task 3, DBIS Exam WS1516 (not published) Task 3, DBIS Exam WS1617 (not published) Task 3, DBIS Exam WS1718 (not published) Task 3.3, DBIS Exam WS1819 (not published) Task 2, DBIS Exam WS1920 (not published) Task 2, DBIS Exam WS2122 (not published) Task 2.4, DBIS Exam WS2324-2 (not published) Task 4.2

Connections

Prerequisites know first1

Prefix-Free Code

Mentioned in inline links3

Prefix-Free Code
UTF-8 is prefix-free on the byte level: the leading bits of the first byte (0, 110, 1110, 11110) determine the number of bytes. §
VL10 – Web Applications, Part 2
UTF-8 — variable-byte encoding with 1–4 bytes; ASCII compatible, self-synchronising §
Encodings and Compression
UTF-8 — prefix-free variable-length byte code for Unicode code points §
UTF-8 as a real-world prefix code. UTF-8 encodes Unicode code points with 1–4 bytes. The leading bits of the first byte (0, 110, 1110, 11110) play the role of the unary length prefix, and continuation bytes start with 10, so the code is pr… §

Search in these mentions →