Character Encoding

Formula#

In Python 3, bytes (immutable) or bytearray (modifiable) vs. str:

"knödel".encode("utf-8")       # b"kn\xc3\xb6del"   (ö becomes the two bytes C3 B6)
b"\xc3\xb6".decode("utf-8")    # "ö"
b"\xc3\xa4".decode("ISO-8859-1")  # "ä"  (wrong charset)
len("ä"), len("ä".encode("utf-8"))   # 1, 2

The encodings of the course:

• ASCII: 1 byte, codes 0–127 (32–126 printable, 0–31 and 127 control characters).
• ISO-8859-1: 1 byte; 0–127 as ASCII, 128–255 the special characters of most European languages (ä = E4). Pre-Unicode, one of many context-dependent variants of 128–255.
• UTF-8: variable-byte encoding of Unicode code points with 1–4 bytes, ASCII compatible (ä = C3 A4, € = E2 82 AC); byte patterns in URL Encoding > UTF-8 Encoding of Non-ASCII Characters and Encodings and Compression.
• UTF-16 (Java String, 2 or 4 bytes per code point) and UTF-32 (always 4 bytes).

Properties#

• Length in bytes ≠ length in characters. "knödel" has 6 characters but 7 UTF-8 bytes. Anything that counts bytes, e.g. the Content-Length header of an HTTP response, must be computed on the encoded bytes (HTTP Server).
• Where the encoding is declared on the web: in the HTML head, <meta charset="utf-8">, and in the response header, Content-Type: text/html; charset=utf-8. fetch and forms URL-encode special characters as UTF-8 bytes by default (URL Encoding).
• Mojibake. Decoding with the wrong charset shows the bytes as other characters: the UTF-8 bytes C3 A4 of ä read as ISO-8859-1 are Ã (C3) and ¤ (A4), so a page declared iso-8859-1 shows ä. Invalid UTF-8 bytes are displayed as �.
• Double encoding. Decoding with the wrong charset and re-encoding produces new bytes: ä misread as ä and encoded to UTF-8 gives C3 83 C2 A4.
• What an editor or terminal shows depends on the encoding it uses. xxd <file> (-g 1 byte groups, -b binary) shows the raw bytes independently of any encoding.
• Language specifics: C++ std::string is an array of 1-byte char (holds UTF-8 bytes), std::wstring of wchar_t (code points); Java String is UTF-16, so "😀".length() is 2 and charAt returns UTF-16 units; Python b"…" vs. "…" with encode/decode.

Example#

Sources#

• Slides 09, p. 10 (bytes vs. strings)
• Slides 10, pp. 24–25 (ASCII, ISO-8859-1), p. 27 (UTF-32/16/8), pp. 31–32 (charset of a page), p. 33 (xxd), pp. 34–36 (C++, Java, Python)
• DBIS_VL09 > Bytes vs. strings, DBIS_VL10 > Unicode, DBIS_VL10 > UTF-32, UTF-16, UTF-8, DBIS_VL10 > URL encoding and decoding, DBIS_VL10 > Encoding in files, DBIS_VL10 > Encoding in C++, Java and Python
• Sheet 10, Exercise 1.4