ConceptDBISAI-written · not yet proofread

URL Encoding

Also Percent-Encoding · URL Decoding

URL encoding (percent-encoding) escapes every character that is not allowed or has a special meaning in a URL by the bytes of its encoding (by default UTF-8), each written as % followed by two hex digits; in form data a space becomes +.

Know these firstCharacter Encoding

Formula#

For a character $c$ with code point $u$ :

If $c$ is allowed and not used as a delimiter, keep it.
Otherwise compute the UTF-8 bytes $b_1 \dots b_k$ of $u$ ( $k \in \{1,2,3,4\}$ ) and write % $\mathrm{hex}(b_1)$ … % $\mathrm{hex}(b_k)$ .
In form data (application/x-www-form-urlencoded, i.e. what an HTML Form sends) a space is written as +; elsewhere as %20.

UTF-8 Encoding of Non-ASCII Characters#

For code points $> 127$ , UTF-8 uses multi-byte encoding; the payload bits of the code point fill the x positions from the right:

Code point range	UTF-8 bytes	Byte pattern
U+0000–U+007F ( $[0, 127]$ )	1	`0xxxxxxx`
U+0080–U+07FF ( $[128, 2047]$ )	2	`110xxxxx 10xxxxxx`
U+0800–U+FFFF ( $[2048, 65535]$ )	3	`1110xxxx 10xxxxxx 10xxxxxx`
U+10000–U+1FFFFF ( $[65536, 2^{21}-1]$ )	4	`11110xxx 10xxxxxx 10xxxxxx 10xxxxxx`

UTF-8 itself (code points, validity, decoding) belongs to Encodings and Compression.

Properties#

Space: + in query strings (forms), %20 in general.
Reserved characters (?, &, =, #, …) are encoded when used as data, not as delimiters: the query a&b is sent as ?q=a%26b.
Non-ASCII characters are first converted to their UTF-8 byte sequence, then each byte is percent-encoded: ä → %C3%A4. Under ISO-8859-1 the same ä would be the single byte %E4; UTF-8 is the default of browsers and fetch.
The page’s encoding is declared in its head, <meta charset="utf-8">.
Every URL-encoded byte of a multi-byte character is $\geq$ 0x80, so a % code $\geq$ %80 is always part of a multi-byte UTF-8 sequence.
Server side: replace + by a space (Sheet 9), then turn each %XY into the byte 0xXY (e.g. Python bytes.fromhex("C3A4")) and decode the resulting bytes as UTF-8 (Sheet 10). Decoding byte by byte into characters is wrong: %C3%A4 would become Ã¤.
URL encoding hides content from the human eye but not from the browser: %3C%73%63%72%69%70%74 is <script (Code Injection Protection).

Example#

Sources#

Slides 09, p. 20 (allowed characters, +, %C3%A4)
Slides 10, p. 28 (UTF-8 patterns), pp. 31–32 (URL encoding and decoding), p. 19 (URL-encoded <script>)
DBIS_VL09 > URL encoding and decoding, DBIS_VL10 > UTF-8 in detail, DBIS_VL10 > URL encoding and decoding, DBIS_VL10 > Code injection
Sheet 10, Exercise 1.4
DBIS Exam WS1516 (not published) T3.4, DBIS Exam WS1617 (not published) T3.3, DBIS Exam WS2324-2 (not published) T4.2

Connections

Mentioned in inline links9

AJAX and Fetch API
Special characters in the URL are URL-encoded as UTF-8 bytes by fetch (ä → %C3%A4). + and & in the input are not escaped by plain string concatenation; encodeURIComponent(q) escapes them too. §
Character Encoding
UTF-8: variable-byte encoding of Unicode code points with 1–4 bytes, ASCII compatible (ä = C3 A4, € = E2 82 AC); byte patterns in URL Encoding and Encodings and Compression. §
Where the encoding is declared on the web: in the HTML head, <meta charset=“utf-8”>, and in the response header, Content-Type: text/html; charset=utf-8. fetch and forms URL-encode special characters as UTF-8 bytes by default (URL Encoding). §
Code Injection Protection
Link in a mail: …index.html?user=guest<script>alert(“Ha!”)</script>. The <script> part can be made less suspicious by URL-encoding it: …index.html?user=guest%3C%73%63%72%69%70…. §
HTML Form
The values are URL-encoded: a space becomes +, ä becomes %C3%A4. The server must decode them. §
HTTP Protocol
GET parameters are in the URL (visible, bookmarkable, limited length) and must be URL-encoded. POST sends arbitrarily large data in the body along with the URL. There are many more request types and headers. §
HTTP Server
parse the request line GET /<path>?<params> HTTP/1.1: the path is the text between GET / and HTTP/1.1; URL-decode the parameters, §
VL09 – Web Applications, Part 1
With forms, the query becomes part of the URL, so forbidden characters must be escaped: URL encoding (and decoding). §
URL Encoding — restricted URL character set, + for space, %C3%A4 for ä §
VL10 – Web Applications, Part 2
In fetch, special characters are URL-encoded: each byte becomes % followed by its code in hexadecimal; the default encoding is UTF-8. E.g. ä has UTF-8 code C3 A4 and is URL-encoded as %C3%A4. §
URL Encoding — percent-encoding of the UTF-8 bytes of special characters (ä → %C3%A4) §
Web Applications
URL Encoding — escaping query parameters as %-encoded UTF-8 bytes, + for space §
A query like bär crosses four representations: the string in the input field → URL-encoded UTF-8 bytes in the request line (b%C3%A4r, URL Encoding) → bytes received by the server, URL-decoded and then decoded as UTF-8 to the string bär (Ch… §

Search in these mentions →