VL09 Web Applications, Part 1

Organisation#

Feedback on ES8 (fuzzy prefix search) was mostly positive; only about 20% of the participants submit the (non-mandatory) sheets (Slides 3–4). The plan for the web-app sheets (Slide 2): ES9 builds a simple static application that shows fuzzy prefix matches for the query (using the code from ES8), ES10 makes it dynamic.

Socket Communication#

What is a socket#

Setting: two programs (processes) communicate with each other, possibly and often on two different machines. In a typical web application:

• the browser asks for a static web page (this lecture),
• code in the web page asks for dynamic contents (Lecture 10).

Request–response protocol#

Server implementation in Python#

Every major language has a standard library for socket communication: Python socket, Java java.net.ServerSocket, C++ boost::asio (asio = asynchronous IO). Basic server code is provided; the rest is written in the lecture and not published, because the tricky details are only learned by developing the code yourself.

from socket import *
server_socket = socket(AF_INET, SOCK_STREAM)
server_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
server_socket.bind(("0.0.0.0", port))
server_socket.listen()
connection, client_address = server_socket.accept()
connection.settimeout(5.0)
request_data = bytearray()
while (...): request_data.extend(connection.recv(num_bytes))
connection.sendall(...)   # answer data as byte array
connection.close()

• AF_INET, SOCK_STREAM: IPv4 addresses and TCP (comment in the course’s server code).
• SO_REUSEADDR: allows reusing the port when the program is restarted after a crash.
• bind(("0.0.0.0", port)): on which machine and port to listen for connections.
• accept() blocks until a client connects and returns a new socket connection for this client plus its address.
• recv(num_bytes) returns at most num_bytes bytes.

Client side and testing#

For a web application it suffices to implement the server; the web browser plays the role of the client. For testing:

telnet <host> <port>            # connect to host on given port, type the request by hand
curl -v http://<host>:<port>    # send an HTTP request; -v prints headers etc.

Entering http://<host>:<port> in the browser’s address bar establishes a connection that speaks HTTP (next section).

Bytes vs. strings#

The data received from or sent to a client are bytes. In Python, use bytes (immutable, sufficient here) or bytearray (modifiable). Elsewhere in the code, strings are more convenient. Converting between the two requires an encoding, i.e. a specification of how characters correspond to bytes:

"knödel".encode("utf-8")    # b"kn\xc3\xb6del"   (ö becomes the two bytes C3 B6)
b"\xc3\xb6".decode("utf-8") # "ö"

For this lecture and ES9, UTF-8 is assumed; it is explained in detail in Lecture 10.

Hypertext#

HTTP#

HTTP (Hypertext Transfer Protocol) is the protocol the browser uses to communicate with a (web) server. A typical request, where /search.html is the part of the URL after http://<host>:<port>:

GET /search.html HTTP/1.1
...

A typical response: status line, headers, an empty line (i.e. \r\n\r\n before the content bytes), then the content:

HTTP/1.1 200 OK
Content-Length: 137
Content-Type: text/html

... the 137 bytes of the content ...

Request types and status codes#

• Besides GET there are many more request types, e.g. POST, which can send arbitrarily large data along with the URL.
• Besides 200 OK there are many more status lines, e.g.
  • HTTP/1.1 404 Not found: the requested resource does not exist,
  • HTTP/1.1 403 Forbidden: the resource exists but may not be accessed by this client.

The ES9 server must implement 404 and 403.

Media types#

Browser development console#

Opened with F12 or Ctrl+Shift+I (Chrome, Firefox). Extremely useful for debugging. Relevant tabs:

• Network: requests sent and results received,
• Elements: elements of the HTML page (Lecture 10),
• Console: output from the JavaScript (Lecture 10).

HTML#

HTML (Hypertext Markup Language) specifies the content of a web page. It is an XML-like language with the general structure:

<html>
  <head>
    ... meta information + includes ...
  </head>
  <body>
    ... contents of the page ...
  </body>
</html>

Example tags in <head>, which include style information and code:

<link rel="stylesheet" type="text/css" href="..."/>
<script src="..."></script>

Example tags in <body>:

CSS#

CSS (Cascading Style Sheets) specifies style information (layout, font, colour, …) independently of the content of the page. It has its own simple syntax; e.g. all level-1 headings blue and bold:

h1 { color: blue; font-weight: bold }

CSS is used by practically every website. For the sheet, the page should get a reasonably “nice” look and feel with CSS.

Forms#

Purpose of an HTML form: upon a user action (typically a click on a button), load a new HTML page with new or updated information.

<form>
  <input type="text" value="xyz" name="abc"/>
  <input type="submit" value="Search"/>
</form>

• The second <input> is shown as a button labelled “Search”.
• Clicking it makes the browser append ?abc=xyz (input name = current input value) to the URL and issue the corresponding GET request.
• With <form action="prefix"> the URL path is changed, i.e. the request goes to prefix?abc=xyz.

Template variables#

A SERP (Search Engine Result Page) should be the original page, with the input field showing the query and augmented by the results. This is realised with template variables in the HTML file:

<form>
  <input type="text" value="%QUERY%" name="query"/>
  <input type="submit" value="Search"/>
</form>
%RESULT%

After computing its result, the server substitutes %QUERY% by the query and %RESULT% by the result (the fuzzy prefix matches) before sending the page.

URL encoding and decoding#

• Only a restricted character set is allowed in a URL: a-z A-Z 0-9 $ % / - _ . + ! * and a few more. Forbidden are e.g. space, ä, ã, â.
• With forms, the query becomes part of the URL, so forbidden characters must be escaped: URL encoding (and decoding).
• A space becomes +; ä becomes %C3%A4 (UTF-8 bytes) or %E4 (ISO-8859-1 byte). Details in Lecture 10.

For ES9 the server must replace every + in the URL by a space; percent-encoded special characters like ä can be ignored until ES10.

Reusing connections#

• Most browsers keep the connection to the server open for future requests; request header Connection: keep-alive.
• The ES9 server serves only one request at a time. This is a problem if serving a request takes time and the browser sends a new request over the same connection while the server is still busy.
• Fix: send the header Connection: close with the response; the browser then uses a new connection for its next request.

References#

Slide 22: Wikipedia articles on network sockets, HTTP, internet media types, HTML, CSS, HTML forms, web template systems, percent-encoding and the list of HTTP header fields.

Concepts#

• HTTP Protocol — request line, status line, headers, 200/403/404, GET vs. POST, Connection header
• HTML Structure — <html>/<head>/<body> skeleton and basic tags
• HTML Forms — form submission appends ?name=value to the URL and issues a GET request
• URL Encoding — restricted URL character set, + for space, %C3%A4 for ä
• Network Socket — endpoint of a communication channel, identified by host and port; bind/listen/accept/recv/send
• Bytes vs. Strings (Character Encoding) — sockets transfer bytes; encode/decode with an explicit encoding
• HTTP Server — accepts connections on a server socket, parses the request line, computes the resource and sends status line, headers and body
• Media Type (MIME Type) — standard content-type names sent in Content-Type
• CSS (Cascading Style Sheets) — separate styling; most specific rule wins
• Template Variables — %QUERY%/%RESULT% substituted by the server to build the SERP