ConceptDBISAI-written · not yet proofread

Language Model

Also Neural Language Model

A language model computes, for a sequence of previous tokens, a probability distribution over the vocabulary for the next token; it is count-based (n-gram model) or neural.

Properties#

Tokens. One text can be tokenized in different ways; a token may include spaces and punctuation. For “To be, or not to be”:

words: To be or not to be
subwords: To be , or not to be (modern language models)
characters: T o b e , … (the lecture’s n-gram models)

Variants.

	Count-based (N-Gram Model)	Neural language model
How $\Pr$ is computed	normalized counts of token sequences in a training text	forward pass through a neural network
Context size	small $n$ ; $\lvert V \rvert^n$ n-grams make large $n$ infeasible	very large, up to $10^6$ tokens
Unseen contexts	no distribution; smoothing (random next token)	the network computes a distribution for every prompt

A neural language model with a very large network is a large language model (LLM). All of these, including LLMs, are “the same kind of model”, only trained and computed differently.

Long prompts: if the prompt has more than $n$ tokens, only the last $n$ are considered, or a subset of the text is selected via Retrieval-Augmented Generation.
Generating text: repeatedly pick a next token from the distribution (e.g. via cumulative counts, N-Gram Model > Formula), append it to the prompt, repeat. LLM services return the next $k$ tokens at once.
Base vs. instruct models, system prompt, temperature and the API: see Large Language Model.

Example#

Sources#

Slides 12, p. 4 (tokens), Slides 12, p. 9 (limits of n-gram models), Slides 12, p. 10 (neural language models), Slides 12, p. 12 (context size, prompt)
DBIS_VL12 > Tokens, DBIS_VL12 > Neural and large language models, DBIS_VL12 > Terminology
Exam: DBIS Exam WS2324-2 (not published) Task 5.2 (simplistic next-word model as one linear layer over $n$ embeddings), Task 5.3 (top- $k$ sampling)

Connections

Mentioned in inline links4

Large Language Model
A neural language model computes the distribution $\Pr(v \mid v_1, \dots, v_n)$ over the next token $v \in V$ with a forward pass through a neural network. If that network is very large, the model is a large language model (LLM). §
N-Gram Model
Let $V$ be the vocabulary (set of distinct tokens) and $n \in \mathbb{N}$ . An n-gram is a sequence of $n$ tokens. An n-gram model is a language model that gives, for each sequence $v_1 \dots v_{n-1} \in V^{n-1}$ of previous tokens (the con… §
Size: there are $|V|^n$ possible n-grams ( $|V|^{n-1}$ contexts), exponential in $n$ , so large $n$ is infeasible. Solution: neural language models. §
VL12 – N-gram Models, LLMs, RAG
A neural language model computes a probability distribution over $V$ (the next token) from given previous tokens $v_1, \dots, v_n \in V$ , just like an n-gram model, but with a forward pass through a neural network instead of simple counts.… §
Language Model — distribution over the next token; neural models and LLMs, base vs. instruct, context size, prompt, system prompt §
Word Embeddings and Language Models
Language Model — distribution over the next token given the previous tokens §
All language models here compute $\Pr(\text{next token} \mid \text{previous tokens})$ ; they differ in how. Tokens can be words, subwords (modern LLMs) or characters (lecture n-gram code). §

Search in these mentions →