ConceptDBISAI-written · not yet proofread

Overfitting (IR Evaluation)

Overfitting in IR evaluation means tuning parameters or methods to achieve good results on a given benchmark, so that the reported quality overestimates the quality on unseen queries.

Know these firstGround Truth

Formula#

Let $M(\theta, Q)$ be a quality measure (e.g. MAP) of a system with parameters $\theta$ (e.g. the BM25 parameters $k, b$ ) on the query set $Q$ . Choosing

\theta^* = \arg\max_\theta M(\theta, Q_{\text{test}})

on the same queries that are reported makes $M(\theta^*, Q_{\text{test}})$ an overestimate of $M(\theta^*, Q_{\text{new}})$ on unseen queries $Q_{\text{new}}$ .

Properties#

Extreme case: for each benchmark query, output exactly the relevant documents from the ground truth. Perfect scores, worthless system.
Avoidance (lecture): develop and tune on a training set; the actual evaluation is on a test set that must not be used, or is not available, during development (as in a real search engine or a competition). ES2: tune $k$ and $b$ on the training benchmark, then evaluate once on the test benchmark.

Example#

Sources#

Slides 02, p. 30, Slides 02, p. 6 (ES2), Slides 03, p. 5 (ES2 results)
DBIS_VL02 > Overfitting, DBIS_VL02 > Organisation and Exercise Sheet 2

Connections

Part of topics1

Evaluation

Prerequisites know first1

Ground Truth

Mentioned in inline links5

BM25
Tuned on the ES2 training benchmark, values of $b$ close to (but not) zero and $k \approx 1$ worked best; $k = 1$ maps $\mathrm{tf}^* \in [1, 2)$ for $\mathrm{tf} > 0$ (at $b = 0$ ). Tuning must happen on a training set, with a single final… §
Ground Truth
A benchmark used for tuning is no longer a fair test: split into a training and a test benchmark, see Overfitting (IR Evaluation). §
Mean Average Precision
Tuning parameters to maximize MAP on the benchmark used for reporting inflates it, see Overfitting (IR Evaluation). §
VL02 – Ranking and Evaluation
Tuning parameters or methods to achieve good results on a given benchmark. (Slide 30, Overfitting (IR Evaluation)) §
Overfitting (IR Evaluation) — tuning on the benchmark; avoided by a training/test split §
Evaluation
Overfitting (IR Evaluation) — tuning on the evaluation benchmark inflates scores; use a training/test split §
Honest evaluation. Parameters such as $k, b$ of BM25 are tuned on a training benchmark; the test benchmark is used once at the end (Overfitting (IR Evaluation)). Typical numbers (ES2): baseline MAP 42 %, best tuned 48 %. §

Search in these mentions →