ConceptDBISAI-written · not yet proofread

Ground Truth

Also Benchmark · Relevance Judgments

The ground truth of a query is the set of IDs of all documents relevant for it; a set of queries together with their ground truth is a benchmark.

Properties#

Every evaluation measure compares a ranked result list with the ground truth: P@k, P@R, AP, MAP, nDCG and bpref. $R = |$ ground truth of $q|$ is the number of relevant documents.
The ground truth is a set, the evaluated results are ranked lists. Alternatives fail (DBIS_VL02 > Set vs. Ranked List): an output set has no natural cut-off; the relevant documents usually have no full order (only levels, cf. nDCG); and absolute score values mean little, they only serve to rank.
Building high-quality benchmarks of sufficient size is an important and time-consuming part of research: relevance is judged by humans.
Incomplete judgments: for very large collections, judging all documents is infeasible. A common approach in competitions is pooling: judge only the top- $k$ results of each participant. The other documents stay unjudged, and measures that need complete judgments (P@k, AP) become unreliable; bpref ignores unjudged documents.
A benchmark used for tuning is no longer a fair test: split into a training and a test benchmark, see Overfitting (IR Evaluation).

Example#

Sources#

Slides 02, p. 21 (ground truth, benchmark), Slides 02, p. 27 (pooling), Slides 02, pp. 31–32 (set vs. ranked list)
DBIS_VL02 > Evaluation: Ground Truth, DBIS_VL02 > Binary Preference (bpref), DBIS_VL02 > Set vs. Ranked List

Connections

Part of topics1

Evaluation

Mentioned in inline links8

Average Precision
Let a query have $R$ relevant documents (ground truth), and let $R_1 < R_2 < \dots < R_R$ be their sorted positions in the result list. The average precision is the average of the $R$ values $\mathrm{P@}R_i$ (Precision at k at the position… §
BPref
Motivation: for very large collections, judging all documents is infeasible; competitions use pooling (judge only the top- $k$ results of each participant), see Ground Truth. Measures like AP need complete judgments; bpref uses only judged… §
Discounted Cumulative Gain
$\mathrm{iDCG@}k$ is the $\mathrm{DCG@}k$ of the ideal ranking (all relevant documents of the ground truth sorted by decreasing relevance), and the normalized DCG is §
Mean Average Precision
Let $Q$ be the set of queries of a benchmark (ground truth for each query) and $\mathrm{AP}(q)$ the average precision of the system’s result list for $q \in Q$ . The mean average precision is §
Overfitting (IR Evaluation)
Overfitting is tuning parameters or methods to achieve good results on a given benchmark (queries + ground truth). The reported quality then reflects the adaptation to those specific queries and judgments, not the quality on new queries. §
Precision at k
Let a query have the ground truth $G$ (set of relevant documents), $R = |G|$ , and let $r_1, r_2, \dots$ be its ranked result list. For $k \ge 1$ , §
VL02 – Ranking and Evaluation
For each query of a set of queries, the set of IDs of all documents relevant for that query is its ground truth. A set of queries together with their ground truth is a benchmark. (Slide 21) §
Ground Truth — relevant-document sets per query; with the queries it forms a benchmark §
Evaluation
A ranking is evaluated by comparing its result list with a ground truth (the relevant documents per query). Binary relevance: P@k, P@R and AP per query, their means (MP@k, MP@R, MAP) per benchmark. Graded relevance: nDCG. Incomplete judgme… §
Ground Truth — relevant documents per query; with the queries it forms a benchmark §

Search in these mentions →