BPref

Properties#

• Motivation: for very large collections, judging all documents is infeasible; competitions use pooling (judge only the top-$k$ results of each participant), see Ground Truth. Measures like AP need complete judgments; bpref uses only judged documents.
• $\mathrm{bpref} \in [0, 1]$, since $|NR(d)| \le \min(|R|, |N|)$ (at most $|R|$ by definition, at most $|N|$ trivially).
• $\mathrm{bpref} = 1$ iff every judged relevant document is in the result list and no judged non-relevant document comes before any of them.
• If not all relevant documents are in the result list, $\mathrm{bpref} < 1$: the $|RR| < |R|$ terms are each at most 1 and the sum is divided by $|R|$ (exam WS 23/24 retake, Task 1.3 (3)).
• Why $\min(|R|, |N|)$ and not $|N|$? The score of $d$ already drops to 0 when $|R|$ judged non-relevant documents (or all of $N$, if fewer) come before it, so a high bpref is harder to achieve than with $|N|$, which is typically much larger than $|R|$ (DBIS_VL02 > Binary Preference (bpref)).
• Why $1/|R|$ and not $1/|RR|$? A relevant document missing from the result list must be punished, for the same reason that AP sets $\mathrm{P@}R_i = 0$ for missing relevant documents.

Example#

Sources#

• Slides 02, p. 27 (definition), Slides 02, p. 28 (example), Slides 02, p. 29 (finer points)
• DBIS_VL02 > Binary Preference (bpref)
• Exams: DBIS Exam WS2324-2 (not published) Task 1.3 (3)