Chapter 12- Measuring Retrieval Quality

Chapter Goal

In Chapter 11, we built a complete multi-stage retrieval pipeline:

Corpus
↓
Candidate Generation
↓
Lexical / Dense / Hybrid Retrieval
↓
Initial Ranking
↓
Reranking
↓
Top Results

But one major question remains:

How do we know whether this system is actually good?

Suppose System A returns:

1. Document A
2. Document B
3. Document C
4. Document D
5. Document E

and System B returns:

1. Document F
2. Document A
3. Document G
4. Document C
5. Document H

Which system is better?

We cannot answer that just by looking at:

BM25 scores
cosine similarity
reranker scores

Those scores belong to the retrieval system itself.

To evaluate the system, we need something external:

What information was actually relevant?

Then we compare:

What the system retrieved
vs.
What it should have retrieved

This is the foundation of retrieval evaluation.

By the end of this chapter, you should understand:

what retrieval evaluation is;
what relevance judgments are;
what ground truth means;
what binary and graded relevance mean;
what Precision is;
what Recall is;
the trade-off between Precision and Recall;
what Precision@K and Recall@K mean;
what Hit Rate@K means;
what Mean Reciprocal Rank (MRR) measures;
what DCG and NDCG measure;
what Average Precision and MAP mean conceptually;
why ranking position matters;
how candidate-generation evaluation differs from final-ranking evaluation;
what offline evaluation is;
what online evaluation is;
why click data can be misleading;
why there is no universally best retrieval metric;
and how these metrics connect to RAG and AI Visibility.

The central idea of this chapter is:

Known Relevant Information
+
Retrieved / Ranked Results
↓
Evaluation Metric
↓
Retrieval Quality

1. What Is Retrieval Evaluation?

Retrieval evaluation is the process of measuring how effectively a retrieval system finds and ranks relevant information.

Suppose a query is:

Persian dentist Toronto

Imagine that, according to our evaluation data, these documents are relevant:

A
C
F
H

The retrieval system returns:

A
B
C
D
E

Now we can compare:

Relevant:
A, C, F, H

Retrieved:
A, B, C, D, E

The system successfully found:

A
C

but missed:

F
H

and retrieved irrelevant items:

B
D
E

Metrics allow us to quantify this behavior.

2. Evaluation Requires a Reference

A retrieval system cannot evaluate itself merely by saying:

Document A score = 0.94

That only tells us:

the system strongly prefers A

It does not tell us whether A is actually relevant.

Evaluation therefore requires some form of reference:

Query
+
Expected Relevant Results

For example:

Query:
Persian dentist Toronto

Relevant:
A
C
F
H

This reference is often called:

ground truth

3. What Is Ground Truth?

In retrieval evaluation, ground truth is the set of relevance judgments used as the reference for evaluating system results.

For example:

Query Q1:
A → Relevant
B → Not Relevant
C → Relevant
D → Not Relevant

The retrieval system is then judged against those labels.

Ground truth may come from:

human assessors
domain experts
curated datasets
historical judgments
task-specific annotation

Ground truth is not necessarily perfect.

It is an evaluation reference.

4. Relevance Judgments

A relevance judgment states how relevant a document is to a query.

The simplest form is binary:

Relevant
Not Relevant

Example:

Query:
best Persian restaurants Toronto

Document A → Relevant
Document B → Relevant
Document C → Not Relevant
Document D → Not Relevant

These judgments make metrics such as Precision and Recall possible.

5. Binary vs. Graded Relevance

Not all relevance is equally strong.

Consider:

Query:
How can I cancel my annual subscription?

Passage A:

Exact instructions for cancelling an annual subscription.

Passage B:

General subscription management information.

Passage C:

Article about subscription pricing.

A binary judgment might say:

A → Relevant
B → Relevant
C → Not Relevant

But that loses an important distinction:

A is much better than B.

So some evaluation systems use graded relevance.

For example:

3 → Highly Relevant
2 → Relevant
1 → Slightly Relevant
0 → Irrelevant

This becomes particularly important for ranking metrics such as NDCG.

6. Precision

Precision asks:

Of the results the system retrieved, how many were relevant?

Formula:

Precision
=
Relevant Retrieved Results
÷
All Retrieved Results

Suppose the system returns:

A
B
C
D
E

and only:

A
C

are relevant.

Then:

Relevant Retrieved = 2
Total Retrieved = 5

So:

Precision = 2 / 5 = 0.40

or:

40%

7. The Intuition Behind Precision

Precision measures how much noise exists in the retrieved set.

High precision means:

Most retrieved results are useful.

Low precision means:

Many retrieved results are irrelevant.

For example:

10 Retrieved
9 Relevant

gives:

90% Precision

This is a very clean result set.

8. Recall

Recall asks:

Of all the relevant information that existed, how much did the system successfully retrieve?

Formula:

Recall
=
Relevant Retrieved Results
÷
All Relevant Results

Suppose four relevant documents exist:

A
C
F
H

The system retrieves:

A
B
C
D
E

Relevant retrieved:

A
C

So:

Recall = 2 / 4 = 0.50

or:

50%

The system found half of the relevant documents.

9. Precision vs. Recall

Precision and Recall answer different questions.

Precision
→ How clean are the retrieved results?

Recall
→ How much relevant information did we successfully find?

Imagine two systems.

System A

Retrieves:

A
C

Both are relevant.

There are actually ten relevant documents in the corpus.

Precision = 100%
Recall = 20%

Very clean, but it misses most relevant information.

System B

Retrieves all ten relevant documents plus ten irrelevant documents.

Precision = 50%
Recall = 100%

It finds everything relevant, but introduces more noise.

Neither metric alone tells the whole story.

10. The Precision–Recall Trade-Off

Retrieving more candidates often improves Recall.

But it may reduce Precision.

For example:

Top 5 results
→ 4 relevant
→ Precision = 80%

If we retrieve 100 results:

12 relevant
88 irrelevant

we may have found more relevant documents overall, but the result set is much noisier.

Retrieve More
↓
Potentially Higher Recall
but
Potentially Lower Precision

This trade-off matters throughout retrieval system design.

11. Why We Need @K Metrics

Search systems usually care most about the top part of the result list.

Suppose:

1. Relevant
2. Relevant
3. Relevant
4. Irrelevant
5. Relevant
...
87. Relevant

A relevant result at rank:

#2

is much more valuable than the same result at:

#87

especially when users or downstream systems only inspect the top few results.

Therefore retrieval evaluation often uses metrics at a cutoff:

@K

For example:

Precision@5
Recall@10
Hit Rate@3

12. What Does K Mean?

K is simply the number of top-ranked results being evaluated.

For example:

K = 5

means:

evaluate only the first 5 results

So:

Precision@5

means:

precision among the top 5 results

and:

Recall@10

means:

how much relevant information was found in the top 10

13. Precision@K

Precision@K measures the proportion of the top K results that are relevant.

Suppose the top 5 results are:

1. Relevant
2. Relevant
3. Irrelevant
4. Relevant
5. Irrelevant

There are:

3 relevant results

among:

5 results

Therefore:

Precision@5
=
3 / 5
=
0.60

or:

60%

14. Recall@K

Recall@K measures how much of all relevant information appears within the top K results.

Suppose six relevant documents exist in total.

The top 5 contain three of them.

Then:

Recall@5
=
3 / 6
=
0.50

So the system found:

50%

of all relevant documents within its first five results.

15. Candidate Recall

Recall is especially important during candidate generation.

Suppose:

10 relevant documents

exist.

The first-stage retriever produces:

Top 100 candidates

and contains nine of the ten relevant documents.

Then:

Recall@100 = 90%

This can be very good candidate generation.

Why?

Because a reranker now has access to:

9 of the 10 relevant documents

It has a strong candidate pool to work with.

But if candidate Recall@100 were:

30%

the reranker would never see most relevant information.

16. Precision Matters More Near the End

Early retrieval often prioritizes:

Recall

Later stages increasingly care about:

Precision at the top

For example:

Corpus
↓
Retriever
↓
Top 500

may optimize for strong Recall@500.

Then:

Reranker
↓
Top 10

may optimize for highly relevant results at the top.

This connects directly to Chapter 11:

Retriever
→ avoid missing useful candidates

Reranker
→ put the best candidates first

17. Hit Rate@K

Sometimes we do not care how many relevant documents appear.

We only care whether the system found at least one relevant result.

This leads to:

Hit Rate@K

For one query:

Top K contains at least one relevant result
→ Hit

Top K contains no relevant results
→ Miss

For example:

Top 5:
Irrelevant
Irrelevant
Relevant
Irrelevant
Irrelevant

Then:

Hit@5 = 1

because at least one relevant result appeared.

18. When Hit Rate Is Useful

Hit Rate is useful when a task only needs one successful result.

For example:

Find the correct support article.

If the correct article is somewhere in:

Top 5

the next system stage may be able to use it.

But Hit Rate does not tell us:

whether the result was rank 1 or rank 5

and does not reward retrieving multiple relevant items.

For that, we need other metrics.

19. Reciprocal Rank

Suppose a query has one especially important relevant answer.

We may care strongly about where the first relevant result appears.

Reciprocal Rank is:

1
÷
Rank of First Relevant Result

Examples:

If the first relevant result is:

Rank 1

then:

Reciprocal Rank = 1 / 1 = 1.0

At:

Rank 2

then:

1 / 2 = 0.5

At:

Rank 5

then:

1 / 5 = 0.2

So the score falls quickly as the first useful result moves down the list.

20. Mean Reciprocal Rank — MRR

Real evaluation usually involves many queries.

For each query:

calculate Reciprocal Rank

Then average the results.

This gives:

Mean Reciprocal Rank, or MRR.

Suppose:

Query 1
First Relevant Result → Rank 1
RR = 1.0

Query 2
First Relevant Result → Rank 2
RR = 0.5

Query 3
First Relevant Result → Rank 4
RR = 0.25

Then:

MRR
=
(1.0 + 0.5 + 0.25) / 3
=
0.583

MRR strongly rewards systems that place the first useful result near the top.

21. What MRR Measures Well

MRR is especially useful when:

finding one correct result quickly

is important.

Examples include:

FAQ retrieval
question answering candidate retrieval
known-item search
support article search

But MRR largely focuses on:

the first relevant result

It does not strongly evaluate the quality of the entire ranked list.

22. Why Position Matters in Ranking Evaluation

Suppose two systems both retrieve five relevant documents.

System A:

1. Relevant
2. Relevant
3. Relevant
4. Relevant
5. Relevant

System B:

1. Irrelevant
2. Irrelevant
3. Irrelevant
4. Irrelevant
5. Irrelevant
...
46. Relevant
47. Relevant
48. Relevant
49. Relevant
50. Relevant

Pure Recall over a sufficiently large result set might eventually treat them similarly.

But from a user perspective:

System A
≫
System B

So ranking metrics need to reward relevant documents appearing earlier.

23. Discounted Cumulative Gain — DCG

DCG introduces two important ideas:

More relevant documents should receive more credit.
Relevant documents appearing earlier should receive more credit.

Suppose relevance is graded:

3 = Highly Relevant
2 = Relevant
1 = Slightly Relevant
0 = Irrelevant

A ranking:

Rank 1 → 3
Rank 2 → 2
Rank 3 → 0
Rank 4 → 1

is better than putting the highly relevant result at the bottom.

DCG therefore applies a position discount.

Relevance Gain
÷
Position Discount

The deeper a document appears:

the less credit it receives

24. Why DCG Uses Graded Relevance

Consider:

Document A
→ perfect answer

Document B
→ useful but incomplete

Document C
→ loosely related

Binary relevance might treat:

A
B
C

all as:

Relevant

But users clearly prefer:

A > B > C

DCG can preserve this distinction through graded relevance.

This makes it useful for evaluating sophisticated ranking systems.

25. NDCG — Normalized Discounted Cumulative Gain

Raw DCG depends on:

how many relevant documents exist
and
their relevance grades

So comparing DCG across different queries can be difficult.

NDCG solves this by normalizing DCG against the best possible ranking.

Actual DCG
÷
Ideal DCG
=
NDCG

The ideal ranking is:

the most relevant documents
in the best possible order

NDCG usually falls between:

0 and 1

where:

1
→ ideal ordering

26. NDCG Example

Suppose relevance grades are:

A → 3
B → 2
C → 1
D → 0

The ideal ranking is:

1. A
2. B
3. C
4. D

Now suppose the system returns:

1. B
2. D
3. A
4. C

It still retrieved useful documents.

But it failed to place the most relevant items in the best positions.

NDCG penalizes that ordering.

Therefore NDCG evaluates both:

relevance quality
+
ranking position

27. NDCG@K

Like other metrics, NDCG is often measured at a cutoff.

For example:

NDCG@10

asks:

How good is the relevance ordering
within the top 10 results?

This is especially useful when only the first part of a ranked list matters.

28. Average Precision

Another ranking metric is Average Precision, often abbreviated AP.

Average Precision rewards systems that place relevant documents early throughout the ranked list.

Each time a relevant document appears
↓
Measure Precision at that rank
↓
Average those Precision values

Suppose relevant results appear at ranks:

1
3
5

We calculate Precision at those positions.

For example:

At rank 1:

1 relevant / 1 result
= 1.0

At rank 3:

2 relevant / 3 results
≈ 0.67

At rank 5:

3 relevant / 5 results
= 0.60

Average Precision averages those values.

29. Mean Average Precision — MAP

When we calculate Average Precision across many queries and then take the mean, we get:

Mean Average Precision, or MAP.

Query 1 → AP
Query 2 → AP
Query 3 → AP
...
↓
Average
↓
MAP

MAP is useful when:

multiple relevant documents

may exist and their ranking positions matter.

30. MRR vs. MAP vs. NDCG

These metrics focus on different aspects of ranking.

MRR

Focuses heavily on:

Where is the first relevant result?

MAP

Focuses on:

Are relevant results consistently ranked early?

NDCG

Focuses on:

Are the most relevant results ranked highest,
especially when relevance has multiple grades?

MRR
→ first useful result

MAP
→ ranking of multiple relevant results

NDCG
→ quality of graded ordering

There is no universally best metric.

31. Precision@K vs. NDCG@K

Suppose top 3 results have relevance grades:

3
2
1

and another system returns:

1
2
3

If all three are considered simply:

Relevant

then both systems have:

Precision@3 = 100%

But clearly:

3, 2, 1

is a better ordering than:

1, 2, 3

NDCG can capture that difference.

Precision@K cannot.

32. Metrics Depend on the Task

Consider three systems.

Search Engine

Users want multiple useful results.

Relevant metrics may include:

NDCG
MAP
Precision@K

Support QA

We mainly need one correct support article near the top.

Useful metrics may include:

MRR
Hit Rate@K

Candidate Generation for RAG

The main goal may be:

Do the top 100 candidates contain the needed evidence?

Then:

Recall@100

may be especially important.

The correct metric depends on the role of the retrieval stage.

33. Retrieval Evaluation for Multi-Stage Systems

A multi-stage system should not necessarily use the same metric at every stage.

Example:

Stage 1:
10 million → 1,000

Goal:

high candidate recall

Metric:

Recall@1000

Stage 2:

1,000 → 100

Goal:

stronger ranking

Possible metric:

NDCG@100

Final stage:

100 → 10

Goal:

excellent top results

Possible metrics:

Precision@10
MRR
NDCG@10

Evaluation should match the purpose of each stage.

34. Evaluation of a Reranker

Suppose Retriever A produces:

100 candidates

and the relevant answer is already included.

Now we compare:

Before reranking:
Relevant result at rank 17

After reranking:
Relevant result at rank 2

Candidate Recall did not change.

The candidate set was already good.

But ranking quality improved dramatically.

Metrics such as:

MRR
NDCG
Precision@K

can capture that improvement.

35. Evaluation of Candidate Generation

Now suppose the relevant document is completely absent from the first-stage candidates.

Before:

Top 100
→ relevant document missing

After improving the retriever:

Top 100
→ relevant document included

This is primarily a:

Recall@100

improvement.

A reranker cannot help if candidate generation fails.

36. What Is a Test Query Set?

Retrieval systems should be evaluated across many queries, not just one example.

A test query set may contain:

Q1
Q2
Q3
...
Q1000

Each query has relevance judgments.

For example:

Q1:
Relevant → A, D, H

Q2:
Relevant → B

Q3:
Relevant → C, E, F

The retrieval system runs all queries.

Metrics are then averaged across the test set.

This provides a more reliable evaluation than inspecting a few hand-picked examples.

37. Query Diversity Matters

A test set should represent real query behavior.

Suppose you evaluate only:

simple exact-match queries

BM25 may look excellent.

But real users may also ask:

natural-language questions
synonym-heavy queries
ambiguous queries
location-specific queries
rare identifier queries
long conversational queries

Evaluation quality depends heavily on whether the query set represents the real task.

38. Offline Evaluation

Offline evaluation means evaluating the retrieval system using a predefined dataset.

Test Queries
+
Relevance Judgments
↓
Run Retrieval System
↓
Calculate Metrics

Advantages:

repeatable
fast to compare systems
safe for experimentation
easy A/B model comparison before deployment

For example:

BM25
vs.
Dense
vs.
Hybrid
vs.
Hybrid + Reranker

can all be evaluated on the same test set.

39. Online Evaluation

Online evaluation measures system behavior with real users.

Possible signals include:

click-through rate
successful task completion
reformulation rate
time to successful result
user satisfaction
conversion

Online evaluation can reveal behavior that offline relevance judgments miss.

But interpreting user behavior is difficult.

40. Clicks Are Not Pure Relevance Labels

Suppose Result A is ranked first.

Result B is ranked fifth.

A receives more clicks.

Does that prove:

A is more relevant?

Not necessarily.

Users are more likely to click highly positioned results simply because they see them first.

This creates:

position bias

Other biases can also exist.

Clicked
≠
Automatically Relevant

User behavior can be valuable, but it must be interpreted carefully.

41. Offline Metrics vs. Online Success

A system can improve:

NDCG@10

without producing a noticeable improvement in business or user outcomes.

Likewise, user satisfaction can improve even if one offline metric changes only slightly.

Offline Evaluation
and
Online Evaluation

answer different questions.

Offline:

Does the ranking match our relevance judgments?

Online:

Does the system actually work better for users?

Strong evaluation programs often use both.

42. Evaluation Metrics Are Not Product Goals

Suppose you optimize aggressively for:

click-through rate

The system might learn to prefer sensational titles.

That could increase clicks but reduce:

quality
trust
task success

Similarly, maximizing one retrieval metric can create unintended trade-offs.

Metric
≠
Goal itself

A metric is a measurement tool.

The real objective remains the quality of the user or system outcome.

43. Retrieval Evaluation in RAG

Now connect retrieval metrics to RAG.

Suppose the correct answer depends on Passage C.

The retriever returns:

Top 5:
A
B
D
E
F

Passage C is missing.

The LLM never receives the required evidence.

This is a retrieval failure.

Now suppose:

Top 5:
A
C
D
E
F

The evidence is available.

If the LLM still answers incorrectly, the failure may occur later.

So RAG evaluation needs to separate:

Retrieval Quality

from:

Generation Quality

44. Context Recall

In RAG systems, one useful conceptual question is:

Did the retrieved context contain the information needed to answer the question?

This is closely related to retrieval Recall.

Required Evidence
↓
Was it retrieved?
↓
Context Recall

If required evidence never reaches the context:

generation quality is constrained

regardless of how capable the LLM is.

45. Context Precision

Another useful question is:

How much of the retrieved context is actually useful?

Suppose a RAG system passes:

10 passages

to the LLM.

Only:

are relevant.

The context contains a lot of noise.

Relevant Context
÷
Retrieved Context
↓
Context Precision

Too much irrelevant context can increase:

context competition
confusion
token cost

So RAG retrieval often needs both:

good Recall
+
good Precision

46. Retrieval Metrics and AI Visibility

Now consider AI Visibility.

Suppose a page exists in the corpus.

For a set of relevant AI queries, we might ask:

How often is the page retrieved?

or:

How often does it appear in the top K candidates?

or:

What is its average rank?

These are retrieval-side visibility questions.

For example:

Page appears in Top 10
for 18 of 100 relevant queries

That tells us something about its retrieval visibility.

But it does not yet tell us whether:

the AI cited it
used it
mentioned the brand
generated the correct claim

Those belong to later stages.

47. Retrieval Presence vs. Final Answer Presence

This distinction is critical.

A source may be:

Retrieved

but not:

Used in the final answer

because:

Retrieved
↓
Ranked
↓
Evidence Selected
↓
Context Constructed
↓
Generation

Each stage can remove or reduce its influence.

Retrieval Visibility
≠
Final AI Visibility

But retrieval evaluation helps us understand one important layer of that pipeline.

48. No Single Metric Tells the Whole Story

Suppose System A has:

Recall@100 = 98%

but poor:

NDCG@10

It finds almost everything relevant, but orders results badly.

System B has:

Precision@5 = 100%

but:

Recall@100 = 35%

Its top results are excellent, but it misses many relevant documents.

Which system is better?

The answer depends on:

what the system needs to do

This is why retrieval evaluation usually uses multiple complementary metrics.

49. A Practical Metric Stack

For a modern multi-stage retrieval system, a useful conceptual evaluation stack might look like:

Candidate Generation
→ Recall@100
→ Recall@500

Initial Ranking
→ NDCG@10
→ NDCG@20

Reranking
→ MRR
→ Precision@5
→ NDCG@10

RAG Retrieval
→ Context Recall
→ Context Precision

The exact metrics depend on the task.

Evaluate each stage according to what that stage is supposed to accomplish.

50. A Complete Simplified Evaluation Example

Suppose the query is:

How can I cancel automatic renewal?

Ground truth says four passages are relevant:

A
C
F
H

The system returns:

1. A
2. B
3. C
4. D
5. F

Precision@5

Relevant results inside the top 5:

A
C
F

So:

Precision@5
=
3 / 5
=
0.60

Recall@5

There are four relevant passages total.

Three were retrieved.

Recall@5
=
3 / 4
=
0.75

Hit Rate@5

At least one relevant result exists in the top 5.

Hit@5 = 1

Reciprocal Rank

The first relevant result is:

Rank 1

So:

RR = 1.0

If we repeat this process across many queries:

Average RR
→ MRR

If relevance grades exist:

Highly Relevant
Relevant
Slightly Relevant
Irrelevant

we could additionally evaluate:

NDCG@5

This one example shows why different metrics reveal different aspects of system quality.

51. Common Misunderstandings

Misunderstanding 1: A high retrieval score means the retrieval system is good

Incorrect. BM25 or cosine scores are internal ranking signals, not evaluation metrics.

Misunderstanding 2: Precision and Recall are the same thing

Incorrect.

Precision
→ quality of retrieved results

Recall
→ coverage of relevant results

Misunderstanding 3: 100% Precision means perfect retrieval

Not necessarily. You could retrieve only one relevant document and miss twenty others.

Misunderstanding 4: 100% Recall means perfect retrieval

Not necessarily. You could retrieve every document in the corpus. You would find all relevant documents but also enormous amounts of noise.

Misunderstanding 5: Precision@10 evaluates the entire result list

Incorrect. It only evaluates the first ten results.

Misunderstanding 6: MRR evaluates all relevant documents equally

Incorrect. MRR mainly cares about the position of the first relevant result.

Misunderstanding 7: NDCG and Precision@K measure exactly the same thing

Incorrect. NDCG considers ranking position and can use graded relevance.

Misunderstanding 8: Candidate retrieval and reranking should always use the same metric

Incorrect. Candidate generation often emphasizes Recall, while later ranking emphasizes top-result quality.

Misunderstanding 9: Clicks are perfect relevance judgments

Incorrect. Clicks can be affected by position, presentation, trust, and other biases.

Misunderstanding 10: One metric is enough to evaluate a search system

Usually not. Different metrics capture different failure modes.

Misunderstanding 11: Good retrieval metrics guarantee good RAG answers

Incorrect. Generation can still fail after good retrieval.

Misunderstanding 12: If a page is retrieved, it is visible in the final AI answer

Incorrect. Later ranking, context selection, and generation stages can still remove its influence.

52. How the Concepts Connect

Chapter 7 introduced:

Corpus
↓
Retrieval
↓
Ranking
↓
Results

Chapter 8 introduced lexical retrieval:

BM25

Chapter 9 introduced dense retrieval:

Embedding Similarity

Chapter 10 introduced hybrid retrieval:

Lexical
+
Dense
↓
Fusion

Chapter 11 introduced:

Ranking
+
Reranking

Chapter 12 now adds:

Evaluation

The complete conceptual pipeline becomes:

Corpus
↓
Candidate Generation
↓
Lexical / Dense / Hybrid Retrieval
↓
Ranking
↓
Reranking
↓
Top Results
↓
Evaluation

And evaluation asks different questions at different stages:

Candidate Generation
→ Did we retrieve the relevant information?

Ranking
→ Did we put the best information near the top?

Reranking
→ Did the stronger model improve the ordering?

RAG
→ Did the necessary evidence reach the LLM?

Mental Model

Imagine you hire someone to search a warehouse for ten specific boxes.

The worker returns with twenty boxes.

You ask two questions.

First:

Of the twenty boxes returned,
how many are actually correct?

That is like:

Precision

Second:

Of the ten correct boxes in the warehouse,
how many did the worker find?

That is like:

Recall

Now suppose the worker also places the boxes in order.

If the most useful box is placed first:

excellent

If it is placed at position 97:

technically found
but practically difficult to use

That is why ranking metrics matter.

MRR asks something like:

How quickly do we encounter the first correct box?

NDCG asks something closer to:

Did we put the most useful boxes
in the best positions?

Candidate-generation evaluation asks:

Did the good boxes survive the first filtering stage?

Reranking evaluation asks:

Did the deeper evaluation move the best boxes upward?

The simplest mental model is:

Precision
→ How clean?

Recall
→ How complete?

MRR
→ How soon is the first good result?

NDCG
→ How good is the overall top ordering?

And the simplest reliable definition is:

Retrieval evaluation compares a system's retrieved and ranked results against known relevance judgments. Different metrics measure different properties: Precision measures result quality, Recall measures coverage, MRR emphasizes the first relevant result, and NDCG evaluates how well highly relevant results are positioned near the top.

Chapter 12 Recap

Ground Truth

The relevance judgments used as the reference for evaluation.

Query
+
Known Relevant Results

Binary Relevance

Relevant
or
Not Relevant

Graded Relevance

Different levels of usefulness.

3 → Highly Relevant
2 → Relevant
1 → Slightly Relevant
0 → Irrelevant

Precision

Relevant Retrieved
÷
All Retrieved

Question:

How clean are the retrieved results?

Recall

Relevant Retrieved
÷
All Relevant

Question:

How much relevant information did we find?

Precision@K

Precision within the first K ranked results.

Relevant Results in Top K
÷
K

Recall@K

The proportion of all relevant documents found within the top K.

Relevant Results in Top K
÷
All Relevant Results

Hit Rate@K

Whether at least one relevant result appears within the top K.

Relevant Result Found?
→ Hit

None Found?
→ Miss

Reciprocal Rank

1
÷
Rank of First Relevant Result

Mean Reciprocal Rank — MRR

Average Reciprocal Rank across multiple queries.

Best when:

getting the first useful answer high

is important.

DCG

A ranking metric that rewards:

higher relevance
+
higher ranking position

NDCG

Normalized DCG.

Actual Ranking Quality
÷
Ideal Ranking Quality

Useful when relevance has multiple levels.

Average Precision — AP

Averages Precision values at positions where relevant results appear.

Mean Average Precision — MAP

Average AP across many queries.

Useful when multiple relevant documents exist and ranking order matters.

Candidate Recall

Measures whether relevant documents survive initial retrieval.

Retriever
↓
Top N Candidates
↓
How much relevant information survived?

Offline Evaluation

Test Queries
+
Relevance Labels
↓
Metrics

Useful for controlled system comparisons.

Online Evaluation

Measures performance with actual users.

Possible signals:

clicks
task completion
reformulation
satisfaction
conversion

Context Recall

In RAG:

Did the retrieved context contain
the evidence required to answer?

Context Precision

In RAG:

How much of the retrieved context
was actually useful?

Metric Selection

Different stages need different metrics.

Candidate Generation
→ Recall@K

Top Ranking
→ Precision@K / NDCG

First Correct Result
→ MRR

RAG Context
→ Context Recall + Context Precision

The complete evaluation view is:

Corpus
↓
Retriever
↓
Candidate Set
↓
Ranker
↓
Reranker
↓
Top-K Results
↓
Compare with Relevance Judgments
↓
Precision / Recall / MRR / NDCG / MAP
↓
Measure Retrieval Quality

We now know how to answer:

Did the system find the right information?

and:

Did it put that information in the right positions?

The next step is to go beyond retrieval mechanics and start examining how systems interpret the user's information need before retrieval begins.

That leads into Chapter 13.

The Single Most Important Idea from Chapter 12

There is no single number called “retrieval quality.” Precision measures how much of what we retrieved is relevant, Recall measures how much relevant information we successfully found, and ranking metrics such as MRR and NDCG measure whether the most useful results appear early enough to matter. The right metric depends on what each stage of the retrieval pipeline is supposed to accomplish.