Chapter 7- Information Retrieval Fundamentals

Chapter Goal

In Chapters 1–6, we focused mainly on the language model itself.

We learned:

Text
↓
Tokens
↓
Numerical Representations
↓
Transformer Processing
↓
Training
↓
Learned Parameters
↓
Generation
↓
Assistant Behavior

But a language model does not always need to rely only on information represented in its learned parameters or manually supplied in the current prompt.

Modern systems often need to find external information.

For example, a user may ask:

What does our company's latest pricing document say?

or:

Which restaurant in this database is closest to the user?

or:

Find documents related to electric vehicle battery recycling.

Before the system can answer, it may first need to locate relevant information from a larger collection.

This is the domain of Information Retrieval, usually abbreviated as IR.

At the simplest level:

User Query
↓
Search a Collection
↓
Find Relevant Information
↓
Return Results

Information Retrieval is a foundational field behind:

search engines
document search
site search
enterprise search
vector search
RAG
AI search
retrieval-augmented assistants

This chapter introduces the fundamental concepts.

By the end of the chapter, you should understand:

• what Information Retrieval is;
• what retrieval means;
• what a retriever is;
• what a corpus is;
• what documents and passages are;
• what a query is;
• the difference between a user query and an internal Attention Query;
• what crawling is;
• what indexing is;
• what a search index is;
• what candidate documents are;
• what relevance means;
• what a relevance score is;
• the difference between query–document and query–passage relevance;
• what candidate generation is;
• what a candidate set is;
• what ranking is;
• what reranking is;
• and how these components form a basic retrieval pipeline.

The central pipeline of this chapter is:

Query
↓
Candidate Generation
↓
Retrieval
↓
Ranking
↓
Results

Later chapters will explain the different techniques used inside this pipeline.

1. Why Information Retrieval Exists

Imagine a company has:

500,000 documents

and a user asks:

What is our refund policy for annual subscriptions?

It would be inefficient to place all 500,000 documents into the model's context.

Instead, the system should identify the small number of documents or passages most likely to contain the answer.

Large Information Collection
↓
User Query
↓
Find Relevant Items
↓
Small Relevant Subset

This is the central problem of Information Retrieval.

The system begins with:

too much information

and tries to identify:

the information most relevant to the current need

2. What Is Information Retrieval?

Information Retrieval is the process of finding information relevant to a user's information need from a collection of available information.

Information Retrieval finds relevant items from a larger collection in response to a query.

For example:

Query:
best treatment options for seasonal allergies

Collection:
100,000 medical documents

Retrieval:
find documents or passages relevant to the query

The collection could contain:

web pages
articles
product records
legal documents
academic papers
support articles
internal company documents
database records
passages

Information Retrieval is not limited to the public web.

Any sufficiently large searchable collection can require retrieval.

3. Retrieval

Retrieval is the act of finding and returning potentially relevant information from a collection.

Suppose a user asks:

How much does the Pro plan cost?

A system may search a product documentation collection and retrieve:

Pricing Page
Billing FAQ
Subscription Documentation

Query
+
Collection
↓
Retrieval
↓
Relevant Items

The word retrieval is important because the information already exists somewhere in the collection.

The system is trying to locate it.

This distinguishes retrieval from generation.

4. Retrieval vs. Generation

Retrieval

Retrieval finds existing information.

Query
↓
Search Collection
↓
Existing Document or Passage

Generation

Generation creates a sequence of output tokens.

Context
↓
Next-Token Prediction
↓
Generated Text

Therefore:

Retrieval
≠
Generation

For example, a search engine may retrieve:

Document A
Document B
Document C

without writing a natural-language answer.

An LLM may generate:

Paris is the capital of France.

without retrieving an external document.

Later, RAG will combine both:

Retrieval
↓
Relevant Information
↓
LLM Generation

But first we need to understand retrieval independently.

5. The Retriever

A retriever is the component responsible for finding relevant items from a collection.

Query
+
Searchable Collection
↓
Retriever
↓
Relevant Candidates

Suppose the corpus contains:

Document A
Document B
Document C
...
Document 100,000

The retriever receives:

Query:
How does solar energy storage work?

and may return:

Document 184
Document 7,920
Document 28,014

because those documents appear relevant.

A retriever does not necessarily produce the final answer.

Its responsibility is primarily:

Find useful information for the query.

6. The Corpus

A corpus is the collection of information available for retrieval.

Examples:

all pages indexed by a search engine

or:

all documents in a company's knowledge base

or:

all academic papers in a research database

or:

all product descriptions in an e-commerce catalog

Corpus
=
Searchable Information Collection

A corpus can be:

small
large
public
private
static
frequently updated
specialized
general

The corpus defines what information the retriever can potentially find.

Corpus Boundaries Matter

Suppose a retrieval system searches only:

Company Support Documentation

The user asks:

What was Apple's revenue last year?

Even if the information exists somewhere on the web, it may not exist in the system's corpus.

Information exists somewhere
≠
Retriever can access it

A more accurate mental model is:

Query
↓
Retriever
↓
Searches Available Corpus

not:

Query
↓
Retriever
↓
Searches all knowledge everywhere

This distinction becomes important later when we discuss source discovery and AI search.

7. Documents and Passages

What Is a Document?

A document is an individual unit of information stored in or represented by the retrieval system.

Examples include:

web page
PDF
news article
product page
support article
email
research paper
database record

Suppose a corpus contains:

10,000 web pages

Each web page can be treated as a document.

Corpus
├── Document 1
├── Document 2
├── Document 3
├── ...
└── Document 10,000

But documents can vary greatly in length.

One document might contain:

200 words

while another contains:

50,000 words

This leads to the concept of a passage.

What Is a Passage?

A passage is a smaller section of a document.

For example, a long article might contain:

Document:
Complete Guide to Solar Energy

with passages such as:

Passage 1:
How Solar Panels Work

Passage 2:
Battery Storage

Passage 3:
Installation Costs

Passage 4:
Maintenance

A user asking:

How long do solar batteries last?

may not need the entire document.

Only one passage may be directly relevant.

Therefore retrieval systems may operate at different levels:

Document Retrieval

or:

Passage Retrieval

Document vs. Passage

Consider this document:

Guide to Toronto Restaurants

It contains:

Section 1:
Italian Restaurants

Section 2:
Persian Restaurants

Section 3:
Japanese Restaurants

Section 4:
Mexican Restaurants

The query is:

Persian restaurants in Toronto

The entire document is broadly relevant.

But:

Section 2

is much more specifically relevant.

Document Relevance
and
Passage Relevance

are related but not identical.

8. What Is a Query?

A query is the expression of an information need submitted to a retrieval system.

Examples:

immigration lawyer Toronto

how does RAG work

best Persian restaurants in Vancouver

refund policy annual subscription

A query can be:

short
long
keyword-like
natural language
question-like
structured

The retriever uses the query to determine what information should be found.

User Query

A user query is the query originating from the user's information need.

For example:

Which Iranian restaurants in Toronto are open late?

The user's language may not perfectly match the language used in the corpus.

The documents might instead contain phrases such as:

Persian restaurant
late-night dining
open until midnight
Iranian cuisine

This mismatch between query wording and document wording is one of the central challenges of retrieval.

Later chapters will show different methods for solving it.

9. Search Query vs. Attention Query

Chapter 3 introduced another concept called Query as part of:

Q / K / V

These are completely different concepts.

Attention Query

Inside Transformer Attention:

Query
Key
Value

The Query is an internal numerical representation involved in Attention calculations.

Search Query

In Information Retrieval:

Query
=
expression of an information need

For example:

best dentist in Toronto

Therefore:

Attention Query
≠
Search Query

They share the same English word but belong to different technical concepts.

10. The Retrieval Problem

Suppose a corpus contains one million documents.

A user asks:

How do electric vehicle batteries get recycled?

The retrieval system must solve several problems.

It needs to determine:

Which documents might be relevant?

Which passages are most relevant?

How relevant is each candidate?

Which results should appear first?

A simplified pipeline might be:

Query
↓
Find Candidates
↓
Score Relevance
↓
Order Candidates
↓
Return Best Results

This is the foundation of retrieval architecture.

11. Crawling

A retrieval system cannot efficiently search information it has never discovered or prepared.

For example, a web search engine must first find web pages.

Information Exists
↓
System Discovers Information
↓
Information Is Processed
↓
Searchable Representation Is Created
↓
Retrieval Becomes Possible

Two important concepts here are:

Crawling
and
Indexing

What Is Crawling?

Crawling is the process of discovering and fetching information that may later become searchable.

For the web, a crawler might:

visit a web page
↓
discover links
↓
follow those links
↓
find additional pages

Known URL
↓
Fetch Page
↓
Discover Links
↓
Fetch More Pages

A web crawler can gradually discover a large portion of the accessible web.

However, crawling is not the same as ranking or retrieval.

It happens earlier.

Crawling Is Discovery, Not Search Ranking

Suppose a crawler discovers:

Page A
Page B
Page C

That does not mean:

Page A is the best result

or:

Page B is relevant to a particular query

Crawling answers a more basic question:

What information exists and can the system access it?

Crawling
→ discovery

Retrieval
→ finding relevant information for a query

Crawling Is Not Required for Every Retrieval System

The term crawling is strongly associated with web search.

But not every retrieval system needs a web crawler.

Suppose a company already has:

10,000 internal documents

stored in a database.

The retrieval system may receive those documents directly.

Internal Data Source
↓
Documents Supplied Directly
↓
Indexing

So the broader requirement is:

the system must obtain or discover the information

Web crawling is one method.

12. Indexing and the Search Index

Indexing

After information has been discovered, it needs to be prepared for efficient search.

This process is called indexing.

Indexing is the process of creating searchable representations or structures from documents.

Documents
↓
Processing
↓
Indexing
↓
Search Index

Why is indexing necessary?

Imagine searching one million documents from scratch for every query.

That could be extremely inefficient.

Instead, the system prepares structures in advance that make retrieval faster.

The Search Index

A search index is a data structure or collection of structures designed to make retrieval efficient.

A useful analogy is the index at the back of a textbook.

Suppose a textbook index contains:

Attention → pages 42, 47, 51
Embedding → pages 21, 27, 34
Transformer → pages 39, 45, 50

Instead of reading the entire book to find every occurrence of:

Embedding

you can use the index.

A search index serves a conceptually similar purpose at much larger scale.

The exact structure of the index depends on the retrieval method.

Later chapters will introduce examples such as:

inverted indexes
vector indexes

A search index is a prepared representation of the corpus that allows relevant information to be found efficiently.

Corpus vs. Search Index

These two concepts should not be confused.

The corpus is the information collection.

The search index is a structure created to help search that collection.

Corpus
=
Original Information Collection

Search Index
=
Prepared Structure for Efficient Retrieval

For example:

Corpus:
100,000 web pages

Search Index:
searchable structures created from those pages

A Basic Preparation Pipeline

Before a query arrives, a retrieval system may perform something like:

Information Sources
↓
Discovery / Crawling
↓
Documents
↓
Processing
↓
Indexing
↓
Search Index

This is largely an offline or preparation stage.

Later, when a query arrives:

User Query
↓
Search Index
↓
Relevant Results

So we can distinguish:

Preparation Stage

from:

Query-Time Retrieval Stage

13. Query-Time Retrieval

Once the index exists, the user submits a query.

For example:

Persian bakery in Toronto

The retrieval system now needs to identify which indexed items are likely relevant.

Query
↓
Search Index
↓
Potentially Relevant Items

But in large systems there may still be too many possibilities to evaluate deeply.

This leads to candidate generation.

14. Candidate Generation

Candidate generation is the process of quickly identifying a manageable subset of items that might be relevant to the query.

Suppose the corpus contains:

10,000,000 documents

It may be impractical to perform expensive analysis on every document for every query.

Instead:

10,000,000 Documents
↓
Candidate Generation
↓
1,000 Candidate Documents

Candidate generation reduces a very large search space to a smaller set of plausible results.

Candidate Documents

The items produced by candidate generation may be called candidate documents.

For example:

Corpus:
1,000,000 documents

Query:
solar battery recycling

Candidate Generation:
↓
Document 84
Document 291
Document 8,410
Document 22,018
...

These documents have not necessarily been proven to be the best results.

They are simply plausible candidates.

Candidate
≠
Final Result

Candidate Set

The collection of candidates produced for a query is called the candidate set.

Query
↓
Candidate Generation
↓
Candidate Set

{
  Document A,
  Document B,
  Document C,
  Document D,
  ...
}

The candidate set should ideally contain the truly relevant documents.

But it should also be small enough for later ranking stages to process efficiently.

Too Few Candidates
→ may miss useful information

Too Many Candidates
→ more expensive to evaluate

We will study retrieval quality more formally in Chapter 12.

Retrieval and Candidate Generation

In practice, terminology can vary.

Sometimes people use:

retrieval

to describe the whole process of finding candidates.

Sometimes they distinguish:

candidate generation

as an early retrieval stage.

For this textbook, a useful model is:

Query
↓
Candidate Generation
↓
Candidate Set
↓
Ranking
↓
Results

15. What Is Relevance?

Relevance describes how well a piece of information matches the user's information need.

Suppose the query is:

how much does private tutoring cost in Toronto

Document A:

Private tutoring prices in Toronto generally depend on subject, tutor experience, and lesson format.

Document B:

The history of Toronto's public transportation system began in...

Document A is much more relevant.

Query
+
Document
↓
Evaluate Relationship
↓
Relevance

But relevance is more complex than simply sharing words.

Relevance Is Not Just Keyword Overlap

Consider the query:

cost of private lessons

Document A says:

Private tutoring prices typically range...

There may be limited exact word overlap:

cost
≠
prices

lessons
≠
tutoring

Yet the document may be highly relevant.

Meanwhile another document might contain:

The cost of building private lesson rooms...

which shares more exact words but expresses a different meaning.

Word Overlap
≠
Relevance

Later chapters will introduce lexical and semantic retrieval approaches for dealing with this problem.

Relevance Depends on the Query

A document does not have one universal relevance score.

Consider the document:

Complete Guide to Persian Restaurants in Toronto

For the query:

Persian restaurants Toronto

it may be highly relevant.

For:

Toronto immigration lawyer

it may be almost completely irrelevant.

Relevance
=
relationship between query and information

not simply an inherent property of the document.

16. Relevance Score

A retrieval system often assigns a numerical value representing estimated relevance.

This is called a relevance score.

Illustratively:

Query:
Persian restaurant Toronto

Document A → 0.91
Document B → 0.82
Document C → 0.37
Document D → 0.08

The exact meaning and scale of the score depend on the retrieval system.

A relevance score is generally useful for comparison:

higher score
→ system estimates stronger relevance

But:

Relevance Score
≠
Probability that the document is correct

and:

Relevance Score
≠
Universal score comparable across all retrieval systems

It is a system-specific ranking signal.

Relevance Score Is Not Factual Accuracy

Suppose a query asks:

When was Company X founded?

A document containing:

Company X was founded in 2018.

may be highly relevant.

But what if the statement is incorrect?

The document can still be highly relevant to the query.

Relevance
≠
Truth

Retrieval asks:

Is this information related to the information need?

Factual verification asks:

Is this information correct?

Those are different problems.

17. Query–Document and Query–Passage Relevance

Query–Document Relevance

Query–document relevance describes how relevant an entire document is to a query.

For example:

Query:
how to apply for a Canadian study permit

Document A:

Complete Guide to Canadian Study Permits

may have high query–document relevance.

Document B:

Canadian Weather Guide

may have low relevance.

Query
↕
Document
↓
Query–Document Relevance

Query–Passage Relevance

Query–passage relevance evaluates a smaller passage instead of the whole document.

Suppose the document is:

Complete Guide to Studying in Canada

It contains:

Section 1 — Universities
Section 2 — Tuition
Section 3 — Study Permit Application
Section 4 — Housing

The query is:

documents needed for Canadian study permit

The entire document is relevant.

But:

Section 3

may have much higher relevance than the other sections.

Query
↕
Passage
↓
Query–Passage Relevance

This more granular relevance will become very important in RAG systems.

Document Relevance Does Not Guarantee Passage Relevance

Imagine a 5,000-word article titled:

Everything You Need to Know About Toronto

The query is:

Toronto property tax rate

The document may contain one short paragraph about property taxes.

Document
→ broadly relevant

while:

one specific passage
→ strongly relevant

and:

most other passages
→ weakly relevant

Relevant Document
≠
Every Passage Is Relevant

18. Ranking

Once a candidate set exists, the system needs to decide which candidates should appear first.

This process is called ranking.

Suppose candidate generation returns:

Document A
Document B
Document C
Document D

The ranking system may produce:

1. Document C
2. Document A
3. Document D
4. Document B

Candidate Set
↓
Estimate Relevance
↓
Order Candidates
↓
Ranked Results

Ranking is essential because users generally care most about the highest-ranked results.

Rank Position

The location of an item within the ordered results is its rank position.

Document C → Rank 1
Document A → Rank 2
Document D → Rank 3
Document B → Rank 4

The result at:

Rank 1

is considered the system's top-ranked result.

Ranking quality matters because users and downstream systems often inspect only the first few results.

Ranking Score

A ranking system may assign each candidate a ranking score.

Illustratively:

Document C → 9.7
Document A → 8.9
Document D → 6.2
Document B → 3.1

The documents can then be ordered by those scores.

However:

Ranking Score
≠
Universal Meaning

The value:

9.7

may mean something completely different in another ranking system.

The important purpose is often relative ordering:

higher score
→ higher rank

within that system.

19. Retrieval vs. Ranking

These terms are related but different.

Retrieval

Find potentially relevant items.

Large Corpus
↓
Retriever
↓
Candidate Set

Ranking

Order those items according to estimated usefulness or relevance.

Candidate Set
↓
Ranking
↓
Ordered Results

Retrieval
→ Which items should we consider?

Ranking
→ In what order should we present them?

This distinction is fundamental.

Why Not Rank Every Document?

Suppose a search engine has:

50 billion documents

and a sophisticated ranking model is expensive to run.

Applying the most expensive ranking process to all 50 billion documents for every query would be impractical.

Instead, systems often use stages.

Huge Corpus
↓
Fast Candidate Generation
↓
Smaller Candidate Set
↓
More Expensive Ranking
↓
Top Results

This staged design balances:

speed
cost
quality

Modern retrieval systems frequently use multiple stages.

20. Reranking

Sometimes an initial ranking is followed by another, more sophisticated evaluation stage.

This is called reranking.

Initial Candidates
↓
Initial Ranking
↓
Top Candidate Subset
↓
Reranker
↓
Improved Ordering

For example:

Initial ranking:

1. Document A
2. Document B
3. Document C
4. Document D

After reranking:

1. Document C
2. Document A
3. Document B
4. Document D

The reranker applies another relevance evaluation to improve the order.

Why Reranking Exists

Different retrieval stages can have different priorities.

An early retriever should often be:

fast
scalable
broad

Its main goal may be to avoid missing useful candidates.

A reranker can afford to be:

slower
more computationally expensive
more precise

because it examines only a smaller number of items.

Stage 1:
Search broadly and quickly

Stage 2:
Evaluate fewer items more carefully

Retriever and Reranker Are Different Components

Retriever
→ finds candidates

Reranker
→ reevaluates and reorders candidates

For example:

1,000,000 Documents
↓
Retriever
↓
100 Candidates
↓
Reranker
↓
10 Best-Ordered Candidates

The retriever and reranker may use different models or methods.

Chapter 11 will study ranking and reranking in much greater depth.

21. Retrieval Can Operate at Multiple Granularities

Retrieval does not always need to return whole documents.

A system might retrieve:

documents
passages
paragraphs
chunks
records
products
web pages

For example:

Query
↓
Document Retrieval
↓
Relevant Document

or:

Query
↓
Passage Retrieval
↓
Relevant Passage

The choice depends on the application.

For RAG, smaller passages are often useful because the system ultimately needs to insert relevant information into the model's context.

22. Retrieval Is a Selection Problem

A useful way to think about retrieval is as a selection problem.

The corpus may contain:

millions of items

but the final system may need only:

5
10
20

useful items.

Large Corpus
↓
Selection
↓
Small Relevant Set

The challenge is to remove huge amounts of irrelevant information without removing the information that matters.

This is why retrieval quality is fundamentally about both:

finding relevant information

and:

excluding irrelevant information

Later, the concepts of precision and recall will formalize this trade-off.

23. A Complete Simplified Retrieval Example

Suppose a company has:

100,000 support documents

and a user asks:

How can I cancel my annual subscription?

Let us follow the process.

Step 1 — The Corpus Exists

Support Document Corpus
↓
100,000 Documents

Step 2 — Documents Are Indexed

Documents
↓
Indexing
↓
Search Index

Step 3 — The User Submits a Query

How can I cancel my annual subscription?

Step 4 — Candidate Generation

The retriever searches the index.

100,000 Documents
↓
Candidate Generation
↓
200 Candidate Documents

Candidates may include:

Subscription Cancellation
Annual Plans
Billing FAQ
Account Management
Refund Policies

Step 5 — Relevance Is Estimated

Illustratively:

Subscription Cancellation → 0.96
Annual Plans              → 0.84
Billing FAQ               → 0.79
Refund Policies           → 0.52
Password Reset            → 0.03

Step 6 — Candidates Are Ranked

1. Subscription Cancellation
2. Annual Plans
3. Billing FAQ
4. Refund Policies
...

Step 7 — Optional Reranking

A stronger model evaluates the top candidates more carefully.

The order may change:

1. Subscription Cancellation
2. Billing FAQ
3. Annual Plans
4. Refund Policies

Step 8 — Results Are Returned

The retrieval system now has a small set of relevant information.

Query
↓
Relevant Documents / Passages

At this point, another system might:

show the results to the user

or later:

provide them to an LLM

Retrieval itself does not require generation.

24. Retrieval in a Traditional Search Engine

A simplified traditional search process might look like:

Web
↓
Crawling
↓
Documents
↓
Indexing
↓
Search Index

Then when the user searches:

User Query
↓
Candidate Generation
↓
Ranking
↓
Search Results

The interface may display:

Page A
Page B
Page C

with titles, snippets, and links.

The fundamental retrieval problem remains:

Which indexed documents are most relevant to this query?

25. Retrieval in an AI System

An AI system may use a similar retrieval foundation.

Suppose the user asks:

What does our employee handbook say about parental leave?

The system may perform:

User Query
↓
Retriever
↓
Employee Handbook Passages
↓
Relevant Passage

Then the passage might be provided to an LLM:

Relevant Passage
+
User Question
↓
LLM
↓
Generated Answer

This later combination is a form of retrieval-augmented generation.

But notice that:

retrieval happens before generation

in this simplified pipeline.

Understanding this separation is essential.

26. Retrieval and Generation Failures

Retrieval Does Not Mean the Result Will Be Used Correctly

Suppose a retriever successfully finds the correct passage.

That does not guarantee that a later LLM will:

interpret it correctly
use it completely
avoid adding unsupported claims
cite it correctly

So there are separate problems:

Retrieval Quality

and:

Generation Quality

Later chapters will also introduce:

Evidence Quality
Groundedness
Faithfulness
Citation Quality

Retrieval Failure vs. Generation Failure

Suppose the correct answer exists in Document A.

The system returns Document B instead.

That is primarily a:

Retrieval Problem

But suppose the system correctly retrieves Document A and the LLM misreads it.

That is primarily a:

Generation / Interpretation Problem

Correct Information Not Retrieved
→ Retrieval Failure

Correct Information Retrieved
but Misused
→ Generation / Grounding Failure

This distinction will become extremely important in AI Visibility diagnosis later in the textbook.

27. Candidate Generation Dependency

Candidate Generation Can Miss Relevant Information

Suppose the truly best document is:

Document X

But candidate generation never includes it in the candidate set.

Then the ranking system cannot rank it highly.

Relevant Document
↓
Not Retrieved as Candidate
↓
Never Reaches Ranking Stage
↓
Cannot Become Top Result

Later ranking stages cannot recover information that never entered the candidate set.

Ranking Cannot Fix Every Retrieval Problem

Suppose:

Candidate Set:
Document A
Document B
Document C

But the truly relevant document is:

Document Z

and Document Z was never retrieved.

Even a perfect reranker can only reorder:

A
B
C

It cannot select:

Z

because Z is absent.

Candidate Generation Quality
↓
Constrains
↓
Possible Ranking Quality

28. Query-Specific Ranking and Intent

Ranking Is Usually Query-Specific

A single document can have different ranks for different queries.

Consider:

Document:
Guide to Persian Food in Toronto

Query A:

Persian food Toronto

The document might rank highly.

Query B:

Italian restaurant Toronto

It might rank much lower.

Rank
depends on
Query

There is usually no single universal rank for a document across all information needs.

Retrieval Is Also Context-Dependent

Even identical words can represent different information needs.

Consider:

apple

Possible intentions include:

Apple the company
apple the fruit
Apple stock
Apple products

A retrieval system needs to interpret the query sufficiently well to retrieve useful information.

Query understanding becomes a major topic later in Chapter 16.

For now, remember:

Query Text
≠
Complete Information Need

The words are an expression of the user's need, not necessarily a perfect representation of it.

29. Basic Retrieval Pipeline

We can now write the core pipeline introduced in the curriculum:

Query
↓
Candidate Generation
↓
Retrieval
↓
Ranking
↓
Results

A slightly more detailed version is:

Corpus
↓
Indexing
↓
Search Index

User Query
↓
Candidate Generation
↓
Candidate Set
↓
Relevance Estimation
↓
Ranking
↓
Optional Reranking
↓
Top Results

And if passages are involved:

Query
↓
Relevant Documents
↓
Relevant Passages
↓
Ranked Evidence Candidates

Later chapters will fill in the technical methods used inside these stages.

What We Have Not Explained Yet

At this point, several major questions remain.

How does the retriever actually find candidate documents?

One method might rely heavily on exact words.

Another might use numerical semantic representations.

We have not yet studied those retrieval methods.

Similarly:

How does the system calculate relevance?

How does ranking work?

How do we measure whether retrieval is good?

Those topics belong to later chapters.

The next chapters will introduce:

Lexical Retrieval
↓
Dense / Semantic Retrieval
↓
Hybrid Retrieval
↓
Ranking and Reranking
↓
Retrieval Evaluation

So Chapter 7 establishes the architecture before we study the methods.

30. Common Misunderstandings

Misunderstanding 1: Retrieval and generation are the same process

Incorrect. Retrieval finds existing information. Generation creates new token sequences.

Misunderstanding 2: A retriever must be an LLM

Incorrect. Retrieval systems can use many different algorithms and models.

Misunderstanding 3: A corpus means the entire internet

Incorrect. A corpus is simply the information collection available to a retrieval system.

Misunderstanding 4: A document must be a PDF or Word file

Incorrect. A document can be a web page, article, record, email, product page, or another searchable information unit.

Misunderstanding 5: A passage and a document are the same thing

Incorrect. A passage is usually a smaller section of a document.

Misunderstanding 6: Search Query and Attention Query mean the same thing

Incorrect. A search query expresses an information need. An Attention Query is an internal numerical representation in the Q/K/V mechanism.

Misunderstanding 7: Crawling means ranking pages

Incorrect. Crawling discovers and fetches information. Ranking happens later for a particular query.

Misunderstanding 8: Indexing means the same thing as retrieval

Incorrect. Indexing prepares searchable structures. Retrieval uses those structures to find relevant information.

Misunderstanding 9: The corpus and search index are identical

Incorrect. The corpus contains the information. The index is a structure designed to search it efficiently.

Misunderstanding 10: Candidate documents are already final results

Incorrect. Candidates are possible results that may need further ranking or reranking.

Misunderstanding 11: High relevance means the information is true

Incorrect. Relevance and factual correctness are separate concepts.

Misunderstanding 12: A relevance score is a universal probability

Incorrect. Its meaning depends on the retrieval or ranking system.

Misunderstanding 13: If a document is relevant, every passage inside it is relevant

Incorrect. A large document may contain both highly relevant and irrelevant passages.

Misunderstanding 14: Ranking finds information that candidate generation missed

Incorrect. A ranker generally operates on the candidates it receives.

Misunderstanding 15: Reranking searches the entire corpus again

Not necessarily. A reranker commonly evaluates a smaller candidate subset more carefully.

Misunderstanding 16: Good retrieval guarantees a correct AI answer

Incorrect. The model can still misunderstand or misuse correctly retrieved information.

31. How the Concepts Connect

We can now connect the LLM foundations from Chapters 1–6 with Information Retrieval.

The LLM side:

Text
↓
Tokens
↓
Numerical Representations
↓
Transformer Processing
↓
Learned Parameters
↓
Generation
↓
Assistant

The retrieval side begins with:

Information Sources
↓
Documents
↓
Indexing
↓
Search Index

Then at query time:

User Query
↓
Candidate Generation
↓
Candidate Set
↓
Relevance Estimation
↓
Ranking
↓
Reranking
↓
Relevant Results

These are separate systems.

Later they can be connected:

User Query
↓
Retrieval
↓
Relevant Information
↓
LLM Context
↓
Generation
↓
Answer

That connection will eventually become Retrieval-Augmented Generation.

But before we reach RAG, we need to understand how retrieval itself works.

Mental Model

Imagine a giant library containing one million books.

A user walks in and asks:

How do solar batteries store electricity?

Reading every book would be impossible.

So the library has already prepared a searchable catalog.

That catalog is analogous to a:

Search Index

The user's question is the:

Query

The library system quickly identifies perhaps:

500 potentially useful books

That is similar to:

Candidate Generation

Those 500 books form the:

Candidate Set

The system then estimates which books are most relevant.

That produces:

Relevance Scores

The books are ordered:

Rank 1
Rank 2
Rank 3
...

That is:

Ranking

A more careful librarian might examine the top 20 and change the order.

That resembles:

Reranking

Finally, only a few books or passages are returned to the user.

The fundamental transformation is:

Huge Collection
↓
Query
↓
Candidate Selection
↓
Relevance Evaluation
↓
Ranking
↓
Small Relevant Result Set

The analogy has limits.

Real retrieval systems use algorithms, indexes, numerical representations, ranking models, and large-scale infrastructure.

But the core mental model is correct:

Information Retrieval takes an information need expressed as a query and selects a small, ranked set of potentially relevant documents or passages from a much larger corpus.

Chapter 7 Recap

Information Retrieval

The process of finding information relevant to an information need from a larger collection.

Query
+
Collection
↓
Relevant Information

Retrieval

The act of finding potentially relevant existing information.

Query
↓
Search
↓
Existing Documents / Passages

Retriever

The component responsible for finding relevant items from a corpus.

Corpus

The collection of information available to the retrieval system.

Corpus
=
Searchable Information Collection

Document

An individual searchable information unit.

web page
article
PDF
product page
email
database record

Passage

A smaller section of a document.

Document
↓
Passages

Passage-level retrieval allows the system to find more specific information inside larger documents.

Query

An expression of an information need submitted to a retrieval system.

Persian restaurant Toronto

User Query

The query originating from the user's need. The query wording may differ from the wording used in relevant documents.

Attention Query vs. Search Query

Attention Query
→ internal Q/K/V representation

Search Query
→ expression of an information need

They are different concepts.

Crawling

The process of discovering and fetching information that may later become searchable.

Discover
↓
Fetch
↓
Discover More

Indexing

The process of creating searchable representations or structures from documents.

Documents
↓
Indexing
↓
Search Index

Search Index

A prepared data structure designed to make retrieval efficient.

Corpus vs. Index

Corpus
→ original information collection

Search Index
→ structure used to search that collection

Candidate Generation

The process of quickly reducing a very large corpus to a smaller set of potentially relevant items.

Huge Corpus
↓
Candidate Generation
↓
Candidate Set

Candidate Document

A document considered potentially relevant and worthy of further evaluation.

Candidate
≠
Final Result

Candidate Set

The collection of candidates selected for a query.

Relevance

The degree to which information matches the user's information need.

Query
↔
Information
↓
Relevance

Relevance Score

A numerical value representing the system's estimate of relevance.

Higher Score
→ Usually Greater Estimated Relevance

The scale depends on the retrieval system.

Relevance vs. Truth

Highly Relevant
≠
Factually Correct

Relevance measures relationship to the information need, not factual accuracy.

Query–Document Relevance

The relevance relationship between a query and an entire document.

Query–Passage Relevance

The relevance relationship between a query and a smaller passage.

Relevant Document
≠
Every Passage Is Relevant

Ranking

Ordering candidates according to estimated relevance or usefulness.

Candidate Set
↓
Ranking
↓
Ordered Results

Rank Position

The position of an item within ranked results.

Rank 1
Rank 2
Rank 3
...

Ranking Score

A numerical signal used to help order candidates. Its scale is system-specific.

Retrieval vs. Ranking

Retrieval
→ Which items should we consider?

Ranking
→ In what order should they appear?

Reranking

A later stage that reevaluates and reorders a smaller candidate set more carefully.

Candidates
↓
Initial Ranking
↓
Reranker
↓
Improved Ordering

Candidate Dependency

Later ranking stages can only work with information that reached them.

Not Retrieved
↓
Not in Candidate Set
↓
Cannot Be Ranked

The complete basic Information Retrieval pipeline is:

Information Sources
↓
Discovery / Crawling
↓
Documents
↓
Indexing
↓
Search Index
↓
User Query
↓
Candidate Generation
↓
Candidate Set
↓
Relevance Estimation
↓
Ranking
↓
Reranking
↓
Top Results

And the simplest query-time version is:

Query
↓
Candidate Generation
↓
Retrieval
↓
Ranking
↓
Results

We now understand the architecture of a retrieval system.

But we have not yet answered the most important implementation question:

How does the system decide which documents are relevant?

One of the oldest and still extremely important approaches uses the actual words contained in the query and documents.

That leads directly into Chapter 8:

Lexical and Traditional Search.

The Single Most Important Idea from Chapter 7

Information Retrieval takes a user's information need, expressed as a query, and searches a much larger corpus to produce a small set of potentially relevant documents or passages. Candidate generation finds what should be considered; ranking and reranking determine which of those candidates should appear first.