Skip to content

Mo Farahani

AI Visibility Guide

02 Part II — Information Retrieval Chapter 02.02

Chapter 8- Lexical and Traditional Search

By Mo Farahani 31 min read

Chapter Goal

In Chapter 7, we built the basic architecture of an Information Retrieval system.

We learned:

Information Sources
↓
Documents
↓
Indexing
↓
Search Index
↓
User Query
↓
Candidate Generation
↓
Ranking
↓
Results

But we deliberately left one major question unanswered:

How does a retrieval system actually decide which documents might be relevant to a query?

One of the oldest and still most important answers is:

look at the words

Suppose the query is:

Persian restaurant Toronto

A traditional search system may look for documents containing terms such as:

Persian
restaurant
Toronto

Documents containing these terms may receive higher relevance scores.

This family of techniques is broadly called:

lexical retrieval.

Lexical retrieval remains extremely important even in modern AI systems.

It is used in:

traditional search engines
site search
enterprise search
e-commerce search
document retrieval
RAG
hybrid retrieval systems

This chapter explains how lexical search works.

By the end of the chapter, you should understand:

  • what keyword search is;
  • what lexical search is;
  • what exact match means;
  • what term matching means;
  • what sparse retrieval is;
  • what a sparse vector is;
  • what an inverted index is;
  • what a posting list is;
  • what term frequency means;
  • what document frequency means;
  • why rare terms can be more informative than common terms;
  • what TF-IDF means conceptually;
  • what BM25 is;
  • why BM25 improves on simple keyword counting;
  • what keyword relevance means;
  • why lexical search works extremely well for some queries;
  • what vocabulary mismatch is;
  • why synonyms cause retrieval problems;
  • what search intent mismatch means;
  • the difference between exact words and meaning;
  • and why lexical retrieval remains important even when semantic retrieval is available.

The central idea of this chapter is:

Query Terms
↓
Match Terms in Documents
↓
Weight Important Matches
↓
Calculate Relevance Score
↓
Rank Documents

1. From Retrieval Architecture to Retrieval Method

Chapter 7 gave us this pipeline:

Query
↓
Candidate Generation
↓
Candidate Set
↓
Ranking
↓
Results

But imagine the query:

Toronto immigration lawyer

How does the system find candidate documents?

One simple strategy is:

Find documents containing the same words as the query.

For example:

Document A:
Toronto immigration lawyer specializing in permanent residence applications.

Document B:
Best restaurants and cafés in downtown Toronto.

Document C:
Immigration law services for clients across Toronto.

Documents A and C have strong lexical overlap with the query.

Document B contains only:

Toronto

So a lexical system would probably consider A and C much more relevant.

This is the foundation of traditional lexical retrieval.

4. Keyword Search vs. Lexical Search

These terms are sometimes used loosely and interchangeably.

For this textbook, a useful distinction is:

Keyword Search
→ broad idea of searching using keywords

Lexical Search
→ retrieval based on matching lexical terms and term statistics

So lexical search may include:

term matching
exact matches
term frequencies
document frequencies
TF-IDF
BM25

The important point is that the retrieval signal comes primarily from the words represented in the query and documents.

5. Term Matching

A term is a searchable lexical unit.

For simplicity, imagine:

Query:
Persian restaurant Toronto

contains three terms:

Persian
restaurant
Toronto

A retrieval system may compare these terms with terms found in each document.

Document A:

Persian restaurant in Toronto serving traditional Iranian food.

Matches:

Persian ✓
restaurant ✓
Toronto ✓

Document B:

Italian restaurant in Toronto.

Matches:

Persian ✗
restaurant ✓
Toronto ✓

Document C:

Persian grocery store in Vancouver.

Matches:

Persian ✓
restaurant ✗
Toronto ✗

A simple system might rank:

A > B > C

because A matches more of the query terms.

This is term matching.

6. Exact Match

An exact match occurs when the lexical form in the query directly matches the lexical form in the document.

For example:

Query:
Toronto dentist

Document:

Toronto dentist offering emergency dental care.

The terms:

Toronto
dentist

appear directly.

This is strong lexical evidence.

Another example:

Query:
GPT-5 API pricing

A document containing exactly:

GPT-5 API pricing

can be especially useful.

Exact lexical matching is often powerful for:

names
product codes
model numbers
technical terminology
locations
rare phrases
brand names

7. Exact Phrase Matching

Sometimes the sequence of words matters.

Compare:

New York

with documents containing:

New York City

versus:

a new restaurant in York

Both contain:

new
York

but only the first contains the phrase:

New York

as the intended expression.

Search systems can therefore use positional information to determine whether query terms occur:

next to each other
in the same order
within a certain distance

Exact phrase matching can provide stronger evidence than unrelated individual term matches.

8. Why Simple Match Counting Is Not Enough

Imagine the query:

machine learning engineer

Document A contains:

machine learning engineer

once.

Document B contains:

machine

twenty times but never discusses machine learning.

A naive counting system could incorrectly overvalue Document B.

Another issue is that some words are much more informative than others.

Consider:

the

versus:

photosynthesis

The word:

the

appears in enormous numbers of documents.

Finding it tells us almost nothing about relevance.

But:

photosynthesis

is much more specific.

Traditional Information Retrieval therefore needs better weighting methods.

9. Keyword Relevance

Keyword relevance describes how useful lexical matches are for determining whether a document relates to a query.

Suppose the query is:

Persian dentist Toronto

Consider:

Document A:
Persian dentist in Toronto

Document B:
Dentist in Toronto

Document C:
Persian restaurant in Toronto

Document A matches all three important concepts.

Document B lacks:

Persian

Document C lacks:

dentist

So lexical evidence may suggest:

A > B and C

But the usefulness of each matching term depends on additional statistics.

Two fundamental statistics are:

Term Frequency
Document Frequency

10. Term Frequency

Term Frequency, often abbreviated as TF, measures how frequently a term appears within a particular document.

Suppose the term is:

dentist

Document A contains it:

1 time

Document B:

8 times

Document C:

0 times
TF(term, document)
=
how often the term occurs in that document

A higher term frequency can be evidence that the document focuses more strongly on that topic.

Term Frequency Example

Consider:

Query:
solar battery

Document A:

Solar batteries store excess electricity generated by solar panels.
Battery capacity affects how much solar energy can be stored.

Document B:

Our company sells many energy products.
We mentioned a solar battery during our annual conference.

In Document A, important query terms appear repeatedly and centrally.

In Document B, they appear incidentally.

Term frequency can help distinguish these situations.

More Term Frequency Is Not Always Linearly Better

Suppose a document contains:

dentist

once.

Another contains it five times.

That difference may be useful.

But compare:

50 occurrences

with:

500 occurrences

The second document is not necessarily ten times more relevant.

Repeatedly writing:

dentist dentist dentist dentist dentist...

should not make a page infinitely relevant.

Therefore modern lexical ranking systems often use term-frequency saturation.

First few occurrences
→ substantial additional evidence

Many additional repetitions
→ diminishing benefit

This will matter when we reach BM25.

11. Document Frequency

Document Frequency, often abbreviated as DF, measures how many documents in the corpus contain a particular term.

Suppose the corpus contains:

1,000,000 documents

The term:

the

might occur in:

900,000 documents

while:

photosynthesis

might occur in:

8,000 documents

and:

FarahaniDental2026

might occur in:

12 documents
Document Frequency
=
number of documents containing the term

Why Document Frequency Matters

A term that appears everywhere is usually less useful for distinguishing documents.

Consider:

the

If nearly every document contains it, matching the provides little information.

But if the query contains:

ketogenic

and only a small portion of the corpus contains that term, the match is more informative.

Very Common Term
→ weak distinguishing signal

Rare Term
→ stronger distinguishing signal

This insight is fundamental to traditional search.

12. Term Frequency vs. Document Frequency

Do not confuse these two.

Term Frequency

Asks:

How often does this term appear
inside this document?

Document Frequency

Asks:

In how many documents
does this term appear?
TF
→ local to one document

DF
→ global across the corpus

Both can help determine lexical relevance.

13. Rare Terms Can Carry More Information

Suppose the query is:

OpenAI GPT API

The term:

API

may appear in many technical documents.

The term:

OpenAI

may appear in fewer.

And a specific model name may appear in even fewer documents.

A match on a rarer, more specific term may therefore tell us more about relevance.

Not all term matches should have equal weight.

Traditional retrieval systems often reward matches on terms that are relatively rare in the corpus.

14. TF-IDF

One famous traditional retrieval concept is:

TF-IDF.

It stands for:

Term Frequency
×
Inverse Document Frequency

At a conceptual level:

TF
→ How important does the term appear inside this document?

IDF
→ How informative or rare is the term across the corpus?

Together:

TF-IDF
↓
Higher weight for terms that are
frequent in this document
but relatively rare across the corpus

Inverse Document Frequency

The Inverse Document Frequency, or IDF, is designed to give greater weight to terms that occur in fewer documents.

High Document Frequency
↓
Low IDF

Low Document Frequency
↓
High IDF

For example:

the
→ appears almost everywhere
→ low IDF

while:

photosynthesis
→ appears in fewer documents
→ higher IDF

The exact mathematical formula can vary.

For this course, the intuition is more important than the formula.

TF-IDF Example

Suppose the query is:

Persian saffron Toronto

Document A contains:

Persian saffron

multiple times.

Document B contains:

Toronto

many times but never discusses saffron.

If:

Toronto

is common across the corpus but:

saffron

is relatively rare, a TF-IDF-style system can give more importance to the saffron match.

Frequent in Document
+
Rare Across Corpus
↓
Strong Lexical Signal

TF-IDF Is Not Semantic Understanding

Suppose the query is:

cost of private tutoring

A document says:

Prices for one-to-one lessons...

The meanings are strongly related.

But the exact vocabulary differs:

cost
↔
prices

private tutoring
↔
one-to-one lessons

A basic TF-IDF system may not naturally know these are semantically similar.

TF-IDF
→ sophisticated lexical weighting

TF-IDF
≠
deep semantic understanding

This limitation will become important later.

15. Sparse Retrieval

Traditional term-based search is often called sparse retrieval.

Why?

Because documents and queries can be represented using very large vectors where most dimensions are zero.

Imagine the vocabulary contains:

100,000 possible terms

A short document might contain only:

150 unique terms

Its representation could therefore have:

100,000 dimensions

but only:

150 non-zero values

Most values are:

0

Hence:

Sparse Vector

16. What Is a Sparse Vector?

A sparse vector is a vector in which most values are zero.

Suppose our tiny vocabulary is:

[
  dentist,
  restaurant,
  Toronto,
  lawyer,
  Persian,
  bakery
]

Document:

Persian dentist Toronto

could be represented conceptually as:

dentist    → 1
restaurant → 0
Toronto    → 1
lawyer     → 0
Persian    → 1
bakery     → 0

Vector:

[1, 0, 1, 0, 1, 0]

In a real corpus the vocabulary can contain hundreds of thousands or millions of terms.

Most documents use only a tiny portion of that vocabulary.

So most dimensions remain zero.

Sparse Vector Dimensions Usually Represent Terms

This is an important distinction from the embeddings we studied earlier.

In a simple lexical sparse representation:

Dimension 1
→ dentist

Dimension 2
→ restaurant

Dimension 3
→ Toronto

The dimensions correspond directly to vocabulary terms.

Compare that with a dense embedding:

[0.13, -0.48, 0.72, ...]

where individual dimensions usually do not correspond neatly to:

dentist
restaurant
Toronto

This difference becomes central in Chapter 9.

Sparse Lexical Vector
→ dimensions often correspond to explicit terms

Dense Embedding
→ dimensions represent learned latent features

Sparse Retrieval Does Not Mean Poor Retrieval

The word:

sparse

does not mean:

weak
bad
primitive

It describes the mathematical representation.

Sparse lexical retrieval can be extremely powerful.

It is especially strong when exact lexical identity matters.

For example:

error code 0x80070005

or:

Canon EOS R5

or:

Farahani v. Company X

Exact lexical matching can be extremely valuable for these queries.

17. What Is an Inverted Index?

Imagine one million documents.

A user searches:

Persian dentist Toronto

One possible approach would be:

Open Document 1
Search for terms

Open Document 2
Search for terms

Open Document 3
Search for terms

...

Open Document 1,000,000

Doing this from scratch for every query would be inefficient.

Instead, traditional search systems commonly create an:

inverted index.

An inverted index maps terms to the documents that contain them.

For example:

Persian
→ Document 2
→ Document 8
→ Document 91

dentist
→ Document 8
→ Document 15
→ Document 91

Toronto
→ Document 1
→ Document 8
→ Document 22
→ Document 91

Now the query:

Persian dentist Toronto

can quickly identify documents appearing in these term lists.

For example:

Document 8
Document 91

appear under all three terms.

They are therefore strong candidates.

Why Is It Called “Inverted”?

The original document perspective looks like:

Document 1
→ words inside Document 1

Document 2
→ words inside Document 2

The inverted perspective is:

Word
→ documents containing that word

So instead of:

Document → Terms

we create:

Term → Documents

That reversal is why it is called an inverted index.

18. Posting Lists

The list of documents associated with a term is commonly called a posting list.

Example:

Term:
dentist

Posting List:
Doc 8
Doc 15
Doc 91
Doc 207
...

Posting lists can contain more information than just document IDs.

They may also contain data such as:

term frequency
term position
field information

This extra information can help with ranking and phrase matching.

Inverted Index Example

Suppose the corpus is:

Doc A:
Persian dentist Toronto

Doc B:
Italian restaurant Toronto

Doc C:
Persian grocery Vancouver

Doc D:
Dentist Vancouver

The inverted index might contain:

Persian
→ A, C

dentist
→ A, D

Toronto
→ A, B

restaurant
→ B

grocery
→ C

Vancouver
→ C, D

Now query:

Persian dentist Toronto

can be processed by checking:

Persian → A, C
dentist → A, D
Toronto → A, B

Document A appears in all three posting lists.

That is strong lexical evidence.

19. Inverted Index and Candidate Generation

Now we can connect the inverted index to Chapter 7.

Recall:

Query
↓
Candidate Generation
↓
Candidate Set

With lexical retrieval:

Query Terms
↓
Look Up Terms in Inverted Index
↓
Collect Matching Documents
↓
Candidate Set

So the inverted index can be one of the main mechanisms used to generate candidates efficiently.

20. Boolean Retrieval

One of the simplest traditional retrieval approaches uses Boolean logic.

For example:

Persian AND dentist AND Toronto

means the document must contain all three terms.

Another query:

dentist OR orthodontist

means documents containing either term may match.

And:

dentist NOT Vancouver

can exclude documents containing Vancouver.

Boolean retrieval is useful when exact filtering rules matter.

But it does not automatically provide sophisticated relevance ranking.

Matching Documents Is Not Enough

Suppose 10,000 documents contain:

Toronto dentist

Which should appear first?

We now need scoring.

Matching Documents
↓
Calculate Relevance Scores
↓
Order Documents

Possible signals include:

term frequency
document frequency
document length
number of matched query terms
importance of rare terms

This leads us from simple matching toward ranking models such as BM25.

21. The Problem of Document Length

Imagine the query:

solar battery

Document A is:

200 words

and mentions:

solar battery

five times.

Document B is:

20,000 words

and mentions the same phrase five times.

Those five occurrences may represent a much stronger topical focus in Document A.

5 mentions in 200 words

can mean something different from:

5 mentions in 20,000 words

Traditional ranking systems can therefore consider document length.

Why TF-IDF Is Not the End of the Story

TF-IDF introduced two important ideas:

term frequency
+
term rarity

But real retrieval needs to handle additional issues such as:

term frequency saturation
document length
ranking stability

One highly influential ranking method designed to address these issues is:

BM25.

22. What Is BM25?

BM25 is a widely used lexical ranking function for estimating how relevant a document is to a query.

You do not need to memorize its mathematical formula.

Conceptually, BM25 considers ideas such as:

Does the document contain the query term?

How often does the term appear?

How rare is the term across the corpus?

How long is the document?

How much additional value should repeated occurrences provide?

A simplified mental model is:

Query–Document Term Matches
+
Term Frequency
+
Term Rarity
+
Document Length Adjustment
+
Frequency Saturation
↓
BM25 Score

23. How BM25 Behaves

BM25 and Term Rarity

Like TF-IDF, BM25 gives more importance to terms that help distinguish documents.

Suppose query:

Persian blue salt Toronto

contains:

Toronto

which may occur in many documents.

But:

blue salt

may occur in far fewer.

A document matching the rarer terms may receive a stronger lexical relevance signal.

BM25 and Term-Frequency Saturation

Imagine the query term:

dentist

appears:

1 time

in Document A,

5 times

in Document B,

and:

500 times

in Document C.

A naive system might assume:

500 occurrences
= 100× better than 5

BM25 does not behave that way.

Instead, additional occurrences produce diminishing returns.

1 → 2 occurrences
may matter substantially

100 → 101 occurrences
usually matters much less

This is term-frequency saturation.

BM25 and Document Length

BM25 can also adjust for document length.

Suppose:

Document A:
300 words
10 relevant term occurrences

and:

Document B:
30,000 words
10 relevant term occurrences

Those occurrences may signal stronger focus in Document A.

BM25 includes document-length normalization so that very long documents do not automatically gain excessive advantage simply because they contain more words.

BM25 Is a Ranking Function

BM25 does not itself mean:

search engine

It is more specifically a ranking function.

Query
+
Document
↓
BM25
↓
Score

For multiple candidate documents:

Doc A → 12.4
Doc B → 9.7
Doc C → 4.3

The documents can then be ranked:

1. Doc A
2. Doc B
3. Doc C

The exact numerical score is usually less important than the ordering it produces.

BM25 Scores Are Not Universal Probabilities

If BM25 gives:

Document A → 14.8

that does not mean:

14.8% relevant

or:

98% likely to be correct

BM25 scores are ranking scores.

Their absolute values depend on:

query
corpus
implementation
parameters
document statistics
BM25 Score
≠
Probability

and:

BM25 Score
≠
Truth Score

A Simplified BM25 Example

Suppose the query is:

Persian bakery Toronto

Document A:

Persian bakery in Toronto offering traditional Iranian pastries.

Document B:

Toronto bakery serving European pastries.

Document C:

Persian grocery store in Toronto.

A BM25-style system might reason conceptually:

Document A:
Persian ✓
bakery ✓
Toronto ✓
strong score

Document B:
Persian ✗
bakery ✓
Toronto ✓
medium score

Document C:
Persian ✓
bakery ✗
Toronto ✓
medium / lower score

If:

Persian

is relatively rare in the corpus, that match can receive extra importance.

24. Why Lexical Search Is Powerful

Lexical retrieval may sound simple compared with modern neural methods.

But it has major strengths.

It can perform extremely well for queries involving:

specific names
rare terms
exact phrases
product codes
technical terms
legal citations
error messages
model numbers
locations

For example:

ERR_CONNECTION_RESET

Exact lexical matching may be more useful than trying to infer broad semantic similarity.

Lexical Search Is Interpretable

Another advantage is that lexical matches can often be inspected.

Suppose a document ranks highly because:

Persian
dentist
Toronto

all appear in the document.

A human can understand why.

Traditional lexical scoring can therefore be easier to debug than some dense neural systems.

Matched Terms
↓
Observable Lexical Evidence
↓
Ranking Score

This interpretability can be valuable in production search systems.

Lexical Search Is Efficient

Inverted indexes are extremely efficient for term lookup.

Instead of scanning every document, the system can immediately access posting lists.

Query Term
↓
Inverted Index Lookup
↓
Matching Document IDs

This makes lexical retrieval practical even for very large corpora.

That scalability is one reason inverted indexes remain foundational in search engines.

25. The Central Weakness: Vocabulary Mismatch

Now we reach the major limitation of lexical retrieval.

Suppose the query is:

cheap private teacher

The relevant document says:

affordable one-to-one tutor

The meanings are similar:

cheap
↔
affordable

private teacher
↔
one-to-one tutor

But the words are different.

A purely lexical system may fail to recognize the relationship.

This is called vocabulary mismatch.

What Is Vocabulary Mismatch?

Vocabulary mismatch occurs when the query and relevant document express similar meaning using different words.

For example:

Query:
car repair

Relevant document:

automobile maintenance

or:

Query:
cheap hotel

Relevant document:

budget accommodation

or:

Query:
heart doctor

Relevant document:

cardiologist

The document may be semantically relevant even though lexical overlap is weak.

Vocabulary Mismatch Can Cause Retrieval Failure

Return to Chapter 7.

Suppose the best document uses:

automobile maintenance

but the query uses:

car repair

If the lexical retriever does not connect those terms:

Relevant Document
↓
Weak Term Match
↓
Not Retrieved
↓
Not in Candidate Set
↓
Cannot Be Ranked Later

This demonstrates why vocabulary mismatch can become a candidate-generation problem.

26. Synonyms

A common cause of vocabulary mismatch is synonymy.

Synonyms are words with similar meanings.

Examples:

cheap ↔ affordable

car ↔ automobile

attorney ↔ lawyer

purchase ↔ buy

Suppose the query is:

immigration attorney Toronto

and a document repeatedly uses:

immigration lawyer Toronto

Humans immediately recognize the relationship:

attorney ≈ lawyer

A strict lexical matching system may not.

This is one major motivation for semantic retrieval.

27. Exact Words vs. Meaning

We can now state the fundamental distinction.

Lexical retrieval asks primarily:

Do these words match?

Semantic retrieval will later try to answer more directly:

Do these meanings match?

Compare:

Query:
cheap private teacher

with:

Document:
affordable one-to-one tutor

Lexically:

weak overlap

Semantically:

strong similarity

This is the central transition from Chapter 8 to Chapter 9.

Lexical Match Can Be Strong but Meaning Can Be Wrong

The opposite problem can also happen.

Suppose query:

Apple repair

Document A:

Apple iPhone and Mac repair services

Document B:

How to repair damaged apple trees

Both contain:

Apple
repair

But they refer to completely different meanings.

Strong Word Overlap
≠
Correct Meaning

Lexical matching alone can struggle with ambiguity.

28. Search Intent Mismatch

A document can also match keywords while failing the user's actual intent.

Suppose the query is:

buy Persian saffron Toronto

Document A:

History of Persian saffron production

Document B:

Persian saffron available for purchase in Toronto

Both may contain:

Persian
saffron
Toronto

but Document B better satisfies the transactional intent:

buy

This is an example of search intent mismatch.

Lexical overlap alone cannot perfectly determine whether a page satisfies the user's underlying goal.

Search Intent Is More Than Keywords

Consider:

python

Possible information needs include:

Python programming language
python snake
Python installation
Python tutorial
Python jobs

The text of the query may be identical.

The user's actual need is not.

Query Words
≠
Complete Search Intent

Later chapters on query understanding will explore this in more depth.

29. Lexical vs. Semantic Strengths

Exact Match Can Sometimes Be Better Than Semantic Similarity

It would be wrong to conclude:

semantic search is always better than lexical search

Consider:

error code E11000

or:

iPhone 17 Pro Max

or:

26 U.S.C. § 501(c)(3)

or:

ZX-4102-B

In such cases, exact lexical identity can be extremely important.

A semantic system that retrieves merely similar concepts may be less useful.

Exact Words
can sometimes matter more than
Broad Meaning

Semantic Similarity Can Sometimes Be Better Than Exact Match

Now consider:

Query:
how much does a private teacher cost

Relevant document:

Average pricing for one-to-one tutoring sessions

Here:

private teacher
↔
one-to-one tutor

cost
↔
pricing

Semantic similarity is important.

So different queries benefit from different retrieval signals.

This motivates combining lexical and semantic approaches later.

30. Lexical Search and Candidate Generation

Lexical retrieval is often extremely useful as an early candidate-generation mechanism.

Huge Corpus
↓
Inverted Index
↓
BM25 / Lexical Retrieval
↓
Candidate Set

Then later systems may apply:

semantic scoring
reranking
other ranking signals

So even in advanced AI search systems, lexical search may remain an important first stage.

31. Lexical Search in RAG

Imagine a RAG system with:

1,000,000 company documents

The user asks:

What does policy HR-204 say about parental leave?

The term:

HR-204

is extremely specific.

Lexical retrieval can quickly locate documents containing that exact identifier.

Query
↓
Lexical Retriever
↓
Relevant Policy Document
↓
Passage
↓
LLM

So lexical retrieval remains useful even when an LLM is part of the final system.

33. A Complete Simplified Lexical Retrieval Example

Suppose the corpus contains:

100,000 business pages

The user searches:

Persian dentist Toronto

Let us follow the process.

Step 1 — The Documents Are Indexed

The system processes the documents and creates an inverted index.

Documents
↓
Terms
↓
Inverted Index

Part of the index might look like:

Persian → Docs 8, 14, 62, 91, ...
dentist → Docs 3, 8, 19, 91, ...
Toronto → Docs 1, 3, 8, 14, 91, ...

Step 2 — Query Terms Are Identified

Persian
dentist
Toronto

Step 3 — Posting Lists Are Retrieved

Persian
→ 8, 14, 62, 91

dentist
→ 3, 8, 19, 91

Toronto
→ 1, 3, 8, 14, 91

Step 4 — Candidate Documents Are Generated

Documents appearing in multiple posting lists become strong candidates.

Doc 8
Doc 91

match all three query terms.

Step 5 — Lexical Relevance Is Scored

The system considers factors such as:

term frequency
term rarity
document length

Suppose:

Doc 8  → BM25 13.8
Doc 91 → BM25 11.4
Doc 14 → BM25 7.6
Doc 3  → BM25 6.9

Step 6 — Documents Are Ranked

1. Doc 8
2. Doc 91
3. Doc 14
4. Doc 3

The lexical retrieval stage has now produced a ranked candidate set.

Later stages may rerank these results using additional signals.

What Happens When the Words Differ?

Now change the query to:

Iranian dental specialist Toronto

Suppose Doc 8 uses:

Persian dentist Toronto

Humans may recognize:

Iranian ↔ Persian

dental specialist ↔ dentist

But lexical overlap has weakened.

The document may receive a lower score or even fail candidate generation.

This is the precise point where lexical retrieval reaches one of its main limits.

34. Traditional Search Is Statistical, Not Truly “Understanding” the Query

BM25 can be extremely effective.

But its fundamental signals still come from lexical statistics.

Which query terms occur?

How often?

How rare are they?

How long is the document?

It does not inherently create the same type of semantic representation we discussed with embeddings.

Lexical Retrieval
→ term-based statistical relevance

Semantic Retrieval
→ meaning-based representation and similarity

This distinction prepares us for the next chapter.

35. Common Misunderstandings

Misunderstanding 1: Keyword search means simply counting repeated words

Incorrect. Modern lexical retrieval can weight terms using frequency, rarity, document length, and other statistics.

Misunderstanding 2: Lexical search understands meaning exactly like an LLM

Incorrect. Its core signals come primarily from words and term statistics.

Misunderstanding 3: Exact matching is always bad

Incorrect. Exact matching is extremely valuable for names, identifiers, rare terminology, codes, and precise phrases.

Misunderstanding 4: More occurrences always mean proportionally more relevance

Incorrect. Methods such as BM25 use term-frequency saturation.

Misunderstanding 5: Term Frequency and Document Frequency are the same

Incorrect.

TF
→ occurrences within one document

DF
→ number of documents containing the term

Misunderstanding 6: A common word should have the same weight as a rare word

Incorrect. Rare terms often provide stronger distinguishing information.

Misunderstanding 7: TF-IDF understands synonyms

Not inherently. TF-IDF is primarily a lexical weighting method.

Misunderstanding 8: BM25 is a semantic embedding model

Incorrect. BM25 is a lexical ranking function.

Misunderstanding 9: BM25 scores are probabilities

Incorrect. They are ranking scores whose absolute values are system-specific.

Misunderstanding 10: A sparse vector is simply a small vector

Incorrect. A sparse vector can be extremely high-dimensional; most of its dimensions are simply zero.

Misunderstanding 11: Sparse means low quality

Incorrect. Sparse describes the representation, not retrieval quality.

Misunderstanding 12: An inverted index stores only complete documents

Incorrect. It maps terms to document-related posting information.

Misunderstanding 13: Vocabulary mismatch means the document is irrelevant

Incorrect. The query and document may express the same meaning using different words.

Misunderstanding 14: Strong keyword overlap guarantees correct intent

Incorrect. Documents can share words while serving different intents.

Misunderstanding 15: Semantic search makes lexical search obsolete

Incorrect. Exact lexical matching remains extremely useful and is often combined with semantic retrieval.

36. How the Concepts Connect

Chapter 7 gave us the retrieval architecture:

Corpus
↓
Indexing
↓
Search Index
↓
Query
↓
Candidate Generation
↓
Ranking
↓
Results

Chapter 8 now explains one way to implement that architecture.

During indexing:

Documents
↓
Terms
↓
Term Statistics
↓
Inverted Index

At query time:

Query
↓
Query Terms
↓
Inverted Index Lookup
↓
Posting Lists
↓
Candidate Documents

Then:

Candidate Documents
+
Term Frequency
+
Document Frequency
+
Document Length
↓
TF-IDF / BM25-Style Scoring
↓
Lexically Ranked Results

So the full lexical retrieval pipeline becomes:

Corpus
↓
Indexing
↓
Inverted Index
↓
User Query
↓
Term Matching
↓
Candidate Generation
↓
BM25 / Lexical Scoring
↓
Ranking
↓
Results

But we have now discovered an important limitation:

Exact Words
≠
Meaning

For example:

car repair

and:

automobile maintenance

can mean very similar things while sharing few or no exact lexical terms.

That leads directly into semantic retrieval.

Mental Model

Imagine a giant library.

Instead of reading every book whenever someone asks a question, the library creates a giant dictionary.

For every important word, the dictionary records:

which books contain that word

For example:

dentist
→ Book 8
→ Book 14
→ Book 91

Toronto
→ Book 3
→ Book 8
→ Book 22
→ Book 91

This is similar to an:

Inverted Index

Now someone searches:

dentist Toronto

The system immediately checks the lists for:

dentist
Toronto

and notices:

Book 8
Book 91

appear in both.

But not all words are equally useful.

If the user searched:

the dentist

the word:

the

appears almost everywhere.

It tells us very little.

The word:

dentist

is much more informative.

That is the intuition behind:

Document Frequency
IDF
TF-IDF
BM25

The system also notices that repeating a word 500 times should not make a document infinitely better.

So BM25 reduces the value of excessive repetition.

And it considers document length.

This gives us:

Exact Terms
+
Term Frequency
+
Term Rarity
+
Length Adjustment
↓
Lexical Relevance

But the librarian has a weakness.

If someone searches:

cheap private teacher

and the best book says:

affordable one-to-one tutor

the traditional word-based system may not connect them.

Humans see the shared meaning.

The lexical index mainly sees different words.

That limitation prepares us for the next stage:

Words
↓
Meaning

The simplest reliable mental model is:

Lexical retrieval finds information by matching query terms against terms in indexed documents. Inverted indexes make that matching efficient, while methods such as TF-IDF and BM25 use term frequency, term rarity, and document-level statistics to rank the matches. Its major limitation is that similar meaning can be expressed with different vocabulary.

Chapter 8 Recap

Keyword Search

Searching for documents using words or terms from a query.

Query Keywords
↓
Matching Documents
Lexical Search

Retrieval based primarily on lexical forms and term statistics.

Query Terms
↔
Document Terms
Exact Match

A direct lexical match between a query term or phrase and the document.

Exact matching is particularly useful for:

names
codes
identifiers
rare terminology
specific phrases
Term Matching

Comparing query terms against document terms to identify possible relevance.

Keyword Relevance

The degree to which lexical matches provide evidence that a document satisfies the query.

Term Frequency — TF

The number of times a term appears in a particular document.

TF
=
frequency inside one document
Document Frequency — DF

The number of documents in the corpus that contain a term.

DF
=
frequency across documents
Inverse Document Frequency — IDF

A weighting concept that gives greater importance to terms occurring in fewer documents.

Common Term
→ Lower IDF

Rare Term
→ Higher IDF
TF-IDF

A lexical weighting approach combining:

Term Frequency
+
Inverse Document Frequency

Conceptually:

Frequent in this document
+
Rare across the corpus
↓
Potentially important term
Sparse Retrieval

Term-based retrieval using high-dimensional representations in which most values are zero.

Sparse Vector

A vector containing mostly zero-valued dimensions.

For lexical retrieval, dimensions can correspond directly to vocabulary terms.

Vocabulary:
dentist
restaurant
Toronto
lawyer
Persian

Document:
Persian dentist Toronto

Vector:
[1, 0, 1, 0, 1]
Inverted Index

A search structure mapping terms to documents containing those terms.

Term
↓
Documents

Example:

dentist
→ Doc 8
→ Doc 15
→ Doc 91
Posting List

The list of document information associated with a term inside an inverted index.

BM25

A widely used lexical ranking function.

Conceptually, it considers:

term matches
term frequency
term rarity
term-frequency saturation
document length

to calculate a ranking score.

Term-Frequency Saturation

The principle that repeated occurrences provide diminishing additional relevance.

1 → 2 mentions
may matter substantially

100 → 101 mentions
usually matters much less
Document Length Normalization

Adjusting lexical relevance so that long documents do not gain an unfair advantage simply because they contain more words.

Vocabulary Mismatch

A situation where the query and relevant document express similar meaning using different vocabulary.

cheap
↔
affordable

car
↔
automobile

lawyer
↔
attorney
Synonyms

Different words that express similar meaning and can create vocabulary mismatch in lexical retrieval.

Search Intent Mismatch

A situation where a document matches query terms but does not satisfy the user's underlying information need.

Keyword Match
≠
Intent Match
Exact Words vs. Meaning

The fundamental limitation of lexical retrieval is:

Word Match
≠
Meaning Match

A relevant document can use different words.

And an irrelevant document can contain the same words.

Strength of Lexical Retrieval

Lexical retrieval is especially strong when:

exact wording matters
rare terminology matters
identifiers matter
product names matter
codes matter
specific phrases matter
Limitation of Lexical Retrieval

Lexical retrieval can struggle when:

query vocabulary
≠
document vocabulary

even when the underlying meanings are similar.

The complete Chapter 8 pipeline is:

Corpus
↓
Documents
↓
Term Processing
↓
Inverted Index
↓
User Query
↓
Query Terms
↓
Term Matching
↓
Candidate Generation
↓
TF / DF / IDF Signals
↓
BM25 / Lexical Scoring
↓
Ranking
↓
Results

This gives us a powerful search system.

But it is still fundamentally centered around:

words

The next question is:

Can we retrieve documents based on meaning even when the exact words are different?

For example:

Query:
cheap private teacher

Document:
affordable one-to-one tutor

A human immediately recognizes the semantic similarity.

How can a machine do the same?

That leads directly into Chapter 9:

Dense and Semantic Retrieval.

The Single Most Important Idea from Chapter 8

Lexical retrieval finds documents by matching the terms in a query against terms in an indexed corpus. Inverted indexes make this efficient, while TF-IDF and BM25 use statistics such as term frequency, document frequency, rarity, and document length to estimate relevance. Its major weakness is vocabulary mismatch: two texts can mean the same thing while using different words.

← Back to AI Visibility Guide