Chapter 9- Dense and Semantic Retrieval

Chapter Goal

In Chapter 8, we learned how lexical retrieval finds information primarily through words and term statistics.

A simplified lexical pipeline was:

Corpus
↓
Documents
↓
Inverted Index
↓
User Query
↓
Term Matching
↓
BM25 / Lexical Scoring
↓
Ranked Results

This works extremely well when the query and relevant document use similar vocabulary.

For example:

Query:
Persian dentist Toronto

Document:
Persian dentist in Toronto offering emergency dental care.

The lexical connection is obvious.

But Chapter 8 also revealed a major limitation.

Suppose the query is:

cheap private teacher

while the relevant document says:

affordable one-to-one tutor

A human immediately sees the relationship:

cheap
≈
affordable

private teacher
≈
one-to-one tutor

But lexical overlap is weak.

This creates the vocabulary mismatch problem.

Chapter 9 introduces another retrieval approach:

Dense Retrieval and Semantic Retrieval.

Instead of representing text mainly through explicit vocabulary terms, we can represent the meaning of a query or document using an embedding vector.

Query
↓
Embedding Model
↓
Query Vector

and:

Document
↓
Embedding Model
↓
Document Vector

Then retrieval can compare these vectors.

If their representations are close in vector space:

Query Vector
≈
Document Vector

the system may consider them semantically related even when the exact words differ.

For example:

cheap private teacher

and:

affordable one-to-one tutor

may receive similar vector representations.

This allows retrieval based more directly on:

meaning

rather than only:

word overlap

By the end of this chapter, you should understand:

what dense retrieval is;
what semantic retrieval is;
what a dense vector is;
how dense vectors differ from sparse vectors;
what a text embedding is in retrieval;
what an embedding model does;
how queries are embedded;
how documents are embedded;
what an embedding space is;
what semantic similarity means;
what vector similarity means;
what cosine similarity is;
what dot product is;
what Euclidean distance is;
what nearest-neighbor search means;
what a vector index is;
what approximate nearest-neighbor search is;
what ANN means;
how dense retrieval performs candidate generation;
why dense retrieval can solve vocabulary mismatch;
why semantic similarity is not the same as relevance;
why semantic similarity is not factual correctness;
where dense retrieval performs well;
where dense retrieval can fail;
and why modern systems often combine lexical and dense retrieval.

The central pipeline of this chapter is:

Documents
↓
Embedding Model
↓
Document Embeddings
↓
Vector Index

User Query
↓
Embedding Model
↓
Query Embedding
↓
Vector Similarity Search
↓
Nearest Documents
↓
Ranked Candidates

1. From Words to Meaning

Chapter 8 centered on:

words

A lexical system asks questions such as:

Does this query term appear in the document?

How often?

How rare is the term?

How long is the document?

Dense retrieval takes a different approach.

Instead of representing a document mainly as explicit vocabulary terms, it creates a numerical representation intended to capture useful aspects of its meaning.

Text
↓
Embedding Model
↓
Dense Vector

Then retrieval becomes a comparison between numerical representations.

Query Meaning
↓
Query Vector

Document Meaning
↓
Document Vector

Compare Vectors
↓
Estimate Similarity

This is the foundation of dense retrieval.

2. What Is Dense Retrieval?

Dense retrieval is a retrieval approach in which queries and searchable items are represented as dense numerical vectors.

For example:

Query:
cheap private math tutor

might become conceptually:

[0.18, -0.42, 0.71, 0.09, ...]

A document:

Affordable one-to-one mathematics lessons

might become:

[0.21, -0.39, 0.69, 0.12, ...]

If the vectors are close, the system can infer that the texts may be related.

Text
↓
Dense Embedding
↓
Vector Comparison
↓
Potential Relevance

The word dense describes the representation.

Most vector dimensions contain non-zero numerical values.

3. What Is a Dense Vector?

A dense vector is a numerical vector where most dimensions contain meaningful non-zero values.

For example:

[0.18, -0.42, 0.71, 0.09, -0.31, 0.54]

Compare this with the sparse vector from Chapter 8:

[0, 0, 2.4, 0, 0, 0, 1.3, 0, 0, ...]

The sparse vector might have:

500,000 dimensions

with only a few hundred non-zero values.

A dense vector might have:

dimensions, for example, with values in most or all dimensions.

The exact dimensionality depends on the embedding model.

4. Sparse Vector vs. Dense Vector

Sparse Lexical Vector

Suppose the vocabulary is:

[
  dentist,
  restaurant,
  Toronto,
  lawyer,
  Persian
]

Document:

Persian dentist Toronto

might be represented conceptually as:

[1, 0, 1, 0, 1]

Here:

Dimension 1 → dentist
Dimension 2 → restaurant
Dimension 3 → Toronto
Dimension 4 → lawyer
Dimension 5 → Persian

The dimensions have relatively explicit lexical meanings.

Dense Vector

The same text might instead become:

[0.14, -0.37, 0.82, 0.11, -0.56, ...]

But we normally cannot say:

Dimension 1 = dentist
Dimension 2 = Persian
Dimension 3 = Toronto

The dimensions are learned features distributed across the representation.

Sparse Vector
→ explicit vocabulary dimensions
→ mostly zeros

Dense Vector
→ learned latent dimensions
→ mostly non-zero values

5. Dense Does Not Mean “More Accurate”

Just as:

sparse

does not mean:

bad

the word:

dense

does not mean:

better
more intelligent
more accurate

It describes the representation.

Dense retrieval has advantages.

Lexical retrieval has advantages.

Different queries benefit from different signals.

This is why modern systems often combine them.

6. What Is Semantic Retrieval?

Semantic retrieval tries to retrieve information based on meaning rather than relying only on exact lexical overlap.

Consider:

Query:
how much does a private teacher cost?

Document:

Average pricing for one-to-one tutoring sessions

Lexically, several words differ.

Semantically:

private teacher
≈
one-to-one tutor

cost
≈
pricing

A semantic retrieval system may recognize this similarity.

Different Words
↓
Similar Meaning
↓
Similar Representations
↓
Potential Retrieval Match

7. Dense Retrieval and Semantic Retrieval Are Related but Not Identical

These terms are often used together.

A useful distinction is:

Dense Retrieval
→ describes the representation and retrieval mechanism

Semantic Retrieval
→ describes the goal of matching meaning

Dense embeddings are commonly used to perform semantic retrieval.

Dense Vector Retrieval
↓
often enables
↓
Semantic Matching

But the concepts are not perfectly synonymous.

A dense vector system can still produce poor semantic matches.

And semantic relevance can sometimes also be improved through non-dense methods.

8. Returning to Embeddings

Chapter 2 introduced embeddings.

An embedding is a learned numerical representation.

For retrieval, we often want to represent a larger piece of text such as:

query
sentence
paragraph
passage
document

as one vector.

For example:

"Persian dentist in Toronto"
↓
Embedding Model
↓
[0.18, -0.42, 0.71, ...]

This is often called a text embedding.

9. Token Embeddings vs. Text Embeddings

Token Embedding

A token receives a numerical representation inside the model.

"mouse"
↓
Token Embedding

Transformer layers then contextualize token representations.

Text Embedding

For retrieval, an entire text unit may be represented by one vector.

For example:

"best Persian dentist in Toronto"
↓
Embedding Model
↓
One Text Vector

Token Embedding
→ represents an individual token

Text Embedding
→ represents a larger piece of text

Dense retrieval generally works with text-level representations.

10. Embedding Is a Representation, Not the Retrieval Process

Do not confuse:

Embedding

with:

Retrieval

An embedding is a numerical representation.

Retrieval is the process of finding relevant information.

Text
↓
Embedding Model
↓
Embedding

then:

Embedding
↓
Similarity Search
↓
Retrieval

Embedding
≠
Retrieval

Embeddings are one tool retrieval systems can use.

11. What Is an Embedding Model?

An embedding model is a model designed to convert input into vector representations useful for comparing relationships between inputs.

For retrieval:

Text
↓
Embedding Model
↓
Vector

For example:

"immigration lawyer Toronto"
↓
Embedding Model
↓
Vector A

and:

"Toronto attorney specializing in immigration cases"
↓
Embedding Model
↓
Vector B

If the embedding model represents these texts effectively:

Vector A
≈
Vector B

even though:

lawyer
≠
attorney

lexically.

12. Embedding Space

The Embedding Model Has Learned a Representation Space

An embedding model does not simply assign random numbers to text.

Through training, it learns a representation space in which useful relationships can be reflected by vector positions.

Similar Meaning
→ Nearby Regions

Different Meaning
→ More Distant Regions

For example:

immigration lawyer
immigration attorney
visa lawyer

may occupy related regions.

While:

chocolate cake recipe

may be far away.

What Is an Embedding Space?

An embedding space is the multidimensional numerical space in which embeddings exist.

A simple two-dimensional illustration might look like:

                        Dental Clinic
                              ●
                         ● Dentist
                    ● Orthodontist


     Immigration Lawyer ●
                ● Attorney


                                      ● Persian Restaurant
                                ● Iranian Restaurant

Real embedding spaces usually contain hundreds or thousands of dimensions.

We cannot visualize them directly like a two-dimensional chart.

But the same idea applies:

position
+
distance
+
direction

can encode useful relationships.

The Meaning Is Distributed Across Dimensions

In a lexical sparse vector, one dimension might correspond directly to:

dentist

But in a dense embedding:

[0.18, -0.42, 0.71, ...]

the meaning of:

dentist

is not usually stored in one single dimension.

Instead, information is distributed across many dimensions.

Meaning
↓
Distributed Pattern Across Vector

This is one reason individual dense-vector dimensions are difficult to interpret directly.

13. Document Embeddings

To perform dense retrieval, documents—or smaller searchable units—can be embedded in advance.

Suppose the corpus contains:

Document A:
Persian dental clinic in Toronto

Document B:
Italian restaurant in Toronto

Document C:
Affordable one-to-one math tutoring

The embedding model produces:

Document A
↓
Vector A

Document B
↓
Vector B

Document C
↓
Vector C

These vectors can then be stored in a searchable structure.

14. Query Embeddings

When the user submits:

Iranian dentist Toronto

the query is passed through the embedding model:

Iranian dentist Toronto
↓
Embedding Model
↓
Query Vector

Now the system can compare:

Query Vector

against:

Vector A
Vector B
Vector C
...

The goal is to identify vectors nearest or most similar to the query vector.

15. Query and Documents Need Comparable Representations

For dense retrieval to work effectively, query and document representations need to exist in a compatible vector space.

Query
↓
Embedding Model
↓
Query Vector

Documents
↓
Embedding Model
↓
Document Vectors

Then:

Query Vector
↔
Document Vectors

can be meaningfully compared.

If two completely incompatible embedding spaces were used, numerical similarity would generally not have the intended meaning.

16. A Simplified Dense Retrieval Example

Suppose we have three documents.

Document A:
Affordable one-to-one mathematics tutoring.

Document B:
Luxury hotels in downtown Toronto.

Document C:
Private math lessons for high school students.

Query:

cheap private math teacher

Lexically:

Document A
cheap ✗
private ✗
math ✓ partially
teacher ✗

Document C
private ✓
math ✓
teacher ✗

Document A has weak exact overlap.

But after embedding:

Query Vector       → [ ... ]

Document A Vector  → [ ... ]
Document B Vector  → [ ... ]
Document C Vector  → [ ... ]

the system may find:

Query ↔ Document A → high similarity
Query ↔ Document C → high similarity
Query ↔ Document B → low similarity

This is semantic retrieval.

17. What Is Vector Similarity?

Once we have vectors, we need a way to compare them.

This is called:

vector similarity.

Query Vector
+
Document Vector
↓
Similarity Function
↓
Similarity Score

For example:

Query ↔ Document A → 0.91
Query ↔ Document B → 0.19
Query ↔ Document C → 0.86

The exact score depends on the model and similarity method.

Common methods include:

cosine similarity
dot product
Euclidean distance

18. Cosine Similarity

Cosine similarity compares the direction of two vectors.

Imagine two arrows.

If they point in almost the same direction:

high cosine similarity

If they point in very different directions:

lower cosine similarity

Conceptually:

Vector A  ↗
Vector B  ↗

→ similar direction
→ high cosine similarity

versus:

Vector A  ↗
Vector B  ↙

→ different direction
→ low similarity

Why Direction Can Represent Similarity

Suppose two text vectors have similar patterns across their dimensions.

Even if their exact magnitudes differ, their directions can remain similar.

Cosine similarity focuses primarily on this angular relationship.

Its common range is:

-1 to 1

although the values observed in practical embedding systems depend on the representation.

closer to 1
→ more similar direction

closer to 0
→ weaker directional relationship

closer to -1
→ opposite direction

Do not interpret those values as factual probabilities.

Cosine Similarity Is Not “Percentage of Same Meaning”

Suppose:

cosine similarity = 0.87

This does not mean:

87% same meaning

It is a mathematical similarity score.

Its interpretation depends on:

embedding model
data
task
distribution

Cosine Similarity
≠
Probability of Relevance

and:

Cosine Similarity
≠
Probability of Truth

19. Dot Product

Another common vector comparison method is the dot product.

At a high level:

Vector A
·
Vector B
↓
Dot Product Score

The dot product combines corresponding dimensions from two vectors.

If the embedding model is trained for dot-product retrieval, higher values may indicate stronger compatibility.

You do not need the detailed mathematics yet.

Dot product is another way to assign a numerical similarity or compatibility score between vectors.

Cosine Similarity vs. Dot Product

These measures are related but not identical.

Cosine similarity focuses on:

vector direction

Dot product depends on:

direction
+
magnitude

If vectors are normalized to the same length, cosine similarity and dot product become closely related.

Different embedding systems may be designed to use different similarity functions.

So you should not assume:

every vector database
uses cosine similarity

20. Euclidean Distance

Another concept is Euclidean distance.

Instead of asking:

How similar are these vectors?

we ask:

How far apart are these vectors?

Small Distance
→ vectors are close

Large Distance
→ vectors are far apart

So:

Similarity
↑
usually corresponds conceptually to
Distance
↓

depending on the representation and metric.

Similarity vs. Distance

These concepts point in opposite numerical directions.

For similarity:

higher
→ more similar

For distance:

lower
→ closer

Example:

Cosine Similarity:

Document A → 0.93
Document B → 0.31

versus conceptually:

Distance:

Document A → 0.12
Document B → 1.48

In both cases Document A is considered closer to the query.

21. What Is Nearest-Neighbor Search?

Suppose we have:

1,000,000 document vectors

and one query vector.

We want to find the document vectors located closest to the query.

This is called nearest-neighbor search.

Query Vector
↓
Search Vector Space
↓
Find Nearest Document Vectors
↓
Return Documents

For example:

Query Vector
↓
Nearest Neighbors

1. Document 84
2. Document 15
3. Document 9,201
4. Document 440

These documents become retrieval candidates.

22. Top-K Retrieval

Dense retrieval systems often return the top K nearest results.

For example:

K = 5

means:

return the 5 nearest documents

Query Vector
↓
Similarity Search
↓
Top-5 Nearest Vectors
↓
5 Retrieved Items

So when you see:

top-k retrieval

it means retrieving a specified number of the highest-scoring or nearest items.

23. Vector Index

In principle, a system could compare a query vector against every document vector.

For:

1,000 document vectors

this may be manageable.

But imagine:

100,000,000 vectors

For every query, computing expensive similarity comparisons against every vector can become costly.

This creates a scaling problem.

Just as lexical search uses structures such as an inverted index, dense retrieval commonly uses a:

vector index.

A vector index is a structure designed to make similarity search across many vectors efficient.

Document Embeddings
↓
Vector Index

Then at query time:

Query Embedding
↓
Vector Index Search
↓
Nearby Vectors
↓
Candidate Documents

Inverted Index vs. Vector Index

Now we can compare Chapters 8 and 9.

Lexical Retrieval

Documents
↓
Terms
↓
Inverted Index

At query time:

Query Terms
↓
Posting Lists
↓
Matching Documents

Dense Retrieval

Documents
↓
Embeddings
↓
Vector Index

At query time:

Query Embedding
↓
Nearest-Neighbor Search
↓
Similar Documents

Inverted Index
→ optimized for term-based lookup

Vector Index
→ optimized for vector-similarity lookup

24. Approximate Nearest-Neighbor Search

Searching every vector exactly can be expensive at large scale.

So many dense retrieval systems use:

Approximate Nearest Neighbor, abbreviated ANN.

Instead of guaranteeing an exhaustive comparison against every vector, the system uses specialized indexing techniques to quickly find vectors that are very likely to be among the closest.

Huge Vector Collection
↓
ANN Index
↓
Fast Approximate Search
↓
Likely Nearest Neighbors

Why “Approximate”?

Suppose the mathematically exact nearest vectors are:

1. A
2. B
3. C
4. D
5. E

An approximate search might return:

1. A
2. B
3. D
4. C
5. F

It may miss:

in exchange for dramatically faster retrieval.

Speed
↔
Exactness

At large scale, this trade-off can be extremely useful.

ANN Does Not Mean Random Search

Approximate nearest-neighbor search does not mean:

randomly choose similar-looking documents

ANN algorithms build data structures that organize vector space so that likely nearby vectors can be found efficiently.

The approximation comes from avoiding an exhaustive comparison against every possible vector.

Exact Search
→ inspect everything

ANN Search
→ intelligently search likely regions

25. Dense Retrieval as Candidate Generation

Now connect dense retrieval back to Chapter 7.

Recall:

Huge Corpus
↓
Candidate Generation
↓
Candidate Set

Dense retrieval can perform this candidate-generation stage.

Query
↓
Query Embedding
↓
Vector Index
↓
Nearest Neighbors
↓
Candidate Set

For example:

10,000,000 documents
↓
Dense Retrieval
↓
Top 200 candidates

Those candidates can then enter later ranking or reranking stages.

26. A Typical Dense Retrieval Preparation Stage

Before queries arrive, document embeddings can be computed.

Corpus
↓
Documents / Passages
↓
Embedding Model
↓
Dense Embeddings
↓
Vector Index

This is largely preparation work.

It does not need to happen from scratch for every query.

If documents change, their embeddings may need to be created or updated.

27. A Typical Dense Retrieval Query-Time Stage

When the user submits:

affordable immigration attorney in Toronto

the system performs:

User Query
↓
Embedding Model
↓
Query Embedding
↓
Vector Index Search
↓
Nearest Document Embeddings
↓
Candidate Documents

Then those candidates may be:

returned directly

or:

reranked

or:

passed into a RAG pipeline

28. Bi-Encoder Retrieval

A common dense-retrieval architecture encodes the query and documents separately.

Query
↓
Encoder
↓
Query Vector

and:

Document
↓
Encoder
↓
Document Vector

Because document vectors can be computed in advance, retrieval can remain efficient.

This general architecture is often associated with:

bi-encoder or dual-encoder retrieval.

What Is a Bi-Encoder?

A bi-encoder represents the query and document separately.

Query
↓
Encoder
↓
Query Vector

and:

Document
↓
Encoder
↓
Document Vector

Then:

Query Vector
↔
Document Vector
↓
Similarity Score

The key advantage is that document vectors can be precomputed.

So at query time the system does not need to run an expensive joint comparison between the query and every full document.

Why Precompute Document Embeddings?

Suppose you have:

10,000,000 documents

If every query required re-encoding all ten million documents, retrieval would be extremely inefficient.

Instead:

Documents
↓
Embed Once
↓
Store Vectors

Then each new query requires only:

Embed Query
↓
Search Existing Vector Index

Expensive document processing
→ done in advance

Query processing
→ done at request time

29. Query–Document Similarity

Dense retrieval estimates relevance through the relationship between:

Query Embedding

and:

Document Embedding

For example:

Query:
heart doctor near me

Document A:

Cardiology clinic providing specialist heart care

may have high vector similarity.

Document B:

Home gardening tools and equipment

should ideally have low similarity.

Query Meaning
↔
Document Meaning
↓
Similarity Score

30. Vocabulary Mismatch and Paraphrases

Dense Retrieval Solves Some Vocabulary Mismatch

Return to Chapter 8:

Query:
car repair

Document:

automobile maintenance services

A lexical system sees:

car ≠ automobile
repair ≠ maintenance

A well-trained dense embedding model may instead place them near each other because their meanings are related.

Different Words
↓
Similar Learned Representation
↓
High Vector Similarity
↓
Document Can Be Retrieved

This is one of dense retrieval's major strengths.

Dense Retrieval Can Capture Paraphrases

Consider:

Query:
How can I cancel my subscription?

Relevant passage:

To terminate your membership, open the billing section and choose End Plan.

Exact lexical overlap is limited.

But semantically:

cancel
≈
terminate

subscription
≈
membership

A semantic retriever may still retrieve the passage.

This ability to match paraphrases is extremely useful.

Dense Retrieval Can Capture Conceptual Relationships

Dense retrieval can sometimes go beyond simple synonyms.

For example:

Query:
symptoms of low blood sugar

A relevant document may discuss:

hypoglycemia symptoms

The connection is not merely a simple word substitution.

The model needs to have learned a conceptual relationship:

low blood sugar
↔
hypoglycemia

Dense representations can help encode these relationships.

31. Limits of Semantic Similarity

Semantic Similarity Is Not the Same as Relevance

This is one of the most important limitations.

Suppose the query is:

buy running shoes

A document discussing:

history of running shoes

may be semantically very similar.

Both discuss:

running shoes

But the user's intent is:

purchase

Semantic Similarity
≠
Perfect Relevance

A retrieval system needs more than broad topical similarity.

Semantic Similarity Can Be Too Broad

Suppose the query is:

Python installation error 0x80070643

Dense retrieval might identify documents about:

Python installation errors
Windows installation problems
software setup failures

These are semantically related.

But the exact identifier:

0x80070643

may be critical.

Lexical retrieval may be better at preserving that exact detail.

So semantic similarity can sometimes be too broad.

Dense Retrieval Can Blur Important Distinctions

Consider:

immigration lawyer Toronto

and:

immigration lawyer Vancouver

Semantically, the documents can be extremely similar.

But:

Toronto

versus:

Vancouver

may be essential to the user.

Similarly:

iPhone 17

versus:

iPhone 16

are semantically close but not interchangeable.

Dense retrieval can sometimes underweight precise lexical differences.

32. Entity and Identifier Problems

Dense retrieval can struggle with highly specific strings such as:

HR-204

E11000

ZX-4102-B

Case No. 21-CV-8842

The semantic meaning of these strings may be limited.

Their exact identity is what matters.

Lexical retrieval is often especially valuable in these situations.

Dense Retrieval Does Not “Understand” Like a Human

It is tempting to say:

Dense retrieval understands meaning.

That is useful shorthand.

But technically, the system computes learned numerical representations and similarity.

Dense retrieval uses learned vector representations that can encode semantic relationships well enough to retrieve texts with similar or related meaning.

It does not imply human-like understanding.

33. Embedding and Retrieval Quality

High Vector Similarity Does Not Guarantee Truth

Suppose the query is:

Who founded Velora Systems?

A fabricated page says:

Velora Systems was founded by David Chen in 2018.

If that page is semantically very close to the query, dense retrieval may rank it highly.

High Semantic Similarity
≠
Factual Accuracy

This is the same relevance-versus-truth distinction from Chapter 7.

High Vector Similarity Does Not Guarantee Good Evidence

A document may be semantically relevant but weak as evidence.

For example:

Query:
What is the recommended dosage of Drug X?

A forum post may be semantically very relevant.

An official prescribing document may also be relevant.

Vector similarity alone does not automatically determine:

authority
trustworthiness
freshness
evidence quality

Those require additional signals.

This distinction becomes very important later in AI Search and RAG.

Dense Retrieval Can Retrieve the Wrong Sense of a Concept

Consider:

Query:
Apple repair

A good embedding system may understand that:

Apple

probably refers to the technology company.

But ambiguity can remain.

Similarly:

Jaguar

might refer to:

animal
car brand
sports team
software project

Context helps, but dense representations are not perfect.

Query Embedding Quality Matters

Suppose the embedding model poorly represents the user's query.

Query
↓
Weak Query Embedding
↓
Wrong Region of Vector Space
↓
Weak Candidate Retrieval

The downstream ranker cannot recover documents that were never retrieved.

This connects directly to Chapter 7:

Candidate Generation Quality
↓
Constrains
↓
Possible Final Ranking Quality

Document Embedding Quality Matters

The same is true for documents.

Suppose a long document contains many unrelated topics:

taxes
restaurants
housing
immigration
weather

Representing the entire document using one vector may blur its individual topics.

A query about:

immigration

might not match as strongly as expected.

This is one reason retrieval systems often operate on smaller passages or chunks rather than entire large documents.

We will explore chunking more deeply in the RAG section.

34. Passage-Level Dense Retrieval

Instead of embedding an entire long document:

10,000-word document
↓
One Vector

we can divide it into smaller units:

Passage 1
↓
Vector 1

Passage 2
↓
Vector 2

Passage 3
↓
Vector 3

Then:

Query Vector
↓
Compare Against Passage Vectors
↓
Retrieve Most Relevant Passage

This can improve retrieval specificity.

Document-Level vs. Passage-Level Dense Retrieval

Consider a document:

Complete Guide to Studying in Canada

containing:

Universities
Tuition
Study Permits
Housing
Healthcare

Query:

documents required for a Canadian study permit

The whole document is relevant.

But the:

Study Permits passage

is much more directly relevant.

Document Embedding
→ broad topic representation

Passage Embedding
→ more focused representation

Neither is always universally better.

Granularity depends on the retrieval task.

35. A Vector Database

You may also encounter the term:

vector database.

A vector database is a system designed to store, manage, and search vector representations, often together with associated metadata.

Conceptually:

Document ID
Text
Metadata
Embedding

might be stored together.

For example:

ID: 184
City: Toronto
Category: Dentist
Text: Persian dental clinic...
Embedding: [0.18, -0.42, ...]

Then the system can search based on vector similarity.

Vector Database vs. Vector Index

These terms should not be treated as perfectly identical.

A vector index is primarily the data structure used to make vector similarity search efficient.

A vector database is a broader system that may provide:

vector storage
metadata storage
filtering
index management
updates
deletion
query APIs

Vector Database
may contain
↓
Vector Index

Vector Index
≠
Entire Vector Database

36. Metadata Filtering

Dense retrieval does not always rely only on vector similarity.

Suppose the query is:

Persian dentist in Toronto

The system might perform:

Semantic Similarity
+
Metadata Filter:
city = Toronto
category = dentist

Vector Search
+
Structured Filters
↓
More Targeted Results

This can help preserve important constraints that pure semantic similarity might blur.

37. Dense Retrieval and Ranking

Dense retrieval can produce its own similarity-based ranking.

For example:

Doc A → 0.91
Doc B → 0.87
Doc C → 0.73
Doc D → 0.42

Then:

1. Doc A
2. Doc B
3. Doc C
4. Doc D

But this does not mean the dense similarity score must be the final ranking signal.

Later systems may combine:

dense score
lexical score
freshness
authority
location
personalization
reranker score

38. Dense Retrieval vs. Reranking

Dense retrieval and reranking are different stages.

Dense retrieval may do:

10,000,000 Documents
↓
Dense Retriever
↓
200 Candidates

Then a reranker may do:

200 Candidates
↓
More Detailed Query–Document Evaluation
↓
20 Better-Ordered Candidates

Dense Retriever
→ broad candidate generation using embeddings

Reranker
→ deeper evaluation of a smaller candidate set

We will study reranking more deeply later.

39. Dense Retrieval vs. LLM Generation

Dense retrieval does not itself generate the final natural-language answer.

It retrieves existing information.

User Query
↓
Query Embedding
↓
Vector Search
↓
Relevant Passage

Generation happens later:

Relevant Passage
+
User Query
↓
LLM
↓
Generated Answer

Dense Retrieval
≠
Generation

40. Semantic Search Is Not “Search the LLM's Memory”

An embedding model may use Transformer technology.

But dense retrieval itself should not be confused with a conversational LLM.

The system may contain:

Embedding Model

whose job is:

Text
↓
Vector

rather than:

Text
↓
Long Generated Answer

Different models can be optimized for different tasks.

Dense retrieval also does not mean:

ask the LLM what it remembers

Instead:

External Documents
↓
Embeddings
↓
Vector Index

Then:

Query
↓
Embedding
↓
Search External Vector Index

The retrieved information exists outside the model parameters.

This distinction becomes central in RAG.

41. Lexical Search and Dense Search Use Different Evidence

Consider:

Query:
affordable immigration attorney Toronto

Lexical retrieval may reward:

affordable
immigration
attorney
Toronto

Dense retrieval may also connect:

cheap
visa lawyer
Toronto

because of semantic similarity.

Lexical Evidence
→ shared terms

Dense Evidence
→ learned representational similarity

These signals can complement each other.

42. Dense Retrieval Strengths and Limitations

Dense Retrieval Can Find Documents with Almost No Word Overlap

For example:

Query:
How can I stop paying for my plan?

Document:

To terminate your subscription, open Billing and select Cancel Membership.

Possible exact overlap:

very little

Yet meaning:

strongly related

Dense retrieval can be especially useful here.

Lexical Retrieval Can Beat Dense Retrieval

Now consider:

Query:
E11000 duplicate key error

A document containing exactly:

MongoDB E11000 duplicate key error

has extremely valuable lexical evidence.

A dense system may return broader documents about:

database errors
duplicate records
MongoDB problems

Lexical retrieval may identify the precise answer more reliably.

Dense
≠
Always Better

The Vocabulary Mismatch Problem Is Reduced, Not Eliminated

Dense retrieval helps with:

synonyms
paraphrases
related concepts
different phrasing

But it does not solve every language mismatch perfectly.

Embedding models can still misunderstand:

rare terminology
new entities
specialized jargon
numbers
identifiers
subtle distinctions

Dense Retrieval
reduces vocabulary mismatch

but does not:

eliminate retrieval error

Numbers Can Be Difficult

Consider:

Model 15

versus:

Model 16

or:

2025 policy

versus:

2026 policy

These texts may be semantically extremely similar.

But the number may completely change the answer.

Dense representations can sometimes treat these texts as very close.

Lexical or structured constraints can help preserve those distinctions.

Negation Can Be Difficult

Consider:

Document A:
This medicine is recommended during pregnancy.

and:

Document B:
This medicine is not recommended during pregnancy.

The texts share most of their vocabulary and overall topic.

Their meanings differ critically because of:

not

Dense representations can capture negation to some degree, but subtle differences like this remain challenging.

Semantic similarity is therefore not identical to logical equivalence.

43. Dense Retrieval and Information Competition

Recall the idea of information competition.

A vector search might find:

50 semantically related passages

but the downstream system may only use:

Top 5

Therefore retrieval scores determine which information gets the opportunity to enter later stages.

Many Relevant Candidates
↓
Similarity Ranking
↓
Top-K
↓
Limited Information Survives

This will become important in AI Visibility.

44. A Complete Simplified Dense Retrieval Example

Suppose a company has:

500,000 support passages

The user asks:

How do I stop my yearly membership?

The best document uses different wording:

To cancel an annual subscription, open Account Billing and select End Subscription.

Let us follow the retrieval process.

Step 1 — Documents Are Prepared

The documents are split into searchable passages.

Documents
↓
Passages

Step 2 — Each Passage Is Embedded

Passage 1
↓
Embedding Model
↓
Vector 1

Passage 2
↓
Embedding Model
↓
Vector 2

...

Step 3 — Vectors Are Indexed

500,000 Passage Vectors
↓
Vector Index

Step 4 — User Query Arrives

How do I stop my yearly membership?

Step 5 — Query Is Embedded

Query
↓
Embedding Model
↓
Query Vector

Step 6 — Vector Search Runs

The query vector is compared with likely nearby passage vectors.

Query Vector
↓
ANN Search
↓
Nearest Passage Vectors

Step 7 — Similarity Scores Are Produced

Illustratively:

Annual Subscription Cancellation → 0.93
Refund Policy                   → 0.81
Monthly Billing                 → 0.77
Password Reset                  → 0.19

Step 8 — Top Candidates Are Returned

1. Annual Subscription Cancellation
2. Refund Policy
3. Monthly Billing

Notice that the user said:

stop
yearly
membership

while the retrieved passage said:

cancel
annual
subscription

Dense retrieval connected the meanings despite vocabulary mismatch.

45. The Same Example with Lexical Retrieval

Query:

How do I stop my yearly membership?

Relevant passage:

To cancel an annual subscription...

A strict lexical system may see weak matches:

stop        ≠ cancel
yearly      ≠ annual
membership  ≠ subscription

Dense retrieval may see:

stop        ≈ cancel
yearly      ≈ annual
membership  ≈ subscription

This illustrates the fundamental difference:

Lexical Retrieval
→ surface-word relationship

Dense Retrieval
→ learned representational relationship

But Now Change the Query

Suppose the query becomes:

Policy HR-204 annual leave

The exact identifier:

HR-204

may be more important than broad semantic meaning.

A dense retriever could find other HR policy documents that are semantically similar.

A lexical retriever can strongly prioritize the exact:

HR-204

match.

So the systems have complementary strengths.

46. Why Hybrid Retrieval Becomes Natural

At this point, we have two powerful approaches.

Lexical Retrieval

Strong at:

exact words
names
identifiers
codes
rare phrases
precise terminology

Dense Retrieval

Strong at:

synonyms
paraphrases
different wording
conceptual similarity
vocabulary mismatch

Instead of asking:

Which one is universally better?

a better question is:

How can we use both?

That leads naturally to Hybrid Retrieval.

47. Common Misunderstandings

Misunderstanding 1: Dense retrieval means searching every word semantically

Incorrect. Dense retrieval represents larger text units as learned vectors and compares those vectors.

Misunderstanding 2: Dense vector means a vector with many dimensions

Not exactly. Dense means that most dimensions contain non-zero values.

Misunderstanding 3: Dense means better than sparse

Incorrect. Dense and sparse describe different representations with different strengths.

Misunderstanding 4: Every dimension in a dense embedding corresponds to one concept

Incorrect. Meaning is generally distributed across many learned dimensions.

Misunderstanding 5: A text embedding is the same as a token embedding

Incorrect. Token embeddings represent individual tokens. Retrieval embeddings commonly represent larger text units.

Misunderstanding 6: Embedding and retrieval are the same thing

Incorrect. An embedding is a representation. Retrieval uses representations to find relevant information.

Misunderstanding 7: Semantic retrieval requires exact matching words

Incorrect. One major advantage is retrieving semantically related text with different vocabulary.

Misunderstanding 8: Cosine similarity of 0.90 means 90% same meaning

Incorrect. It is a mathematical similarity score, not a semantic probability.

Misunderstanding 9: High cosine similarity means the document is true

Incorrect. Similarity and factual correctness are separate problems.

Misunderstanding 10: Every dense retrieval system uses cosine similarity

Incorrect. Systems can also use dot product, Euclidean distance, or other measures.

Misunderstanding 11: Nearest-neighbor search always scans every vector

Incorrect. Large systems often use ANN indexes.

Misunderstanding 12: Approximate nearest-neighbor search is random

Incorrect. It uses specialized structures to efficiently search likely nearby regions.

Misunderstanding 13: A vector index and a vector database are exactly the same thing

Incorrect. The vector index is the search structure; a vector database can provide broader storage and management capabilities.

Misunderstanding 14: Dense retrieval understands meaning exactly like a human

Incorrect. It uses learned numerical representations that capture useful semantic relationships.

Misunderstanding 15: Semantic similarity guarantees relevance

Incorrect. Topical similarity can exist even when user intent differs.

Misunderstanding 16: Dense retrieval always beats BM25

Incorrect. Lexical retrieval can be superior for exact identifiers, names, codes, and specific terms.

Misunderstanding 17: Dense retrieval completely solves vocabulary mismatch

Incorrect. It reduces the problem but can still fail on rare or specialized language.

Misunderstanding 18: Retrieved semantic similarity guarantees useful evidence

Incorrect. Authority, freshness, factual quality, and evidence quality are separate concerns.

48. How the Concepts Connect

Chapter 8 gave us lexical retrieval:

Documents
↓
Terms
↓
Inverted Index

At query time:

Query
↓
Query Terms
↓
Term Matching
↓
BM25
↓
Lexical Results

Chapter 9 gives us dense retrieval:

Documents / Passages
↓
Embedding Model
↓
Dense Vectors
↓
Vector Index

At query time:

User Query
↓
Embedding Model
↓
Query Vector
↓
Vector Similarity Search
↓
Nearest Neighbors
↓
Dense Retrieval Results

The comparison is:

LEXICAL RETRIEVAL

Query Terms
↕
Document Terms
↓
Lexical Similarity

versus:

DENSE RETRIEVAL

Query Embedding
↕
Document Embedding
↓
Vector Similarity

The main strength of lexical retrieval:

Exact lexical identity

The main strength of dense retrieval:

Semantic similarity despite different wording

And now the natural next step is:

Lexical Retrieval
+
Dense Retrieval
↓
Hybrid Retrieval

Mental Model

Imagine a library with millions of books.

In Chapter 8, the librarian built a giant dictionary.

If you searched:

dentist Toronto

the system found books containing:

dentist
Toronto

That was lexical retrieval.

Now imagine a different librarian.

Instead of recording only words, this librarian reads each passage and creates a numerical meaning fingerprint.

For example:

Affordable one-to-one mathematics tutoring
↓
[0.18, -0.42, 0.71, ...]

Another passage:

Cheap private math lessons
↓
[0.21, -0.39, 0.68, ...]

The words differ.

But their numerical representations are close.

Now the user asks:

low-cost private math teacher

The question is also converted into a vector:

Query
↓
[0.20, -0.41, 0.70, ...]

The system asks:

Which stored vectors are closest to this query vector?

That is:

Nearest-Neighbor Search

If millions of vectors exist, comparing every vector may be expensive.

So the library builds a special structure:

Vector Index

and may use:

Approximate Nearest Neighbor Search

to locate likely nearby vectors quickly.

The result is:

Query Meaning
↓
Query Embedding
↓
Vector Space
↓
Nearest Document Embeddings
↓
Semantic Candidates

But this librarian also has weaknesses.

If the user asks:

HR-204

the exact code may matter more than its broad meaning.

Or:

Model 16

may be semantically almost identical to:

Model 15

even though they are different products.

So the first librarian is excellent at:

exact words

and the second librarian is excellent at:

related meanings

This gives us the central mental model:

Lexical Retrieval
→ What documents use these words?

Dense Retrieval
→ What documents have similar learned representations?

The simplest reliable definition is:

Dense retrieval converts queries and documents into learned numerical embeddings, searches for nearby vectors in an embedding space, and uses vector similarity to retrieve semantically related information even when the exact words differ.

Chapter 9 Recap

Dense Retrieval

Retrieval using dense numerical representations of queries and documents.

Text
↓
Dense Embedding
↓
Vector Similarity Search
↓
Relevant Candidates

Dense Vector

A numerical vector in which most dimensions contain non-zero values.

[0.18, -0.42, 0.71, 0.09, ...]

Sparse vs. Dense

Sparse Vector
→ explicit vocabulary dimensions
→ mostly zeros

Dense Vector
→ learned latent dimensions
→ mostly non-zero values

Semantic Retrieval

Retrieval that aims to match meaning rather than relying only on exact lexical overlap.

Different Words
+
Similar Meaning
↓
Potential Match

Text Embedding

A vector representation of a larger text unit such as:

query
sentence
passage
document

Token Embedding vs. Text Embedding

Token Embedding
→ representation of one token

Text Embedding
→ representation of a larger text unit

Embedding Model

A model that converts text into a vector representation useful for comparison.

Text
↓
Embedding Model
↓
Vector

Embedding Space

The multidimensional numerical space in which embeddings exist.

Semantically related texts can occupy nearby regions.

Document Embedding

The vector representation of a document or passage.

Query Embedding

The vector representation of a user's query.

Vector Similarity

A mathematical comparison between two vectors.

Query Vector
+
Document Vector
↓
Similarity Score

Cosine Similarity

A similarity function based primarily on the direction of two vectors.

Similar Direction
→ Higher Cosine Similarity

Dot Product

Another numerical method for comparing vector compatibility.

Euclidean Distance

A measure of distance between vectors.

Smaller Distance
→ Closer Vectors

Nearest-Neighbor Search

Finding document vectors closest to a query vector.

Query Vector
↓
Search Vector Space
↓
Nearest Document Vectors

Top-K Retrieval

Returning the K highest-scoring or nearest results.

K = 10
→ return top 10 candidates

Vector Index

A structure designed to make similarity search across many vectors efficient.

Document Embeddings
↓
Vector Index

Approximate Nearest Neighbor — ANN

A method for finding likely nearest vectors efficiently without exhaustively comparing every stored vector.

Large Vector Collection
↓
ANN Search
↓
Likely Nearest Neighbors

Bi-Encoder

An architecture that independently encodes:

Query
↓
Query Vector

and:

Document
↓
Document Vector

so their representations can be compared efficiently.

Precomputed Document Embeddings

Document vectors can be generated before queries arrive.

Documents
↓
Embed Once
↓
Store in Vector Index

At query time:

Query
↓
Embed Query
↓
Search Existing Vectors

Vocabulary Mismatch

Dense retrieval can help when:

query words
≠
document words

but:

query meaning
≈
document meaning

Paraphrase Retrieval

Dense retrieval can connect expressions such as:

cancel subscription
≈
terminate membership

even with limited word overlap.

Semantic Similarity vs. Relevance

Semantic Similarity
≠
Perfect Relevance

A document may discuss the same topic while failing the user's exact intent.

Semantic Similarity vs. Truth

High Vector Similarity
≠
Factual Accuracy

Retrieval similarity does not verify claims.

Dense Retrieval Strengths

Dense retrieval is especially useful for:

synonyms
paraphrases
conceptual relationships
different wording
vocabulary mismatch
natural-language questions

Dense Retrieval Weaknesses

It can struggle with:

exact identifiers
codes
numbers
rare entities
subtle distinctions
negation
precise lexical constraints

Vector Database

A broader system for storing and searching vectors and associated information.

It may include:

vectors
metadata
filters
vector indexes
update operations
search APIs

Passage-Level Dense Retrieval

Long documents can be divided into smaller passages, each with its own embedding.

Document
↓
Passages
↓
Passage Embeddings
↓
More Focused Retrieval

The complete Chapter 9 preparation pipeline is:

Corpus
↓
Documents / Passages
↓
Embedding Model
↓
Dense Embeddings
↓
Vector Index

And the query-time pipeline is:

User Query
↓
Embedding Model
↓
Query Embedding
↓
Vector Index
↓
Nearest-Neighbor / ANN Search
↓
Similarity Scores
↓
Top-K Candidates
↓
Dense Retrieval Results

We now have two major retrieval approaches.

From Chapter 8:

Lexical Retrieval
↓
Words
↓
Inverted Index
↓
BM25

From Chapter 9:

Dense Retrieval
↓
Embeddings
↓
Vector Index
↓
Vector Similarity

Each solves problems the other can struggle with.

So the next question is:

What happens if we use both lexical and semantic retrieval together?

That leads directly into Chapter 10:

Hybrid Retrieval.

The Single Most Important Idea from Chapter 9

Dense retrieval represents queries and documents as learned numerical embeddings and retrieves documents whose vectors are close to the query vector. This allows the system to find semantically related information even when the query and document use different words, but semantic similarity does not guarantee exact relevance, factual correctness, or preservation of precise lexical details.