Skip to content

Mo Farahani

AI Visibility Guide

03 Part III — Retrieval Augmented Generation (RAG) Chapter 03.01

Chapter 13- Why LLMs Need Retrieval

By Mo Farahani 24 min read

Chapter Goal

Up to this point, we have studied two major systems separately.

First, we learned how an LLM works:

Input
↓
Tokens
↓
Embeddings
↓
Transformer
↓
Next-Token Generation
↓
Answer

Then we learned how information retrieval works:

Query
↓
Retriever
↓
Candidate Documents
↓
Ranking
↓
Reranking
↓
Best Evidence Candidates

Now we connect them.

The central question of Part III is:

Why would an LLM need retrieval if it already learned enormous amounts of information during training?

The answer is that an LLM's internal knowledge has important limitations.

A model may:

know something
not know something
know something outdated
know only part of something
have no access to private information

So instead of relying only on information stored implicitly in model parameters, an AI system can retrieve external information at inference time.

The basic idea is:

User Question
↓
LLM Needs Information
↓
External Retrieval
↓
Relevant Information
↓
LLM Context
↓
Generated Answer

This connection between:

Language Model
+
External Information

is the foundation of modern RAG and many AI search systems.

By the end of this chapter, you should understand:

  • what the limitations of model knowledge are;
  • what a training cutoff means;
  • why knowledge can become stale;
  • why a model can have missing knowledge;
  • why private knowledge is fundamentally different from public training data;
  • what long-tail knowledge is;
  • what external knowledge means;
  • how retrieval gives an LLM access to external information;
  • the difference between web search and knowledge-base retrieval;
  • what search augmentation means;
  • what Search-Augmented Generation means;
  • what Retrieval-Augmented Generation means;
  • and why retrieval changes what an LLM can answer.

The central pipeline of this chapter is:

LLM Internal Knowledge
↓
Knowledge Limitations
↓
Need for External Information
↓
Retrieval
↓
Relevant External Evidence
↓
LLM Context
↓
Generated Answer

13.1 Limitations of Model Knowledge

1. What Does It Mean for an LLM to “Know” Something?

An LLM does not normally contain a traditional database of facts such as:

France → capital → Paris
OpenAI → founded → 2015

Instead, during training, the model learns statistical patterns in its parameters.

Training Data
↓
Training
↓
Parameter Updates
↓
Learned Statistical Associations

So information becomes encoded indirectly inside the model.

This is often called:

parametric knowledge

because the information is represented through model parameters.

Training Data
↓
Model Parameters
↓
Internal Knowledge

But this knowledge is neither complete nor continuously updated.

That creates the need for retrieval.

2. Training Cutoff

An LLM is trained using data collected during some training period.

After that training process ends, its parameters do not automatically absorb every new event on the internet.

Suppose training finishes at time:

T1

Then an event happens later:

T2
Training Data
↓
Training Completed at T1
↓
Model Parameters Fixed

Later Event at T2
↓
Not Automatically Added to Parameters

This creates what is commonly called a training cutoff.

Information that appears after the model's training data was collected cannot automatically be assumed to exist in its internal parametric knowledge.

3. Training Cutoff Does Not Mean the AI System Cannot Access New Information

This distinction is extremely important.

A model's internal parameters may not contain new information.

But the broader AI system may still have access to:

web search
retrieval systems
databases
APIs
documents
tools
Model Internal Knowledge
≠
Everything the AI System Can Access

For example:

LLM Parameters
→ may contain older knowledge

Web Search
→ can retrieve newer information

Then:

Old Internal Knowledge
+
Current External Information
↓
Current Answer

This distinction becomes fundamental in AI search.

4. Knowledge Freshness

Even if a model learned a fact during training, that fact may later change.

This creates a knowledge freshness problem.

Consider:

CEO of Company X

At training time:

CEO = Person A

Later:

CEO = Person B

The model's internal association may still reflect:

Person A

even though the real-world answer has changed.

Other examples include:

prices
laws
regulations
product specifications
company leadership
sports results
exchange rates
software versions
opening hours
medical guidance
Knowledge Was Correct
at Training Time
≠
Knowledge Is Correct
Now

5. Training Cutoff vs. Knowledge Freshness

These concepts are related but slightly different.

Training Cutoff

The information may have appeared only after training.

Event
happens after training

The model may never have learned it.

Knowledge Freshness

The model may have learned information that was once correct but later became outdated.

Old Fact
↓
Learned During Training
↓
World Changes
↓
Stored Association Becomes Stale
Training Cutoff
→ new information may be absent

Freshness Problem
→ previously learned information may now be outdated

6. Missing Knowledge

A model can also lack information that existed before its training cutoff.

This is missing knowledge.

Why?

Because no training corpus contains every piece of information in existence.

A fact might be:

not present in the training data
rarely mentioned
poorly represented
hidden inside inaccessible sources
too specialized
too local

For example, imagine a small business:

Persian Dental Clinic X

Its exact:

opening hours
services
staff
pricing
language availability

may exist online.

But those details may never have been sufficiently represented in the model's training data.

Information Existed
≠
Model Necessarily Learned It

7. Learning Something Is Not the Same as Storing It Perfectly

Even if information appears in training data, this does not guarantee perfect recall later.

Training is not equivalent to:

Copy every document
↓
Store exact text
↓
Retrieve exact fact later

Instead:

Training Data
↓
Statistical Learning
↓
Compressed Distributed Representations

The model may learn some associations strongly and others weakly.

Seen During Training
≠
Guaranteed Reliable Knowledge

This is another reason external evidence can be useful.

8. Private Knowledge

A major limitation is information the model could never have learned from public training data.

Examples:

internal company documents
private customer records
personal files
private emails
internal policies
private financial data
proprietary databases
unpublished research

Suppose a company employee asks:

What is our internal refund policy for Enterprise customers?

The answer exists inside:

Company Knowledge Base

but not publicly on the web.

A general-purpose LLM cannot simply know that private information.

Instead:

Private Knowledge Base
↓
Retrieval
↓
Relevant Internal Document
↓
LLM Context
↓
Answer

This is one of the most important practical reasons organizations use RAG.

9. Public Knowledge vs. Private Knowledge

Public Knowledge
→ potentially available in public training data or web search

Private Knowledge
→ accessible only through authorized internal sources

For example:

"What is Microsoft's CEO?"
→ public knowledge

versus:

"What was discussed in our private strategy meeting?"
→ private knowledge

A model cannot derive private information merely from general training.

It needs access to the relevant external source.

10. Long-Tail Knowledge

Another major limitation is long-tail knowledge.

Some information is extremely common.

Examples:

capital of France
largest planet
basic Python syntax

These concepts appear repeatedly across many sources.

Other information appears rarely.

Examples:

a small local business
a niche legal ruling
a rare technical bug
a specific local service
an obscure academic detail
a regional organization
a specialized product model

This is the long tail.

Very Common Knowledge
████████████████████

Common Knowledge
██████████

Less Common Knowledge
████

Rare Knowledge
██

Extremely Rare Knowledge
█

The further we move into the long tail:

less training exposure
↓
weaker learned associations
↓
greater uncertainty

11. Why Long-Tail Knowledge Matters for AI Visibility

This becomes especially important when discussing:

brands
local businesses
professionals
niche products
specialized services

A model may have strong internal associations for:

Apple
Amazon
Microsoft

but weak or nonexistent internal associations for:

a local Persian dentist
a small immigration law firm
a regional restaurant
a niche tutoring platform

For these entities, external retrieval may be much more important.

Weak Parametric Knowledge
+
Strong External Web Evidence
↓
Potential AI Answer Presence

This is one of the bridges between retrieval and AI Visibility.

12. Five Major Limitations of Internal Model Knowledge

1. Training Cutoff
   → newer information may not exist internally

2. Knowledge Freshness
   → learned information may become outdated

3. Missing Knowledge
   → some information may never have been learned

4. Private Knowledge
   → some information exists outside public training data

5. Long-Tail Knowledge
   → rare information may be weakly represented

These problems all point toward the same solution:

External Information

13.2 External Information

13. What Is External Knowledge?

External knowledge is information obtained from sources outside the model's internal parameters.

Examples include:

web pages
search results
databases
company knowledge bases
PDF documents
product catalogs
APIs
private files
news sources

So we can distinguish:

Parametric Knowledge
→ encoded inside model parameters

External Knowledge
→ supplied to the model at inference time

This is a fundamental distinction.

14. Parametric Knowledge vs. External Knowledge

Suppose you ask:

What is BM25?

The model may answer from internal knowledge:

Question
↓
LLM Parameters
↓
Answer

But suppose you ask:

What did Company X announce this morning?

A better pipeline may be:

Question
↓
Search External Sources
↓
Retrieve Announcement
↓
Provide It to LLM
↓
Answer

So external knowledge allows the system to answer questions that internal parametric knowledge alone may not reliably support.

15. Retrieval as a Bridge

Retrieval creates the bridge between:

External Information

and:

LLM Generation
External Knowledge Source
↓
Retriever
↓
Relevant Information
↓
LLM Context

The model does not need to permanently learn the retrieved information.

Instead, the information is temporarily available inside the current context.

Training
→ changes model parameters

Retrieval
→ changes the information available in context

Retrieval normally does not retrain the model.

16. Retrieval at Inference Time

Recall from Chapter 4:

Training
≠
Inference

Retrieval usually happens during inference.

User asks question
↓
System retrieves document
↓
Document enters prompt/context
↓
LLM generates answer

The model parameters remain unchanged.

External Knowledge Access
≠
Learning New Parameters

This is one of the most important technical distinctions in RAG.

18. Web Search Is Not the Same as the LLM

When an AI system performs web search, two systems are involved.

Search System
→ finds external information

LLM
→ interprets and generates language
Question
↓
Search
↓
Evidence
↓
LLM
↓
Answer

The LLM is not simply generating an answer from its parameters.

It is working with externally retrieved evidence.

19. Knowledge Base

Another major external source is a knowledge base.

A knowledge base is an organized collection of information that a retrieval system can search.

Examples:

company documentation
support articles
internal policies
product manuals
research documents
legal documents
customer knowledge systems
Company Documents
↓
Knowledge Base
↓
Retriever
↓
Relevant Passages
↓
LLM
↓
Answer

Unlike web search, the source collection may be:

private
controlled
domain-specific
curated

20. Web Search vs. Knowledge Base Retrieval

Web Search

Searches:

public web information

Often useful for:

fresh
broad
public
open-domain
information

Knowledge Base Retrieval

Searches:

defined document collection

Often useful for:

private
specialized
controlled
organization-specific
information
External Retrieval
├── Web Search
└── Knowledge Base Retrieval

Both can feed information into an LLM.

21. External Information Does Not Automatically Become Model Knowledge

Suppose the system retrieves:

Company X Revenue = €14M

and provides it to the LLM.

The model can use that information during the current inference.

But that does not necessarily mean:

Model Parameters
now permanently contain
Company X Revenue = €14M

Instead:

Retrieved Information
↓
Current Context
↓
Temporary Availability

When the context disappears, the base model's parameters remain unchanged.

This is why retrieval is fundamentally different from training.

22. Retrieval Can Provide Evidence, Not Just Facts

Retrieval is useful not only because it supplies unknown facts.

It can also provide:

source text
supporting passages
citations
context
definitions
current evidence
Question
↓
Retrieve Evidence
↓
Generate from Evidence

This will become important later when we study:

grounding
claims
citations
faithfulness

13.3 Search Augmentation

23. What Does Augmentation Mean?

Augmentation means adding an external capability or information source to improve the model's ability to answer.

A standalone LLM behaves roughly like:

Question
↓
Model Parameters
↓
Answer

An augmented system behaves more like:

Question
↓
External Information System
↓
Relevant Information
↓
LLM
↓
Answer

The LLM has gained access to something outside itself.

That is the core idea of augmentation.

24. Search Augmentation

Search augmentation means augmenting the LLM with information obtained through search.

LLM
+
Search

Pipeline:

User Query
↓
Search System
↓
Search Results
↓
Relevant Information
↓
LLM

The model can now use information discovered through search rather than relying only on internal knowledge.

25. Why Search Augmentation Matters

Consider:

Who won yesterday's match?

Without external retrieval:

LLM Internal Knowledge
↓
possibly unavailable or outdated

With search augmentation:

Question
↓
Search Current Sources
↓
Retrieve Result
↓
LLM Reads Result
↓
Answer

The system transforms the problem from:

Does the model already know this?

into:

Can the system find reliable information about this?

That is a major architectural change.

26. Search-Augmented Generation

When retrieved search information is used directly during answer generation, we can describe the process as:

Search-Augmented Generation.

User Query
↓
Search
↓
Search Results
↓
Relevant Evidence
↓
LLM Generation
↓
Answer

The generation is:

augmented by search results

rather than produced entirely from parametric knowledge.

27. Search-Augmented Generation vs. Ordinary Search

Traditional search often ends with:

Query
↓
Search Results
↓
Links Shown to User

Search-Augmented Generation continues further:

Query
↓
Search Results
↓
Relevant Information
↓
LLM
↓
Generated Answer

So the search results become input to generation.

This is one of the core patterns behind AI search systems.

28. Retrieval-Augmented Generation

The broader concept is:

Retrieval-Augmented Generation, commonly abbreviated:

RAG

At the simplest level:

User Query
↓
Retriever
↓
Relevant External Information
↓
LLM
↓
Generated Answer

Generation is augmented by information retrieved from an external source.

That source could be:

web search
private knowledge base
document collection
database
enterprise search system

29. Search-Augmented Generation vs. Retrieval-Augmented Generation

Search-Augmented Generation
→ generation augmented by search results

while:

Retrieval-Augmented Generation
→ generation augmented by retrieved information more generally

Web search can therefore be one form of retrieval.

Retrieval-Augmented Generation
├── Web Retrieval
├── Knowledge Base Retrieval
├── Document Retrieval
└── Other External Retrieval

Search-Augmented Generation is closely related but emphasizes search as the external information mechanism.

30. RAG Does Not Give the Model Permanent New Knowledge

RAG does not normally mean:

Retrieve document
↓
Train model
↓
Model permanently learns document

Instead:

Retrieve document
↓
Insert relevant information into context
↓
Generate answer
RAG
→ inference-time knowledge access

not

RAG
→ automatic model retraining

31. Internal Knowledge and Retrieved Knowledge Can Work Together

RAG does not require the model to ignore everything it already knows.

An answer may use:

Internal Language Knowledge
+
Retrieved External Evidence

For example:

Retrieved Evidence
→ provides current factual information

LLM Internal Capabilities
→ interpret
→ summarize
→ explain
→ structure
→ generate
External Evidence
+
Model Capabilities
↓
Answer

Retrieval supplies information.

The LLM supplies language processing and generation.

32. Retrieval Changes the Source of the Answer

Without retrieval:

Question
↓
Model Parameters
↓
Generated Answer

With retrieval:

Question
↓
External Retrieval
↓
Evidence
↓
LLM Context
↓
Generated Answer

This creates an important distinction between:

Parametric Answering

and:

Retrieval-Augmented Answering

The second system can use information that was never encoded in the model's parameters.

33. Retrieval Can Reduce Some Hallucination Risk

Suppose the model does not know the answer.

Without evidence:

Question
↓
Incomplete Internal Knowledge
↓
Generation
↓
Possible Unsupported Answer

With retrieval:

Question
↓
Retrieve Relevant Evidence
↓
Evidence Available
↓
Generate from Evidence

This can reduce some forms of unsupported generation.

But:

RAG
≠
No Hallucination

The system can still:

retrieve wrong information
misinterpret evidence
ignore evidence
combine sources incorrectly
generate unsupported claims

Retrieval improves the information environment.

It does not guarantee perfect factuality.

34. The Quality of RAG Depends on Retrieval

Suppose the correct answer exists in Document C.

But the retriever returns:

A
B
D
E
F

Then:

Correct Evidence
→ never reaches LLM

The generator cannot reliably use evidence it never sees.

Poor Retrieval
↓
Poor Context
↓
Constrained Generation Quality

This connects Part II directly to Part III.

Everything we learned about:

lexical retrieval
dense retrieval
hybrid retrieval
ranking
reranking
retrieval evaluation

now becomes part of RAG quality.

35. Retrieval Is Not Automatically Grounding

Suppose a system retrieves a correct document.

That alone does not mean the final answer is grounded.

There are still later steps:

Retrieved Document
↓
Relevant Passage
↓
Context
↓
Generation
↓
Claim

The model could still misrepresent the evidence.

Retrieval
≠
Grounding

Retrieval provides potential evidence.

Grounding concerns whether the answer is actually supported by that evidence.

We will study this distinction later.

36. Why RAG Is So Important for Private Systems

Imagine a company has:

50,000 internal documents

including:

policies
contracts
product documentation
support procedures
engineering documents

Training a new foundation model whenever those documents change would be impractical.

Instead:

Documents
↓
Retrieval System
↓
Relevant Internal Information
↓
Existing LLM
↓
Answer

When documents change:

Knowledge Base Updates

can be reflected without retraining the entire LLM.

This is one of RAG's major architectural advantages.

37. Why RAG Is Important for Fresh Information

The same principle applies to rapidly changing public information.

Instead of expecting model parameters to contain:

today's news
today's price
today's regulation
today's sports result

the system can retrieve them when needed.

Static Model
+
Dynamic Retrieval
↓
Access to Dynamic Information

This combination is extremely powerful.

38. Why RAG Is Important for Long-Tail Information

Recall long-tail knowledge.

Suppose the query concerns:

a small local law firm

The model may have almost no meaningful internal representation of the business.

But external retrieval can discover:

official website
directory profile
news mention
professional listing
review source

Then:

Weak Internal Knowledge
+
Strong Retrieved Evidence
↓
Useful Answer

This is one of the reasons web presence matters for AI Visibility.

39. What Retrieval Does Not Solve

Retrieval is powerful, but it does not solve everything.

A retrieval system may still face:

wrong query interpretation
poor document discovery
weak retrieval
bad ranking
outdated sources
conflicting sources
low-quality evidence
insufficient evidence

And the LLM may still:

misread evidence
ignore evidence
overgeneralize
generate unsupported claims
Retrieve Something
↓
Perfect Answer

is not a valid assumption.

RAG is a pipeline containing many possible failure points.

40. A Complete Example: Current Public Knowledge

User asks:

Who is the current CEO of Company X?

The model's internal knowledge may say:

Person A

But that association may be stale.

A search-augmented system can perform:

User Question
↓
Web Search
↓
Current Company Source
↓
Retrieve Relevant Passage
↓
LLM Context
↓
Answer:
Person B

The answer can now use current external information.

41. A Complete Example: Private Knowledge

Employee asks:

What is our refund policy for Enterprise customers?

The public web may not contain the answer.

The system searches:

Internal Knowledge Base

and finds:

Enterprise Refund Policy

Pipeline:

Employee Question
↓
Internal Retrieval
↓
Relevant Policy Passage
↓
LLM Context
↓
Generated Answer

Here the limitation was not freshness.

It was:

Private Knowledge

42. A Complete Example: Long-Tail Knowledge

User asks:

Does a small local Persian bakery in Vancouver
offer custom wedding cakes?

The LLM may have no reliable internal knowledge about that business.

But retrieval can find:

business website
menu page
local directory
social profile

Then:

Query
↓
External Search
↓
Relevant Business Information
↓
LLM
↓
Answer

The model does not need to have memorized the business.

The system needs to be able to:

find
retrieve
select
use

the relevant external evidence.

43. The Core Architectural Shift

A standalone language model asks:

What can I generate
from my internal parameters
and current prompt?

A retrieval-augmented system can instead ask:

What information do I need?

Where can I retrieve it?

What evidence should enter the context?

How should I answer using that evidence?

This is the architectural shift from:

Model-Centric Knowledge

toward:

Model + External Knowledge System

45. Common Misunderstandings

Misunderstanding 1: If an LLM has a training cutoff, it cannot answer questions about newer information

Incorrect. The model's parameters may not contain the information, but the broader system may retrieve current external information.

Misunderstanding 2: Everything published before the training cutoff must be known by the model

Incorrect. Information can be missing, rare, poorly represented, inaccessible, or weakly learned.

Misunderstanding 3: If a model saw information during training, it can always recall it accurately

Incorrect. Training does not create a perfect document database.

Misunderstanding 4: Private company knowledge can be expected to exist inside a general-purpose model

Incorrect. Private information generally requires authorized external access.

Misunderstanding 5: Retrieval trains the model on new information

Incorrect. Retrieval normally provides information at inference time without changing model parameters.

Misunderstanding 6: Web search and knowledge-base retrieval are identical

Incorrect. Both are retrieval, but they search different information environments.

Misunderstanding 7: Search-Augmented Generation and ordinary search are the same

Incorrect. Ordinary search can stop at ranked results. Search-Augmented Generation feeds retrieved search information into a generator.

Misunderstanding 8: RAG means web search

Incorrect. Web search can be one retrieval source, but RAG can also retrieve from private knowledge bases, document collections, databases, and other external systems.

Misunderstanding 9: RAG permanently adds retrieved information to the model

Incorrect. The information usually exists temporarily in the inference context.

Misunderstanding 10: RAG guarantees factual answers

Incorrect. Retrieval, ranking, source quality, context construction, and generation can all fail.

46. How the Concepts Connect

Part I explained:

How the LLM represents,
learns,
and generates language

Part II explained:

How a system finds relevant external information

Chapter 13 connects them:

LLM Knowledge Limitations
↓
Need for External Information
↓
Retrieval
↓
External Knowledge
↓
LLM Generation

More specifically:

Training Cutoff
Knowledge Freshness
Missing Knowledge
Private Knowledge
Long-Tail Knowledge
↓
Internal Knowledge Is Not Enough
↓
External Information
├── Web Search
└── Knowledge Base
↓
Retrieval
↓
Search / Retrieval Augmentation
↓
Generated Answer

We now understand why retrieval is necessary.

The next chapter will explain exactly how Retrieval-Augmented Generation works as an architecture.

That leads directly into:

Chapter 14 — RAG Fundamentals

where we will study:

RAG
Retriever
Generator
Retrieved Context
Context Injection
External Context
Evidence Context
Grounding
Grounded Generation
Grounded Answer

Mental Model

Imagine an expert sitting in a room.

The expert has spent years studying many books.

That knowledge is inside their head.

This is similar to:

LLM Parameters

You ask:

What is the capital of France?

The expert answers immediately:

Paris

No external search is needed.

Now you ask:

What did Company X announce this morning?

The expert may say:

I don't know.
That happened after I finished studying.

This is like:

Training Cutoff
+
Freshness Problem

So you give the expert access to today's newspaper.

Question
↓
Search Newspaper
↓
Read Relevant Article
↓
Answer

That is external retrieval.

Now you ask:

What is our company's private refund policy?

The newspaper cannot help.

But you give the expert access to:

Internal Company Documents

They search those documents and answer.

That is private knowledge retrieval.

Finally, you ask about:

an obscure small local company

the expert has never heard of.

Instead of guessing, they search for reliable information.

That solves part of the:

Long-Tail Knowledge Problem

So the expert now works like:

Existing Knowledge
+
Ability to Retrieve Missing Information

rather than:

Existing Knowledge Only

That is exactly the architectural idea behind retrieval-augmented systems.

The simplest mental model is:

LLM
→ knows what training encoded

Retriever
→ finds what the LLM needs now

Context
→ gives retrieved information to the LLM

Generator
→ uses that information to produce the answer

And the simplest reliable definition is:

LLMs need retrieval because their internal parametric knowledge can be outdated, incomplete, private-data-blind, or weak for rare long-tail information. Retrieval allows the system to obtain relevant external information at inference time and provide it to the LLM as context, enabling generation to use knowledge that does not need to be permanently stored inside the model.

Chapter 13 Recap

Parametric Knowledge

Information represented indirectly inside model parameters as a result of training.

Training Data
↓
Training
↓
Model Parameters
↓
Internal Knowledge
Training Cutoff

Information appearing after the model's training period may not exist in its internal parameters.

Training Ends
↓
New Event Happens
↓
Not Automatically Learned
Knowledge Freshness

Previously learned knowledge can become outdated when the real world changes.

Correct at Training Time
≠
Necessarily Correct Now
Missing Knowledge

Some information may never have been sufficiently represented during training.

Information Exists
≠
Model Necessarily Knows It
Private Knowledge

Information available only through authorized private sources.

Examples:

internal documents
emails
private databases
company policies
customer information
Long-Tail Knowledge

Rare or specialized information with relatively little representation.

Rare Entity
↓
Limited Training Exposure
↓
Weak Internal Knowledge
External Knowledge

Information obtained outside the model parameters.

Examples:

web
knowledge bases
documents
databases
APIs
Retrieval

The mechanism that connects external information to the AI system.

External Source
↓
Retriever
↓
Relevant Information
Web Search

Retrieval from public web information.

Query
↓
Web Search
↓
Current Public Information
Knowledge Base

A defined collection of searchable information.

Documents
↓
Knowledge Base
↓
Retriever

It may be private or domain-specific.

Search Augmentation

Adding search capability to the LLM system.

LLM
+
Search
Search-Augmented Generation
Query
↓
Search
↓
Search Results
↓
LLM
↓
Generated Answer

Generation is augmented by information discovered through search.

Retrieval-Augmented Generation — RAG
Query
↓
Retriever
↓
Relevant External Information
↓
LLM Context
↓
Generation
↓
Answer

The retrieved information can come from multiple external sources.

Retrieval vs. Training
Training
→ changes model parameters

Retrieval
→ changes available inference context
RAG and Freshness
Static Model Parameters
+
Dynamic External Retrieval
↓
Access to Fresh Information
RAG and Private Knowledge
Private Documents
↓
Retriever
↓
LLM Context
↓
Answer

The model does not need the private information to have existed in its training data.

RAG and Long-Tail Knowledge
Weak Internal Knowledge
+
External Evidence
↓
Useful Answer

The complete Chapter 13 pipeline is:

LLM Internal Knowledge
├── Training Cutoff
├── Freshness Limits
├── Missing Knowledge
├── Private Knowledge Gap
└── Long-Tail Weakness
↓
Need External Information
↓
External Sources
├── Web
└── Knowledge Base
↓
Retrieval
↓
Relevant External Information
↓
LLM Context
↓
Search-Augmented / Retrieval-Augmented Generation
↓
Answer

We have now answered:

Why does an LLM need retrieval?

The next chapter answers:

Exactly how does RAG connect
the retriever,
the retrieved context,
and the generator?

That is:

Chapter 14 — RAG Fundamentals

The Single Most Important Idea from Chapter 13

An LLM's internal parameters are not a complete, continuously updated database of the world. Knowledge can be outdated, missing, private, or too rare to be reliably represented. Retrieval solves this architectural limitation by finding relevant external information at inference time and placing it into the model's context, allowing generation to use knowledge that the model did not need to memorize during training.

← Back to AI Visibility Guide