Chapter 3- Transformer Architecture

Chapter Goal

In Chapters 1 and 2, we built two important foundations.

First:

Text
↓
Tokenization
↓
Tokens
↓
Token IDs

Then:

Text
↓
Numerical Representation
↓
Embeddings
↓
Vectors
↓
Semantic Relationships

But one major problem remains.

A word or token does not always mean the same thing in every sentence.

Consider:

"I sat on the bank of the river."

"I deposited money at the bank."

The word bank appears in both sentences.

But its meaning changes because of the surrounding words.

So a modern language model needs more than numerical representations of individual tokens.

It needs a mechanism that can examine the relationships between tokens and create representations that depend on the current context.

This chapter explains the architecture that makes this possible: the Transformer.

By the end of this chapter, you should understand:

• what a Large Language Model is;
• what model parameters are;
• what a neural network means at a conceptual level;
• what a Transformer is;
• what Transformer layers do;
• what Attention and Self-Attention mean;
• what an Attention Score represents;
• the conceptual roles of Query, Key, and Value;
• why Query in Attention is different from a search query;
• what a Contextual Representation is;
• what a Contextual Embedding is;
• how token relationships influence meaning;
• why Transformers can model long-range dependencies;
• and how the concepts from Chapters 1–3 connect.

The core dependency of this chapter is:

Tokens
↓
Embeddings
↓
Attention
↓
Transformer
↓
Contextual Representation

1. The Problem: Meaning Depends on Context

Consider the sentence:

"The mouse is next to the keyboard."

The word mouse probably refers to a computer device.

Now consider:

"The mouse ran into the kitchen."

Here, mouse probably refers to an animal.

The token itself has not changed.

But its meaning inside the sentence has.

Why?

Because of the surrounding context.

In the first sentence, words such as keyboard provide evidence for the computer-device interpretation.

In the second, words such as ran and kitchen support the animal interpretation.

So understanding language requires more than asking:

What token is this?

or even:

What does this token usually mean?

The model needs to ask something closer to:

What does this token mean here, given the other tokens around it?

That is the problem Transformer architecture helps solve.

2. From Static Representation to Contextual Representation

Let us simplify the problem.

Suppose we begin with the word:

bank

We need some numerical representation for it.

But now compare:

river bank

with:

bank account

If we used exactly the same final representation for bank in both situations, we would lose an important distinction.

The surrounding context should influence the representation.

Conceptually, we want:

Initial representation of "bank"
+
information from surrounding tokens
↓
context-sensitive representation of "bank"

In river bank, the representation should reflect the river-related meaning.

In bank account, it should reflect the financial meaning.

This leads to one of the central ideas of modern language models:

A token's useful representation can be updated according to the other tokens in its context.

The mechanism that allows tokens to influence one another is called Attention.

But before we study Attention, we need to understand the larger system in which it operates.

3. Language Models and Large Language Models

What Is a Language Model?

A language model is a model that learns patterns in language.

At a high level, it learns statistical relationships such as:

• which words or tokens tend to occur together;
• which sequences are plausible;
• how concepts are related;
• how linguistic structures behave;
• and how context changes likely continuations.

For example, consider:

"The capital of France is ___"

A capable language model may strongly favor:

Paris

over:

banana

Or consider:

"She opened the umbrella because it started to ___"

A model may assign high probability to:

rain

because it has learned patterns connecting:

umbrella
weather
rain

This does not yet explain how the model learns those patterns. That belongs primarily to Chapter 4.

For now, the important point is:

A language model is a computational model designed to represent and predict patterns in language.

What Is a Large Language Model?

A Large Language Model, usually abbreviated as LLM, is a language model with a large number of learned parameters, typically trained on very large amounts of data and capable of modeling complex language patterns.

Examples of tasks that modern LLMs can perform include:

• answering questions;
• summarizing text;
• rewriting;
• translation;
• classification;
• information extraction;
• reasoning over provided context;
• and generating structured text.

But the word large deserves clarification.

It does not simply mean a large file or long answers.

It primarily refers to scale, especially the number of learned parameters and the overall scale of the model and its training.

4. Parameters and Model Size

What Is a Parameter?

A parameter is a numerical value inside a model that is learned during training.

A real modern model may contain an enormous number of such values.

Conceptually:

Model
↓
many learned numerical parameters

You can imagine a simplified model containing values such as:

0.183
-0.427
1.204
0.058
...

These individual numbers are not normally interpreted like:

Parameter 1 = knowledge about lawyers
Parameter 2 = knowledge about restaurants
Parameter 3 = grammar

That would be misleading.

Useful capabilities are generally distributed across many parameters and computations.

The important idea is:

Parameters are learned numerical values that influence how the model transforms input into output.

For example, when the model processes:

The capital of France is ...

many parameters participate in the numerical computations that eventually make some possible next tokens more likely than others.

A simplified result might look like:

Paris   → high probability
London  → lower probability
Berlin  → lower probability
Rome    → lower probability
...

There is not normally one individual parameter that means:

France → Paris

Instead, many parameters work together across the model's computations.

During training, these values are adjusted. During ordinary inference, the model uses the learned values to process new input.

We will study training and parameter updates properly in Chapter 4.

Model Size

You will often hear expressions such as:

7B model
70B model

where B means billions of parameters.

Conceptually:

7B
≈
7 billion parameters

This is one way people describe model size.

However:

More parameters
≠
automatically better model in every situation

Model quality can also depend on factors such as:

• architecture;
• training data;
• training process;
• post-training;
• task;
• and inference system.

For this chapter, remember:

Parameter count is one important measure of model scale, not a universal measure of intelligence or quality.

5. Neural Networks and Architecture

Neural Network — Conceptual Understanding

Modern LLMs are based on neural networks.

We do not need to study neural-network mathematics in this course.

At the conceptual level, a neural network is a computational system that transforms numerical inputs through layers of mathematical operations using learned parameters.

Very roughly:

Numerical Input
↓
Layer
↓
Layer
↓
Layer
↓
Numerical Output

Each layer transforms the representation it receives.

For language:

Tokens
↓
Numerical Representations
↓
Many Learned Transformations
↓
Output Representation

The model does not operate on human-readable concepts directly.

It operates on numerical representations.

That is why Chapter 2 was necessary before this chapter.

What Is an Architecture?

In machine learning, an architecture describes the general structural design of a model.

It answers questions such as:

• What kinds of computational components exist?
• How are they arranged?
• How does information move through them?
• How do different representations interact?

An analogy is the architecture of a building.

Two buildings may contain different furniture and different people while sharing a similar structural design.

Likewise, different AI models can be based on the same general architecture while having different parameters, training data, sizes, behaviors, and capabilities.

The Transformer is one such architecture.

6. The Transformer

What Is a Transformer?

A Transformer is a neural-network architecture designed to process sequences of information while allowing elements in the sequence to interact through a mechanism called Attention.

For language, the sequence consists of tokens.

Conceptually:

Tokens
↓
Numerical Representations
↓
Transformer
↓
Contextualized Representations

The Transformer does not simply process every token as an isolated object.

Instead, it can model relationships among tokens in the context.

Consider:

"The lawyer who represented the company won the case."

Understanding this sentence requires relationships among concepts such as:

lawyer
represented
company
won
case

The Transformer architecture provides mechanisms for these token representations to influence one another.

The most important mechanism for understanding this interaction is Attention.

Why Transformers Were Important

Imagine trying to understand:

"The restaurant that my friend recommended last week was closed."

The model needs to preserve relationships among tokens even when related information is separated by other words.

For example:

restaurant
...
was closed

belong together.

Similarly:

friend
...
recommended

belong together.

Language contains many such dependencies.

A useful architecture therefore needs to model relationships across a sequence rather than treating each word independently.

Transformers became especially important because Attention provides a powerful way to model these relationships.

At a simplified level:

Token
↓
looks at relevant information from other tokens
↓
updates its representation

This process happens mathematically, not through human-style conscious reading.

7. Transformer Layers

A Transformer is not usually just one single Attention operation.

It contains multiple Transformer layers.

Conceptually:

Input Representations
↓
Transformer Layer 1
↓
Transformer Layer 2
↓
Transformer Layer 3
↓
...
↓
Final Representations

Each layer performs transformations on the representations it receives.

As representations pass through successive layers, they can become increasingly informed by the surrounding context.

Imagine:

"The mouse is next to the keyboard."

The initial representation of mouse enters the Transformer.

Through interactions with other token representations—especially useful contextual signals such as keyboard—the representation can be transformed.

Later layers receive the already-updated representation and continue processing it.

So the model does not simply have:

mouse = fixed meaning

Instead, its internal representation can become context-dependent.

Layers Do Not Have Simple Human Jobs

It is tempting to imagine:

Layer 1 = grammar
Layer 2 = entities
Layer 3 = facts
Layer 4 = reasoning

This is generally too simplistic.

Different patterns and capabilities can be distributed across many components and layers.

Therefore, we should not treat Transformer layers like manually labeled departments.

A safer mental model is:

Layer 1
↓
transforms representations

Layer 2
↓
transforms the already updated representations

Layer 3
↓
transforms them again

...

Through repeated transformations, the model builds increasingly contextual representations.

8. Attention and Self-Attention

Attention: The Central Idea

Attention is a mechanism that allows the model to determine how strongly different pieces of information should influence one another during a particular computation.

Another useful way to think about it is:

When a Transformer layer updates the representation of a token, Attention helps determine which other tokens should influence that update and by how much.

Consider:

"The mouse is next to the keyboard."

When the model updates the representation of mouse, Attention can evaluate relationships such as:

mouse ↔ The
mouse ↔ is
mouse ↔ next
mouse ↔ keyboard

Some relationships can be more useful than others.

For example, keyboard provides strong contextual information about what mouse means in this sentence.

Now compare:

"The mouse ate the cheese."

Here, relationships involving ate and cheese provide a very different context.

So the Attention pattern can change:

mouse + keyboard context
↓
computer-related contextual representation

mouse + ate + cheese context
↓
animal-related contextual representation

This gives us a useful distinction:

The Transformer layer transforms representations. Attention is one of the main mechanisms inside that layer that determines which other token representations should influence that transformation and by how much.

Attention Is Not Human Attention

The word Attention can be misleading.

When humans say:

Pay attention to this.

we usually mean conscious focus.

Transformer Attention is not consciousness.

It is a mathematical operation.

The model computes relationships between numerical representations and uses those relationships to determine how information should be combined.

Human attention
≠
Transformer Attention

A safer definition is:

Attention is a mathematical mechanism for weighting relationships between representations when constructing updated representations.

Self-Attention

When Attention operates among elements of the same input sequence, it is called Self-Attention.

Suppose the input is:

"The cat chased the mouse."

The tokens in that sequence can interact with other tokens in the same sequence.

For example, while updating the representation of chased, information from cat and mouse may be particularly useful.

For mouse, information from chased and cat may help establish its role in the sentence.

This is Self-Attention because the sequence is attending to information within itself.

Self-Attention Is Not Simply Looking at Nearby Words

Consider:

"The lawyer who moved to Toronto several years ago opened her own firm."

The relationship between lawyer and her extends across several intervening tokens.

Likewise:

lawyer
...
firm

may form a meaningful relationship.

Self-Attention can model relationships across separated positions in the context.

Token A ─────────────→ Token G

not only:

Token A → Token B

This ability becomes important for understanding long-range dependencies later in the chapter.

9. Attention Scores

Not every token should necessarily contribute equally when another token's representation is being updated.

Suppose we have:

"The doctor examined the patient because she was concerned."

When processing she, a simplified illustration might be:

doctor     → strong contribution
examined   → some contribution
patient    → some contribution
because    → lower contribution
concerned  → useful contextual contribution

These are not literal model values.

The point is that Attention allows contributions to be weighted.

An Attention Score is a numerical value used within the Attention mechanism to represent how strongly one token's current processing relates to another token representation before the final attention weighting and combination.

For a simplified example, imagine that while processing she, the model produces relationship scores such as:

doctor    → 4.8
patient   → 1.7
examined  → 0.9
the       → 0.2

These numbers are completely illustrative.

The system then transforms such scores into normalized weights used to combine information.

Relationship Scores
↓
Normalized Attention Weights
↓
Weighted Combination of Information

The exact mathematics is not necessary yet.

What matters is:

Attention does not treat every token relationship as equally important.

Attention Scores Are Context-Dependent

Suppose the word bank appears in:

"The fisherman sat on the bank of the river."

Relationships with tokens such as fisherman, river, and sat can influence its representation.

Now consider:

"The customer transferred money to the bank."

Relationships with customer, transferred, and money provide a different context.

Therefore, the patterns of Attention can differ between the two sentences.

bank + river context
↓
one pattern of relationships

bank + money context
↓
different pattern of relationships

This is one of the mechanisms that allows the model to construct context-sensitive representations.

10. Query, Key, and Value

We now need to understand one of the most famous pieces of Transformer terminology:

Query
Key
Value

usually abbreviated:

Q
K
V

These words often confuse beginners because they sound more complicated than the underlying conceptual idea.

For now, think of Attention as trying to solve this problem:

For the token I am currently processing, what information from the other tokens is useful, and what information should I collect from them?

Q, K, and V create a structured mathematical way to answer that question.

Query in Attention

Inside Attention, a Query is a numerical representation used to ask:

What information am I looking for?

Suppose we are updating a token representation.

The model derives a Query representation from it.

Current token representation
↓
Query representation
↓
"What information is relevant to me?"

This is not a literal English question.

The Query is a vector.

Q = [ ... numerical values ... ]

The word Query describes its role in the Attention calculation.

Attention Query Is Not a Search Query

This distinction is extremely important for the rest of the textbook.

Later, in Information Retrieval, we will use the word Query to mean something such as:

"best family lawyer in Toronto"

That is a search query or user query.

But inside Transformer Attention, Query means an internal numerical representation used in the Attention calculation.

Attention Query
≠
Search Query

Whenever we discuss Q / K / V, we mean the internal Attention concept.

Key in Attention

A Key is another numerical representation associated with a token.

Conceptually, if Query asks:

What information am I looking for?

Key helps answer:

What kind of information do I contain, and how well do I match what you are looking for?

Imagine:

Query from Token A
↓
compared with
↓
Key from Token B

The comparison helps determine how strongly Token A should attend to Token B.

Query A ↔ Key B
↓
relationship score

Value in Attention

The Value is the representation containing the information that can actually contribute to the updated output.

A useful simplified distinction is:

Query
→ What am I looking for?

Key
→ How well do I match what you are looking for?

Value
→ What information can I contribute?

This is not a literal implementation description, but it is a useful conceptual model.

So the basic process becomes:

Query
↓
compared with Keys
↓
Attention Scores
↓
determine importance
↓
Values are weighted
↓
information is combined

This is the heart of Attention.

A Simple Q/K/V Example

Consider:

"The cat drank the milk because it was thirsty."

Suppose the model is processing it.

We want the representation of it to become informed by useful context.

Conceptually, it produces a Query:

Query(it)

Other tokens provide Keys:

Key(cat)
Key(drank)
Key(milk)
Key(thirsty)
...

The Query is compared with those Keys:

Query(it) ↔ Key(cat)

Query(it) ↔ Key(milk)

Query(it) ↔ Key(thirsty)

These comparisons contribute to Attention Scores.

Suppose, illustratively:

cat      → high attention
milk     → lower attention
thirsty  → useful attention

The corresponding Values are then weighted and combined.

Value(cat) × weight
+
Value(milk) × weight
+
Value(thirsty) × weight
+
...
↓
Combined contextual information

That information contributes to the updated representation of it.

Again, the model is not literally thinking:

It probably means cat.

It is performing numerical transformations.

Q, K, and V Come from Representations

Query, Key, and Value are not three separate pieces of text.

They are numerical representations derived from the token representations.

Token Representation
├──→ Query Representation
├──→ Key Representation
└──→ Value Representation

The model uses learned parameters to create these different representations.

Why three?

Because Attention needs to separate three roles:

what is being sought
what can be matched
what information can be contributed

The same token can therefore participate in Attention through different numerical projections.

We do not need the matrix mathematics behind those projections for this course.

11. The Attention Pipeline

We can now construct the process step by step.

Start with token representations:

Token Representations
↓
Create Q, K, V representations

For a particular token:

Its Query
↓
compared with
↓
Keys of other tokens

This produces relationship scores:

Q ↔ K
↓
Attention Scores

The scores are converted into weights:

Attention Scores
↓
Attention Weights

Those weights determine how much information to take from the Values:

Values
×
Attention Weights
↓
Weighted Values

The weighted information is combined:

Weighted Values
↓
Combined Contextual Information

This contributes to an updated token representation:

Original Representation
+
Contextual Information
↓
Updated Representation

This is the essential conceptual mechanism.

Self-Attention Across a Sentence

Consider:

"The restaurant near the station serves excellent sushi."

Every token can participate in Self-Attention.

A simplified relationship pattern might look like:

restaurant
↕
near
↕
station

restaurant
↕
serves
↕
sushi

excellent
↕
sushi

The model is not limited to one relationship.

Many token-to-token relationships can be evaluated.

This means the representation of restaurant can become informed by station, serves, and sushi, while sushi can become informed by serves, excellent, and restaurant.

Each token can therefore develop a representation that reflects the wider context.

12. Contextual Representations and Embeddings

From Attention to Contextual Representation

A contextual representation is a numerical representation whose value depends on the surrounding context.

Consider:

Sentence A:
"The fisherman sat on the bank."

Sentence B:
"She deposited money in the bank."

The initial token bank may begin from a representation associated with that token.

But after Attention incorporates information from surrounding tokens, its representation can differ.

bank
+
fisherman
+
sat
↓
Contextual Representation A

versus:

bank
+
deposited
+
money
↓
Contextual Representation B

So:

same token
+
different context
↓
different contextual representation

Contextual Embedding

A Contextual Embedding is an embedding whose representation depends on the context in which the token or element appears.

Compare:

"I opened an account at the bank."

"The boat reached the river bank."

If we represent bank without considering context, we have ambiguity.

After contextual processing:

bank in financial sentence
→ context-sensitive vector

bank in river sentence
→ different context-sensitive vector

The exact internal behavior of a model is more complex than this simplified diagram.

But the essential principle is correct:

Transformer processing allows representations to become dependent on surrounding context.

Contextual Embeddings vs. Text Embeddings from Chapter 2

We need to distinguish two related ideas.

In Chapter 2, we discussed embeddings such as:

Query Embedding
Document Embedding
Passage Embedding

These can represent an entire piece of text as a vector for tasks such as semantic comparison.

Here, we are discussing contextual representations inside a Transformer.

Sentence
↓
Token representations
↓
Transformer processing
↓
Contextualized token representations

These concepts are related, but they should not be collapsed into one thing.

So a whole-text semantic representation and a context-sensitive internal token representation are related but different ideas.

13. Token Relationships and Context

Transformer processing is fundamentally relational.

Consider:

"Sarah gave Emma the book because she had finished reading it."

Understanding the sentence involves relationships such as:

Sarah ↔ gave
Emma ↔ gave
book ↔ gave
she ↔ possible person reference
it ↔ book
reading ↔ book

Language understanding depends heavily on such relationships.

A token does not exist semantically in isolation.

Its interpretation can depend on:

• nearby tokens;
• distant tokens;
• sentence structure;
• entities;
• actions;
• attributes;
• and references.

Self-Attention gives the model a mechanism for representing these interactions numerically.

Context Changes Representation

Consider the word:

apple

in:

"She ate an apple after lunch."

versus:

"Apple released a new device."

The surface word is the same.

But surrounding tokens provide very different signals.

First context:

ate
lunch

Second context:

released
device

Through contextual processing, the model can develop different internal representations.

apple + food context
↓
fruit-related representation

Apple + technology context
↓
company-related representation

This illustrates why contextual representation is more powerful than treating each word as having one fixed meaning.

Contextual Understanding

The phrase contextual understanding should be used carefully.

A model does not necessarily “understand” in exactly the same way a human does.

In this textbook, when we say a model uses contextual understanding, we mean that its internal representations and outputs can depend on relationships among information in the current context.

Token alone
↓
limited information

Token + surrounding context
↓
richer representation

14. Long-Range Dependencies

A long-range dependency is a meaningful relationship between elements that are separated by other elements in a sequence.

Consider:

"The lawyer who moved from London to Toronto after working abroad for several years opened her own practice."

The relationship:

lawyer ↔ her

spans many intervening tokens.

Likewise:

lawyer ↔ opened

is important.

Another example:

"The restaurant that my brother recommended when he visited the city last summer has finally reopened."

The relationship:

restaurant ↔ reopened

extends across a long phrase.

A model needs mechanisms that can preserve and use these relationships.

Self-Attention makes direct interaction between separated positions possible.

Why Self-Attention Helps with Long-Range Dependencies

Imagine two important tokens separated by many other tokens:

Token A
↓
token
token
token
token
token
token
↓
Token B

With Self-Attention, Token A and Token B can participate directly in an Attention relationship within the sequence.

Token A ───────────────────── Token B

The relationship does not have to be understood only through the immediately neighboring token.

This is an important strength of the Transformer architecture.

However, the fact that Attention can connect distant tokens does not mean that all long contexts are understood perfectly, all tokens receive equal effective attention, or information can never be lost or underused.

Chapter 1 already introduced the idea of information competition inside a finite context.

Long-range access is a capability, not a guarantee of perfect use.

Attention Does Not Mean Every Token Is Equally Important

Suppose a context contains 5,000 tokens.

A model does not need every token to influence every computation equally.

Attention provides a mechanism for differential weighting.

Current token
↓
Token A → strong influence
Token B → weak influence
Token C → moderate influence
Token D → very weak influence
...

This connects back to a concept from Chapter 1: Information Competition.

A model may technically have information inside its context while different pieces of that information have different influence on a particular computation.

15. Multi-Head Attention

One token may participate in several different types of relationships simultaneously.

Consider:

"The experienced lawyer opened a new office in Toronto."

Different relationships matter:

experienced ↔ lawyer

lawyer ↔ opened

opened ↔ office

office ↔ Toronto

For example, lawyer has:

an attribute → experienced
an action → opened
a relationship to an object → office

Language therefore cannot always be represented well by asking only one question about each token.

Transformers address this partly through a mechanism called Multi-Head Attention.

Instead of performing only one Attention pattern, Transformers can use multiple Attention mechanisms in parallel.

These are commonly called Attention Heads.

Input Representations
↓
Attention Head 1
Attention Head 2
Attention Head 3
...
↓
Combined Information

Different heads can learn different useful relationship patterns.

It would be too simplistic to say:

Head 1 = grammar
Head 2 = entities
Head 3 = location

Real learned behavior is not so cleanly assigned.

But the broader idea is useful:

Multiple Attention heads allow the model to represent multiple patterns of interaction at the same time.

Attention Is Only Part of a Transformer Layer

Another important correction:

Transformer
≠
Attention alone

Attention is a central component, but a Transformer layer contains additional operations.

At a simplified conceptual level:

Input Representations
↓
Attention-related processing
↓
Additional learned transformations
↓
Output Representations

There are also mechanisms that help stabilize and combine information as it passes through the network.

We do not need their detailed mathematics for this course.

Attention is a central mechanism inside the Transformer architecture, not the entire architecture by itself.

16. Repeated Contextualization Across Layers

Now combine Transformer layers with Attention.

Suppose we start with:

"The mouse is beside the keyboard."

At an early stage:

mouse
↓
initial numerical representation

A Transformer layer allows information from the context to influence it:

mouse
+
keyboard
+
beside
↓
updated representation

The next layer receives that updated representation:

updated mouse representation
+
other updated token representations
↓
further contextualization

And this continues through many layers.

Initial Representation
↓
Contextualization
↓
Updated Representation
↓
More Contextualization
↓
Richer Contextual Representation

This repeated processing is one reason deep Transformer models can represent complex relationships.

The Representation of Every Token Can Change

It is easy to focus only on ambiguous words such as:

bank
mouse
apple

But contextualization is broader than word-sense disambiguation.

Consider:

"The lawyer represented the doctor."

The representations of lawyer, represented, and doctor can all become informed by one another.

Now reverse the sentence:

"The doctor represented the lawyer."

The same words appear.

But the relationships have changed.

Therefore, contextual representations should also change.

Context is not only about which words appear. It is also about how those words relate to one another.

Word Order Matters

Compare:

"The lawyer called the doctor."

with:

"The doctor called the lawyer."

The same major words appear:

lawyer
called
doctor

But the meanings differ.

The model therefore needs information about token position and order.

Transformer systems include mechanisms that allow position information to influence processing.

We will not explore specific positional encoding implementations here because they are not required for the later AI Visibility concepts in this curriculum.

Transformer processing needs both token information and information about where tokens occur in the sequence.

Contextual Representation Is Dynamic

Suppose we isolate:

cold

Now place it in different contexts:

"The weather is cold."

"She has a cold."

"He gave me a cold response."

The same surface form participates in three different meanings:

temperature
illness
emotional/social tone

The model does not need a completely separate vocabulary entry for every possible contextual meaning.

token representation
+
current context
↓
contextualized representation

This is why contextual representation is dynamic.

Static Parameters, Dynamic Representations

During normal inference, the model's learned parameters are generally not being rewritten for every sentence.

Yet its internal representations change from one input to another.

Same Model Parameters
+
Input A
↓
Contextual Representations A

Same Model Parameters
+
Input B
↓
Contextual Representations B

Therefore:

Changing contextual representation
≠
changing model parameters

This distinction becomes especially important in Chapter 4 when we compare training with inference.

17. A Complete Simplified Transformer Example

Let us connect the process using:

"The mouse is next to the keyboard."

Step 1 — Text Becomes Tokens

From Chapter 1:

Text
↓
Tokenizer
↓
Tokens

Simplified:

"The"
"mouse"
"is"
"next"
"to"
"the"
"keyboard"
"."

Real tokenization may differ.

Step 2 — Tokens Receive Numerical Representations

The model needs numerical representations for processing.

Token
↓
Initial Numerical Representation

Step 3 — Q, K, and V Representations Are Produced

Token Representations
↓
Q / K / V Representations

Step 4 — Queries Are Compared with Keys

For the representation associated with mouse, the system can calculate relationships with other token representations:

Q(mouse) ↔ K(The)

Q(mouse) ↔ K(is)

Q(mouse) ↔ K(keyboard)

...

Step 5 — Attention Scores Determine Weighting

Contextually useful relationships can receive stronger weights.

For interpreting mouse, the relationship with keyboard may be especially informative.

Step 6 — Values Are Combined

Weighted Values
↓
Combined Information

Step 7 — The Token Representation Is Updated

Now mouse is no longer represented only as an isolated vocabulary item.

Its representation contains information influenced by the current context.

Step 8 — More Transformer Layers Repeat the Process

Layer 1
↓
updated representations

Layer 2
↓
further updated representations

Layer 3
↓
further updated representations

...

The result is a set of deeply contextualized representations.

From Token to Contextual Meaning

Token
↓
Initial Representation
↓
Self-Attention
↓
Relationships with Other Tokens
↓
Weighted Contextual Information
↓
Updated Representation
↓
More Transformer Layers
↓
Contextual Representation

This is a major step beyond Chapter 2.

Chapter 2 taught us that numerical representations can encode useful semantic relationships.

Chapter 3 adds:

Those representations can be transformed according to the specific context in which they appear.

18. Why These Concepts Matter

Why This Matters for LLM Behavior

Suppose you ask:

"Can jaguars climb trees?"

The context strongly suggests:

jaguar = animal

Now consider:

"Does Jaguar manufacture electric cars?"

The context strongly suggests:

Jaguar = automotive brand

A system that treated jaguar as one fixed isolated meaning would struggle.

Contextual processing allows the representation to change according to surrounding information.

This helps modern LLMs handle:

• ambiguity;
• references;
• relationships;
• word roles;
• sentence structure;
• context-dependent meanings;
• and long-range dependencies.

Why This Matters for Long Context

Chapter 1 introduced:

Context Window
Context Limit
Context Length
Token Budget
Information Competition

Now we can add another layer to that mental model.

The context window determines what information is available for processing.

Attention helps determine how information inside that context can influence representations.

Context Window
→ what information is available

Attention
→ how information can interact and influence processing

These are not the same thing.

A piece of information being inside the context window does not automatically mean that it will dominate the model's computation.

Available in context
≠
equally influential everywhere

This distinction will become useful later when we study retrieved context in RAG.

Why This Matters for AI Visibility

We should keep this connection limited because retrieval, ranking, and citation mechanisms have not yet been introduced.

But one useful implication can already be understood.

AI systems do not necessarily interpret words and entities independently.

Suppose a passage says:

"Northstar Dental provides emergency dental treatment in Toronto and accepts same-day appointments."

The representation of concepts such as:

Northstar Dental
emergency
dental treatment
Toronto
same-day appointments

can be influenced by their relationships inside the passage.

This helps explain why meaningful context around an entity matters.

Simply repeating:

Northstar Dental
Northstar Dental
Northstar Dental

does not provide the same contextual information as clearly expressing relationships such as:

Northstar Dental
↓
provides
↓
emergency dental treatment
↓
in Toronto
↓
with same-day appointments

However, we should not turn this into a ranking rule.

Nothing in Transformer architecture alone proves that a particular writing format will cause an AI system to retrieve, recommend, mention, or cite a business.

The useful principle at this stage is simply:

Meaning emerges partly from relationships within context, not from isolated terms alone.

19. Common Misunderstandings

Misunderstanding 1: An LLM stores every concept in one parameter

Incorrect. Knowledge and capabilities are generally distributed across many parameters and computations.

Misunderstanding 2: More parameters always means a better model

Incorrect. Parameter count is one dimension of model scale, not a universal quality score.

Misunderstanding 3: Attention means the model consciously focuses

Incorrect. Attention is a mathematical mechanism.

Misunderstanding 4: Self-Attention only compares neighboring tokens

Incorrect. It can model relationships between separated positions within the available context.

Misunderstanding 5: Attention Score is semantic similarity from Chapter 2

Incorrect.

A semantic similarity score compares representations for semantic similarity according to a chosen similarity method.

An Attention Score is part of the internal Attention mechanism used to determine how representations interact.

Misunderstanding 6: Attention Query means user search query

Incorrect.

Attention Query
= internal Q representation

Search Query
= information request such as "best dentist in Toronto"

Misunderstanding 7: Query, Key, and Value are pieces of text

Incorrect. They are numerical representations derived from token representations.

Misunderstanding 8: Value means how valuable a token is

Incorrect. Value is the technical name for the representation carrying information that can be weighted and combined in Attention.

Misunderstanding 9: Transformer means Attention

Incomplete. Attention is a central component of Transformer architecture, but it is not the entire Transformer.

Misunderstanding 10: A token always has one fixed final representation

Incorrect. Its contextual representation can change according to surrounding context.

Misunderstanding 11: Contextual representation means model parameters changed

Incorrect.

Parameters can remain fixed
while
internal representations change dynamically

Misunderstanding 12: Being inside the context window guarantees strong influence

Incorrect. Availability and influence are different concepts.

20. How the Concepts Connect

We can now connect everything introduced in this chapter.

Start with human-readable text:

"The mouse is beside the keyboard."

Chapter 1 gives us:

Text
↓
Tokenizer
↓
Tokens
↓
Token IDs

The model needs numerical representations:

Token IDs
↓
Initial Token Representations

Now Transformer processing begins.

From those representations, the Attention mechanism creates:

Query
Key
Value

For each relevant interaction:

Query
↓
compared with Keys
↓
Attention Scores
↓
Attention Weights

The weights determine how information from Values is combined:

Values
+
Attention Weights
↓
Weighted Contextual Information

That information contributes to updated representations:

Initial Token Representation
+
Contextual Information
↓
Updated Token Representation

Multiple Transformer layers repeat and extend this process:

Updated Representation
↓
Next Transformer Layer
↓
Further Contextualization
↓
Next Layer
↓
Further Contextualization

The result is:

Contextual Representations

Therefore, the core dependency is:

Tokens
↓
Embeddings / Numerical Representations
↓
Self-Attention
↓
Q / K / V
↓
Attention Scores
↓
Weighted Information
↓
Transformer Layers
↓
Contextual Representations

And conceptually:

Token Meaning
+
Relationships with Context
↓
Context-Sensitive Representation

Mental Model

Imagine a meeting.

Each person initially arrives with their own information.

That is roughly analogous to each token beginning with its own representation.

But nobody stays isolated.

Each person listens differently to the others depending on what is relevant to the current discussion.

One person may strongly influence another.

Someone else's contribution may matter only slightly.

Different relationships matter for different purposes.

After the discussion, each person's understanding has changed because they now contain information influenced by the group.

Individual Token Representations
↓
Tokens interact through Attention
↓
Relevant information receives different weights
↓
Information is combined
↓
Representations become context-sensitive

The analogy has limits.

Tokens are not people.

The model is not consciously listening.

Attention is mathematical.

But the analogy captures the essential transformation:

Isolated representation
↓
interaction with context
↓
contextual representation

The Q/K/V mechanism gives us a more technical version:

Query
→ What information is relevant to this representation?

Key
→ How well does this information match what is being sought?

Value
→ What information can be contributed?

Query × Key relationships
↓
Attention Scores
↓
Weighted Values
↓
Contextual Information

Repeated across Transformer layers:

Tokens
↓
Representations
↓
Attention
↓
Updated Representations
↓
More Layers
↓
Deep Contextual Representations

The simplest reliable mental model is:

A Transformer repeatedly lets token representations interact through Attention, allowing each representation to become informed by the context around it.

Chapter 3 Recap

Language Model

A computational model designed to learn and represent patterns in language.

Large Language Model

A large-scale language model containing many learned parameters and capable of modeling complex language patterns.

Parameter

A numerical value learned during training that influences how the model transforms input into output.

Model Size

A description of model scale, often including the number of parameters. More parameters do not automatically guarantee better performance.

Neural Network

A computational system that transforms numerical representations through layers of learned mathematical operations.

Architecture

The structural design describing how a model's major computational components are organized and interact.

Transformer

A neural-network architecture that processes sequences and uses Attention to model relationships among their elements.

Transformer Layer

One processing layer inside a Transformer that transforms representations using Attention and additional learned operations.

Attention

A mathematical mechanism for weighting relationships between representations and combining information accordingly.

Self-Attention

Attention among elements of the same sequence.

Attention Score

A numerical value used inside Attention to help determine the strength of a relationship between representations.

Attention Weight

A normalized weighting derived from Attention Scores that determines how strongly information contributes to the combined result.

Query — Q

An internal Attention representation describing what information is being sought for the current computation. It is not the same as a user search query.

Key — K

An internal Attention representation used to determine how well another representation matches the Query.

Value — V

An internal Attention representation containing information that can be weighted and combined into the Attention output.

Contextual Representation

A numerical representation whose value has been influenced by surrounding context.

Contextual Embedding

An embedding whose representation depends on the context in which the represented token or element appears.

Token Relationships

The semantic, grammatical, referential, and other contextual relationships that can exist between tokens in a sequence.

Long-Range Dependency

A meaningful relationship between elements separated by other elements in a sequence.

Multi-Head Attention

The use of multiple Attention mechanisms in parallel, allowing the model to represent multiple patterns of interaction.

Static Parameters vs. Dynamic Representations

During normal inference, model parameters generally remain fixed while internal contextual representations change according to the current input.

The dependency developed across the first three chapters is now:

Human Text
↓
Tokenization
↓
Tokens
↓
Token IDs
↓
Numerical Representations
↓
Embeddings
↓
Vectors
↓
Self-Attention
↓
Q / K / V
↓
Attention Scores and Weights
↓
Transformer Layers
↓
Contextual Representations

We can now explain how a model can take tokens and build context-sensitive representations from them.

But one major question remains:

Where did all of the useful parameters and representations come from?

The Transformer architecture tells us how the model processes information.

It does not yet explain how the model learned to perform those transformations.

That leads directly into Chapter 4:

How LLMs Learn.

The Single Most Important Idea from Chapter 3

A Transformer uses Attention to let token representations interact with other tokens in the context, repeatedly transforming them into context-sensitive representations whose meaning depends on their relationships with the surrounding information.