Skip to content

AI Visibility Guide

01 Part I — Foundations of Large Language Models Chapter 01.02

Chapter 2- Numerical Representation of Meaning

By Mo Farahani 33 min read

Chapter Goal

In Chapter 1, we established that a language model does not receive text directly as human-readable words. Text is processed by a tokenizer, converted into tokens, and each token is associated with a token ID. We also established that token IDs are identifiers, not representations of meaning.

That creates the central question of this chapter:

How can a computer represent meaning numerically?

To answer it, we need to move from discrete identifiers such as token IDs to mathematical representations that can express relationships between pieces of language.

This chapter introduces four connected layers:

Vector fundamentals
↓
Embeddings
↓
Semantic relationships
↓
Similarity

These concepts form the foundation for later topics such as semantic search, vector search, and dense retrieval.

By the end of this chapter, you should understand:

  • what a vector is;
  • what dimensions represent;
  • what a vector space is;
  • why useful language representations often require many dimensions;
  • what an embedding is;
  • what an embedding model does;
  • the difference between text, query, document, and passage embeddings;
  • what an embedding space is;
  • what semantic meaning, similarity, distance, and relatedness mean;
  • how cosine similarity compares vectors;
  • what a similarity score represents;
  • why semantic similarity is not the same as relevance;
  • why lexical similarity is different from semantic similarity;
  • and why exact matching and meaning matching solve different problems.

The goal is not to turn you into a machine-learning engineer. The goal is to give you a precise mental model of how meaning can become something a computer can compare mathematically.

1. The Problem: Computers Need Numbers

Suppose we have these three pieces of text:

A: “cheap hotels in London”

B: “affordable accommodation in London”

C: “how to repair a bicycle”

A human immediately sees that A and B are closely related.

C is about something completely different.

But a computer ultimately operates on numbers. It needs some numerical representation that makes the relationship between A and B detectable.

Chapter 1 gave us one numerical object already: the token ID.

Suppose, purely illustratively, that some tokenizer produces:

"hotel" → token ID 4172
"accommodation" → token ID 9811
"bicycle" → token ID 2547

Those numbers do not tell us that hotel and accommodation are related.

In particular:

9811 - 4172 = 5639

does not represent any meaningful semantic relationship.

Token ID 9811 is not “5639 units farther in meaning” from token ID 4172.

A token ID is essentially an identifier used to distinguish one vocabulary item from another.

This distinction is fundamental:

Token IDs identify. Semantic representations describe relationships.

We therefore need another kind of numerical representation.

That representation begins with a mathematical object called a vector.

2. Vector Fundamentals

2.1 What Is a Vector?

At the level required for this textbook, a vector is an ordered list of numbers.

For example:

[0.4, -0.2, 0.7]

is a vector.

Another vector might be:

[0.1, 0.8, -0.3]

The order matters.

These two vectors are different:

[0.4, -0.2, 0.7]

[0.7, -0.2, 0.4]

even though they contain the same three numbers.

Each position in a vector is called a dimension.

So the vector:

[0.4, -0.2, 0.7]

has three dimensions.

Vector
↓
Ordered numerical values
↓
One value per dimension

Vectors are useful because mathematical operations can compare them.

That gives us a way to transform the vague idea of “these two pieces of text mean similar things” into something measurable.

2.2 Vector Representation

A vector representation is the use of a vector to represent something.

That “something” could be many kinds of objects.

Object → Vector

A simplified system might represent a product as:

Product A → [0.8, 0.2, 0.6]

Another product might become:

Product B → [0.7, 0.3, 0.5]

The important idea is that once objects are represented numerically, mathematical methods can compare them.

For language, we want something more powerful:

Text → Vector

For example:

"affordable hotel in London"
↓
[0.17, -0.42, 0.81, ..., 0.23]

The numbers here are illustrative.

The vector is intended to encode useful information about the text in numerical form.

That does not mean that the original sentence has literally been converted into a human-readable list of semantic properties. We will return to this distinction because it prevents a common misunderstanding about embeddings.

2.3 Dimensions

A dimension is one position, or numerical coordinate, within a vector.

Consider:

[0.4, -0.2, 0.7]

This vector has:

Dimension 1 → 0.4
Dimension 2 → -0.2
Dimension 3 → 0.7

It is therefore a three-dimensional vector.

A vector with five values:

[0.3, -0.1, 0.8, 0.5, -0.6]

has five dimensions.

In real language systems, vectors can contain hundreds or thousands of dimensions.

Instead of:

[x1, x2, x3]

we may conceptually have:

[x1, x2, x3, ..., xn]

where n represents the total number of dimensions.

2.4 Do Individual Dimensions Have Human-Readable Meanings?

This requires an important clarification.

Imagine that we create a simple representation of restaurants ourselves.

We might deliberately define:

Dimension 1 = price
Dimension 2 = distance
Dimension 3 = rating

Then:

[0.3, 0.8, 0.9]

would have a clear interpretation because we explicitly designed the dimensions.

Language embeddings generally should not be understood this way.

It is usually incorrect to imagine:

Dimension 1 = food
Dimension 2 = location
Dimension 3 = price
Dimension 4 = happiness
Dimension 5 = business

Real embedding dimensions usually do not correspond cleanly to individually named concepts.

Instead, semantic information is distributed across the vector.

This means that a concept such as:

"restaurant"

is not normally stored in one dedicated “restaurant dimension.”

Its representation emerges from the overall pattern of values across many dimensions.

A vector may encode meaningful structure even when individual dimensions are not independently interpretable to humans.

This is one reason we should focus on relationships between vectors rather than trying to read individual coordinates as if they were labeled database fields.

2.5 From Dimensions to Space

Once vectors have multiple dimensions, we can think of them as positions in a space.

Start with something familiar.

A point on a two-dimensional map can be represented with two coordinates:

[x, y]

For example:

A = [2, 3]
B = [3, 4]
C = [9, 10]

You can imagine these as points on a flat plane.

A and B are relatively close.

C is farther away.

With three dimensions:

[x, y, z]

we can imagine points inside a three-dimensional space.

The same mathematical idea continues beyond three dimensions.

A vector with 100 dimensions represents a point in a 100-dimensional space.

A vector with 1,000 dimensions represents a point in a 1,000-dimensional space.

Humans cannot visualize these spaces directly, but mathematics can still operate on them.

This leads to the concept of high-dimensional space.

2.6 High-Dimensional Space

A high-dimensional space is a space whose points are represented using many dimensions.

Suppose an embedding model produces vectors with hundreds of numerical values.

Then every text processed by that model can be represented as a point in that high-dimensional space.

Text A
↓
[... hundreds of values ...]
↓
Point A

Text B
↓
[... hundreds of values ...]
↓
Point B

We cannot draw this space accurately on paper because our physical intuition is built around three dimensions.

But mathematical comparison does not require human visualization.

Visualization is limited to a few dimensions; computation is not.

A system can compare vectors containing hundreds or thousands of dimensions even though we cannot mentally picture the geometry.

2.7 Vector Space

A vector space, in the practical sense needed here, is the mathematical space in which vectors exist and can be compared.

Imagine that several pieces of text have been converted into vectors:

"best Italian restaurants"
"good places for Italian food"
"how to make pasta"
"weather tomorrow"
"laptop battery replacement"

Each becomes a point in a vector space.

If the representation captures semantic information effectively, the geometry of that space can reflect relationships between the texts.

"best Italian restaurants"
          ●

"good places for Italian food"
        ●


"how to make pasta"
                 ●



                          ● "weather tomorrow"


                                      ● "laptop battery replacement"

This drawing is only a simplified two-dimensional analogy.

Real embedding spaces have many more dimensions.

Meaningful relationships can be represented geometrically as relationships between vectors.

That brings us to embeddings.

3. Embeddings

3.1 What Is an Embedding?

An embedding is a numerical vector representation of an object designed so that useful relationships between objects are reflected in the structure of the vector space.

In this chapter, the objects we care about are primarily pieces of text.

Text
↓
Embedding process
↓
Vector

For example:

"dentist in Toronto"
↓
[0.21, -0.48, 0.73, ..., 0.14]

Again, those numbers are illustrative.

The vector is the embedding of the text.

A useful text embedding does more than assign arbitrary numbers to text. It places texts into a shared numerical space in a way that captures useful semantic relationships.

For example:

"dentist in Toronto"
"Toronto dental clinic"

should generally receive representations that indicate stronger semantic similarity than:

"dentist in Toronto"
"how to bake sourdough bread"

Embeddings make aspects of meaning mathematically comparable.

3.2 Embedding vs Vector

These terms are closely related but should not be treated as perfect synonyms.

A vector is a mathematical object:

[0.2, 0.7, -0.4]

An embedding is a representation of something as a vector within a learned representation space.

Vector
= mathematical structure

Embedding
= representation of an object using that structure

Every embedding discussed here is represented as a vector.

But not every vector is necessarily an embedding.

For example:

[age, height, weight]

could be a manually constructed vector describing a person.

That does not automatically make it a learned semantic embedding.

3.3 What Is an Embedding Model?

An embedding model is a model that converts input objects—such as text—into embeddings.

Text
↓
Embedding Model
↓
Embedding Vector

Suppose we provide:

"family lawyer in Toronto"

The model might produce something conceptually like:

[0.08, -0.31, 0.62, ..., -0.19]

Then:

"Toronto family law attorney"

might produce another vector:

[0.11, -0.29, 0.59, ..., -0.17]

The exact numbers are not important for our conceptual understanding.

What matters is that both vectors belong to the same embedding space and can therefore be meaningfully compared.

3.4 The Same Embedding Model Matters

Suppose Model A represents text using 768 dimensions.

Model B represents text using 1,024 dimensions.

Even more importantly, the two models may have learned entirely different coordinate systems.

Therefore, a vector generated by one embedding model generally cannot simply be compared as though it belonged to the embedding space of another unrelated model.

Embedding Model A
Text → Vector in Space A

Embedding Model B
Text → Vector in Space B

Space A and Space B are not automatically interchangeable.

Even if two models happened to produce vectors with the same number of dimensions, that would not mean that corresponding coordinates had the same meaning.

Vectors are meaningful in relation to the representation space that produced them.

3.5 Text Embeddings

A text embedding is an embedding representing text.

The input might be:

  • a short phrase;
  • a sentence;
  • a question;
  • a paragraph;
  • a passage;
  • or a larger document.
Text
↓
Embedding Model
↓
Text Embedding

For example:

"Where can I find an immigration lawyer in Vancouver?"
↓
Embedding Model
↓
Vector

The purpose is not to preserve the original text word-for-word inside the vector.

Instead, the representation captures patterns useful for comparing that text with other text.

An embedding is not a compressed sentence that can simply be read backward.

It is a learned numerical representation.

3.6 Embeddings Are Not Lossless Encodings

Consider a ZIP file.

You can compress a text file and later reconstruct the original text exactly.

An embedding should not be understood in the same way.

Sentence → Embedding

does not mean that we can assume:

Embedding → Exact original sentence

The embedding is designed to represent useful characteristics of the input, not necessarily to preserve every character and word in a reversible form.

Encoding for exact reconstruction

≠

Representation for useful comparison

Text embeddings belong primarily to the second category.

3.7 Query Embeddings

A query embedding is the vector representation of a query.

Query:
"Where can I find a Persian dentist in Toronto?"
↓
Embedding Model
↓
Query Embedding

The term query here means an information request that may later be used for retrieval.

We are introducing only its representation here. The retrieval process itself belongs to later chapters.

The important point is that a query can be transformed into the same kind of numerical space as other text.

This makes mathematical comparison possible.

3.8 Document Embeddings

A document embedding is a vector representation of a document.

Document
↓
Embedding Model
↓
Document Embedding

Now both a query and a document can have numerical representations:

Query    → Query Vector
Document → Document Vector

If both belong to a compatible embedding space, their semantic relationship can be measured.

This idea will later become extremely important in semantic retrieval.

For now, we only need the representation layer.

3.9 Passage Embeddings

A passage embedding represents a smaller section of text rather than an entire document.

Document
├── Passage A
├── Passage B
├── Passage C
└── Passage D

Each passage can potentially receive its own embedding:

Passage A → Vector A
Passage B → Vector B
Passage C → Vector C
Passage D → Vector D

Consider a long page about a dental clinic containing:

Passage 1 → clinic history
Passage 2 → dentist qualifications
Passage 3 → emergency dental services
Passage 4 → opening hours
Passage 5 → payment methods

A single representation of the entire document combines information from many subjects.

Individual passage embeddings allow smaller units of meaning to receive their own representations.

We are not yet discussing how a retrieval system chooses between them. That belongs later.

Document Embedding
= representation of a larger document

Passage Embedding
= representation of a smaller section of text

3.10 Query, Document, and Passage Embeddings Are Roles

The labels query embedding, document embedding, and passage embedding describe the role of the represented text.

For example:

"emergency dentist open on Sunday"

could be treated as a query.

A paragraph explaining Sunday emergency service could be treated as a passage.

Both become vectors.

Query
↓
Query Embedding
        \
         → comparison
        /
Passage
↓
Passage Embedding

This allows systems to compare information requests with information-containing text in a common numerical framework.

3.11 Embedding Space

An embedding space is the vector space created by an embedding model in which embedded objects are represented.

Suppose we embed these sentences:

A: "I need a lawyer for my divorce."

B: "Looking for a family law attorney."

C: "The restaurant serves Persian kebabs."

D: "This café offers traditional Iranian food."

E: "My laptop battery is damaged."

A useful embedding model may arrange them so that:

A ↔ B

are relatively close, and:

C ↔ D

are relatively close, while E is farther from both groups.

A ●     ● B


              C ●
                 ● D



                                  ● E

Again, this is only a visual analogy.

The actual vectors may have hundreds or thousands of dimensions.

Relationships between vectors in embedding space can reflect relationships between the represented texts.

3.12 Embedding Space Is Relational

An individual embedding vector is difficult to interpret by itself.

[0.137, -0.084, 0.921, ..., -0.224]

Without knowing the model and without comparing the vector with others, this tells you very little.

The power of embedding spaces comes from relationships.

Vector A compared with Vector B
Vector A compared with Vector C
Vector A compared with Vector D

We care about questions such as:

  • Which vectors are similar?
  • Which are different?
  • Which are nearby according to some mathematical measure?
  • Which texts occupy related regions of the space?

This is why the next part of the chapter focuses on semantic relationships.

4. Semantic Relationships

4.1 Semantic Meaning

Semantic meaning refers to meaning expressed by language rather than merely its exact surface wording.

Consider:

"cheap hotel"

"affordable accommodation"

The wording differs.

But their meanings overlap strongly.

Now compare:

"cheap hotel"

"hotel management software"

These share the word hotel, but their meanings are substantially different.

Words and meaning are related, but they are not the same thing.

A representation concerned with semantics attempts to capture meaning-related patterns beyond simple character or word overlap.

4.2 Semantic Similarity

Semantic similarity measures how similar two items are in meaning.

A: "How much does a private math tutor cost?"
B: "What is the price of one-to-one mathematics tutoring?"
C: "How do I become a mathematics teacher?"

A and B express very similar information needs.

C shares some vocabulary and topic with them but asks something different.

semantic_similarity(A, B) = high
semantic_similarity(A, C) = lower

The actual score depends on the model and similarity method.

The important point is that semantic similarity concerns meaning, not simply shared words.

4.3 Semantic Distance

If similarity asks:

How alike are these representations?

Semantic distance expresses the complementary idea:

How far apart are these representations in semantic space?

More similar meaning
→ smaller semantic distance

Less similar meaning
→ larger semantic distance

Imagine:

"dentist near me"        ●──● "local dental clinic"

versus:

"dentist near me"        ●────────────────────● "history of aircraft engines"

The visual distances are illustrative, not literal.

The key idea is that semantic relationships can be expressed geometrically.

However, similarity and distance are related concepts, but they are not automatically the same mathematical quantity.

Similarity → degree of likeness

Distance → degree of separation

4.4 Semantic Relatedness

Semantic relatedness is broader than semantic similarity.

Two things may be strongly related without having similar meanings.

"doctor"
"hospital"

These are related concepts, but they do not mean the same thing.

Similarly:

"restaurant"
"menu"

are strongly related, but a restaurant is not a menu.

And:

"lawyer"
"court"

are related without being semantically equivalent.

Compare that with:

"lawyer"
"attorney"

These are much closer in meaning.

Semantic similarity
→ How alike are the meanings?

Semantic relatedness
→ How strongly are the concepts associated?

Relatedness can include relationships such as:

profession ↔ workplace

product ↔ component

business ↔ service

cause ↔ effect

tool ↔ activity

This distinction becomes important whenever someone assumes that two nearby concepts must be synonyms.

They do not.

5. Measuring Similarity

5.1 Similarity Needs a Measurement

We now have two vectors:

Vector A
Vector B

and we want to know how similar they are.

We need a mathematical comparison method.

There are multiple possible methods.

One particularly important method is cosine similarity.

5.2 Cosine Similarity

Cosine similarity is a mathematical measure commonly used to compare vectors by examining the angle between their directions.

You do not need advanced geometry to understand the core idea.

       A
      ↗
     /
    /
   •────────→ B

If the arrows point in very similar directions, their cosine similarity is high.

If they point in very different directions, the similarity is lower.

Similar direction
→ high cosine similarity

Different direction
→ lower cosine similarity

For embeddings, those arrows exist in a high-dimensional space rather than the two-dimensional drawing above.

The mathematics still works.

5.3 Why Direction Matters

Consider two simplified vectors:

A = [1, 2]
B = [2, 4]

B is twice as large as A numerically, but both point in exactly the same direction.

A ↗
B ↗

Cosine similarity focuses on that directional alignment rather than simply asking whether every coordinate has exactly the same magnitude.

This property makes cosine similarity useful for many embedding comparisons.

The precise usefulness depends on how a particular embedding model and system are designed, so we should not treat cosine similarity as a universal law of semantic meaning.

It is a mathematical comparison method applied to vector representations.

5.4 The Cosine Similarity Formula

For two vectors A and B, cosine similarity is:

cos(θ) = (A · B) / (||A|| ||B||)

where:

  • A · B is the dot product of the two vectors;
  • ||A|| is the magnitude, or length, of vector A;
  • ||B|| is the magnitude of vector B;
  • θ is the angle between them.

You do not need to calculate this manually in normal AI Visibility work.

The numerator measures how strongly the vectors align.

The denominator adjusts for their lengths.

The resulting value reflects their directional similarity.

5.5 A Simple Cosine Similarity Example

Consider:

A = [1, 1]
B = [2, 2]

They point in the same direction.

Their cosine similarity is:

1

Now consider:

A = [1, 0]
B = [0, 1]

These vectors are perpendicular.

Their cosine similarity is:

0

And:

A = [1, 0]
B = [-1, 0]

point in opposite directions.

Their cosine similarity is:

-1

So mathematically:

-1 ≤ cos(θ) ≤ 1

1  → same direction
0  → perpendicular
-1 → opposite direction

For practical embedding systems, observed score distributions depend on the model and data. Therefore, you should not assume that every embedding application regularly uses the entire mathematical range.

5.6 Similarity Score

A similarity score is a numerical value expressing the similarity between two representations according to some chosen similarity function.

Query:
"affordable family lawyer"

Document A:
"Low-cost legal help for divorce and custody cases"

Similarity score: 0.86

and:

Document B:
"Best restaurants for family dinners"

Similarity score: 0.31

These numbers are illustrative.

The first pair is semantically more similar than the second according to our hypothetical system.

But a score such as:

0.86

does not automatically mean:

86% same meaning

or

86% relevant

or

86% probability of being correct

It is a score produced by a particular representation and comparison process.

Its interpretation depends on the model, metric, data, and application.

5.7 Similarity Scores Are Relative to the System

Suppose one embedding system produces:

A ↔ B = 0.82

and another produces:

A ↔ B = 0.67

This does not automatically mean the first system understands the relationship better.

The score scales may behave differently.

Even within one model, the meaning of a score depends on the distribution of scores for the task being performed.

This score expresses similarity according to this model and this comparison method, and should be interpreted within that system.

6. Important Distinctions

6.1 Lexical Similarity

Lexical similarity concerns similarity in the words or textual forms being used.

A: "best dentist in Toronto"
B: "best dentist in Vancouver"

These sentences have high lexical overlap.

Now consider:

A: "cheap lawyer"
B: "affordable attorney"

These have much less exact word overlap.

Yet their meanings may be very similar.

Lexical similarity
→ similarity of words/forms

Semantic similarity
→ similarity of meaning

6.2 Lexical Similarity vs Semantic Similarity

A: "cheap hotels in London"

B: "affordable accommodation in London"

C: "cheap flights to London"

D: "London hotels with low room rates"

A and B have strong semantic similarity despite different wording.

A and C share words such as cheap and London, but the central object differs:

hotels vs flights

A and D use somewhat different phrasing but communicate closely related meanings.

This shows why lexical overlap cannot fully represent semantic similarity.

cheap ↔ affordable

lawyer ↔ attorney

car repair ↔ automobile repair

Embedding-based representations are designed to capture useful relationships beyond exact lexical overlap.

6.3 Exact Match

An exact match occurs when the system requires the same textual item or pattern to appear.

For example:

"Persian restaurant"

An exact-match approach may look specifically for that phrase.

A document containing:

"This Iranian dining establishment serves traditional dishes."

may express a closely related idea without containing the exact phrase Persian restaurant.

Exact matching is not inherently bad.

It is extremely useful when exact wording matters.

Examples include:

product codes
names
identifiers
quoted phrases
specific model numbers

Exact matching and semantic matching solve different problems.

6.4 Meaning Match

A meaning match occurs when two pieces of text correspond semantically even if their wording differs.

Query:
"affordable dentist for children"

Text:
"low-cost pediatric dental care"

There is little exact phrase overlap.

But there is substantial semantic correspondence:

affordable ↔ low-cost

dentist for children ↔ pediatric dental care
Different words
↓
Similar meaning
↓
Nearby or aligned representations

This is one of the central motivations for embeddings.

6.5 Exact Match vs Meaning Match

Concept Main question
Exact match Are the same words or strings present?
Meaning match Do the texts express similar meanings?

Consider:

Query:
"car mechanic near me"

Document A:

"Find a car mechanic near you."

This has strong lexical and semantic similarity.

Document B:

"Local automobile repair services."

This has weaker lexical overlap but potentially strong semantic similarity.

Document C:

"Car insurance prices near me."

This has noticeable lexical overlap but a different information need.

Word overlap ≠ meaning equivalence

Different wording ≠ different meaning

6.6 Semantic Similarity vs Relevance

This is perhaps the most important distinction in the chapter.

Semantic similarity is not the same as relevance.

Semantic similarity asks:

How similar are these pieces of text in meaning?

Relevance asks:

How useful or appropriate is this information for the specific information need?

Consider:

"How much does a private math tutor cost in Toronto?"

Document A:

"Private math tutors help students improve their mathematics skills through one-to-one lessons."

This document is semantically related to the query, but it does not answer the price question.

Document B:

"Private math tutoring in Toronto typically costs between X and Y per hour."

This is both semantically similar and directly relevant to the information need.

Semantic similarity
≠
Relevance

A text can be about the same subject and still fail to answer the question.

6.7 Why Similarity Is Not Relevance

Imagine the query:

"Is Restaurant A open on Sunday?"

Passage A

Restaurant A is a popular family-owned restaurant serving traditional dishes.

Passage B

Restaurant A is open from 12:00 to 22:00 on Sundays.

Both passages concern Restaurant A.

Passage A may therefore have meaningful semantic similarity to the query.

But Passage B directly addresses the requested information.

Relevance can depend on:

  • the user's specific intent;
  • the requested attribute;
  • whether the information actually answers the question;
  • the context of the request;
  • and other factors we will study later.

Similarity is evidence of semantic closeness, not proof of usefulness.

6.8 Topic Similarity Is Not Enough

Query:
"side effects of medication X"

A passage explaining:

"Medication X was approved in 2018."

is about the same entity and general topic.

But it does not address side effects.

Likewise:

Query:
"price of private English lessons"

Page:
"Benefits of private English lessons"

The page is topically related but may not satisfy the query.

Embedding similarity can help identify semantic relationships.

It does not magically solve every question of relevance.

7. A Complete Simplified Example

Suppose we have a query:

"affordable divorce lawyer"

and three passages:

A:
"Low-cost family law attorney for divorce cases."

B:
"History of divorce law in Europe."

C:
"Affordable wedding photography packages."

An embedding model transforms each item into a vector:

Query
↓
Embedding Model
↓
Query Vector

Passage A
↓
Embedding Model
↓
Passage Vector A

Passage B
↓
Embedding Model
↓
Passage Vector B

Passage C
↓
Embedding Model
↓
Passage Vector C

A similarity function can then compare:

Query Vector ↔ Passage Vector A
Query Vector ↔ Passage Vector B
Query Vector ↔ Passage Vector C

We might obtain illustrative scores such as:

A → 0.91
B → 0.63
C → 0.52

Passage A is strongly aligned with the query's meaning.

Passage B is related to divorce but not closely aligned with the request for an affordable lawyer.

Passage C shares the concept of affordability but concerns an unrelated service.

This illustrates why embeddings are useful:

They allow meaning-related comparison even when the exact words differ.

But it also illustrates their limitation:

A similarity score is not itself a complete relevance judgment.

8. Connecting the Concepts

8.1 From Text to Semantic Comparison

Chapter 1 gave us:

Text
↓
Tokenization
↓
Tokens
↓
Token IDs

Chapter 2 adds a new conceptual layer:

Text
↓
Numerical Semantic Representation
↓
Vector
↓
Embedding Space
↓
Similarity Comparison

These are different forms of numerical representation serving different purposes.

Token IDs allow the system to identify vocabulary units.

Embeddings allow relationships between represented objects to be expressed numerically.

Token ID
≠
Embedding

Token identification
≠
Semantic representation

8.2 Token IDs vs Embeddings

Suppose:

"doctor" → token ID 4812
"physician" → token ID 937

The IDs:

4812
937

do not tell us that the words are related.

Now imagine embeddings:

doctor    → [ ... ]
physician → [ ... ]
banana    → [ ... ]

The embedding space can be structured so that:

doctor ↔ physician

shows stronger semantic similarity than:

doctor ↔ banana
Representation Main purpose
Token ID Identify a token
Embedding Represent an object numerically so useful relationships can be compared

This is the conceptual bridge between Chapter 1 and Chapter 2.

8.3 An Embedding Is Not “The Meaning”

It is tempting to say:

“The embedding is the meaning of the sentence.”

That is too strong.

A better statement is:

An embedding is a learned numerical representation that captures aspects of the input useful to the model or task.

Human meaning is complex.

A sentence may contain:

  • literal meaning;
  • implied meaning;
  • emotional tone;
  • cultural references;
  • factual claims;
  • ambiguity;
  • context-dependent interpretation;
  • relationships between entities;
  • pragmatic intent.

No single vector should automatically be treated as a perfect mathematical container for everything a human could understand about a sentence.

Embeddings are powerful representations.

They are not identical to human understanding.

8.4 Similarity Is Model-Dependent

Suppose two embedding models process:

"Java"

and

"programming language"

One model may represent their relationship strongly.

Another may behave differently.

Likewise, context and model design can affect how:

"Java"

relates to:

"Indonesia"

The embedding is not an objective coordinate assigned by the universe.

It is produced by a particular model.

Text
+
Embedding Model
→
Embedding

not simply:

Text
→
one universal embedding

Different embedding models can create different representation spaces.

8.5 Why High-Dimensional Representation Helps

Why use hundreds or thousands of dimensions instead of two or three?

Human language contains many overlapping relationships.

A piece of text can simultaneously express information about:

topic
location
action
object
profession
purpose
style
intent
relationships

and many other patterns.

A tiny representation has limited capacity to encode complex relationships.

Higher-dimensional representations give a model more numerical structure with which to represent patterns.

However, this does not mean that each dimension corresponds neatly to one human concept.

Meaning is typically distributed across many dimensions.

Many dimensions
↓
Rich distributed numerical pattern
↓
Useful relationships in vector space

rather than:

One dimension
=
one concept

9. Why These Concepts Matter

9.2 Why This Matters for AI Visibility

Suppose a business page repeatedly contains the phrase:

"family law services"

A user may ask:

"Who can help me with a divorce?"

The user does not necessarily use the same words as the page.

A system capable of semantic representation can recognize relationships such as:

divorce
↕
family law

and:

who can help
↕
lawyer / attorney / legal service

This helps explain why modern information systems cannot be understood only through exact keyword matching.

However, we must not jump to the conclusion that semantically similar content will automatically be selected or cited by an AI system.

That would confuse similarity with relevance and would skip many later layers of the curriculum.

Meaning can be represented numerically in ways that make semantic comparison possible.

Selection, retrieval, ranking, evidence, citations, and AI Visibility involve additional mechanisms that come later.

10. Common Misunderstandings

Misunderstanding 1: Token IDs represent meaning

Incorrect.

Token ID → identifier
Embedding → learned numerical representation

A token ID's numerical value does not encode semantic distance.

Misunderstanding 2: Every embedding dimension has a clear human label

Usually incorrect.

Meaning is generally distributed across many dimensions.

Misunderstanding 3: Similar texts must contain the same words

Incorrect.

"cheap lawyer"
"affordable attorney"

can be semantically similar despite limited lexical overlap.

Misunderstanding 4: Shared words guarantee semantic similarity

Incorrect.

"Apple laptop repair"
"apple pie recipe"

share a lexical item but express very different meanings.

Misunderstanding 5: High similarity means high relevance

Incorrect.

A text can be semantically related to a query without satisfying its information need.

Misunderstanding 6: A similarity score is a probability

Incorrect.

A cosine similarity of 0.82 does not automatically mean an 82% probability of relevance, correctness, or semantic equivalence.

Misunderstanding 7: An embedding perfectly contains human meaning

Too strong.

An embedding is a learned numerical representation that captures useful patterns and relationships.

Misunderstanding 8: All embedding models produce the same semantic space

Incorrect.

Different models can produce different vector spaces and different relationships between inputs.

Misunderstanding 9: Semantic matching makes exact matching obsolete

Incorrect.

Exact matching remains extremely valuable when exact strings matter.

Semantic and lexical matching provide different kinds of information.

11. How the Concepts Connect

Start with text:

"affordable dentist for children"

An embedding model processes the text:

Text
↓
Embedding Model

The output is an embedding:

Embedding
↓
[0.18, -0.37, 0.62, ..., 0.09]

That embedding is a vector:

Vector
↓
Ordered numerical values

Each value occupies a dimension:

Dimension 1
Dimension 2
Dimension 3
...
Dimension n

Together, those dimensions define a point in a high-dimensional vector space:

Vector
↓
Point in Embedding Space

Another text can be processed by the compatible embedding model:

"low-cost pediatric dental care"
↓
Embedding
↓
Another Vector
↓
Another Point

Now the representations can be compared:

Vector A
↕
Similarity Function
↕
Vector B

One possible similarity function is cosine similarity.

Vector A + Vector B
↓
Cosine Similarity
↓
Similarity Score

That score expresses semantic closeness according to the model and comparison method.

Semantic Similarity
≠
Guaranteed Relevance

Semantic Similarity
≠
Lexical Similarity

Meaning Match
≠
Exact Match

This produces the full conceptual chain:

Text
↓
Embedding
↓
Vector
↓
Semantic Space
↓
Similarity

That chain is the foundation on which semantic retrieval will later be built.

Mental Model

Imagine a gigantic invisible map.

But instead of containing cities, the map contains numerical representations of language.

Every piece of text can become a point:

"divorce attorney"                  ●

"family lawyer for divorce"       ●


"legal help ending a marriage"     ●




                         ● "corporate tax accountant"



                                      ● "pizza restaurant"

The map is not actually two-dimensional.

It may have hundreds or thousands of dimensions.

The positions are not manually assigned.

An embedding model produces them.

Texts with related meanings can occupy related regions of this space.

Mathematical functions can compare their vectors and produce similarity scores.

Different wording
↓
Numerical representations
↓
Geometric comparison
↓
Detectable semantic relationship

But never forget the boundary of this mental model.

Being close on the map does not necessarily mean:

same words
same meaning
same intent
same answer
same relevance
same factual correctness

It means only that the representation system considers the vectors similar according to the properties captured by the model and the comparison method.

An embedding turns an object such as text into a point in a learned numerical space, allowing relationships between texts to be compared mathematically.

Chapter 2 Recap

Vector

An ordered list of numbers.

[0.2, -0.4, 0.7]
Vector Representation

The representation of an object using a vector.

Object → Vector
Dimension

One coordinate or position in a vector. A vector containing 768 values has 768 dimensions.

High-Dimensional Space

A mathematical space containing many dimensions—far more than humans can directly visualize.

Vector Space

A mathematical space in which vectors exist and can be compared.

Embedding

A numerical vector representation of an object designed so that useful relationships between objects can be expressed in the structure of the representation space.

Embedding Model

A model that transforms an input such as text into an embedding.

Text → Embedding Model → Vector
Text Embedding

An embedding representing a piece of text.

Query Embedding

An embedding representing an information request or query.

Document Embedding

An embedding representing a document.

Passage Embedding

An embedding representing a smaller section of a document.

Embedding Space

The vector space in which embeddings produced by an embedding model are represented.

Semantic Meaning

The meaning expressed by language beyond merely its exact surface wording.

Semantic Similarity

The degree to which two pieces of text are similar in meaning.

Semantic Distance

A representation of how separated two items are in semantic space.

Semantic Relatedness

The broader degree to which concepts are meaningfully associated, even when they do not have similar meanings.

doctor ↔ hospital
Cosine Similarity

A mathematical method for comparing vectors based on their directional alignment.

cos(θ) = (A · B) / (||A|| ||B||)

Similarity Score

A numerical value produced by a similarity function that expresses how similar two representations are according to that system.

It is not automatically a probability.

Lexical Similarity

Similarity based on words or textual forms.

Semantic Similarity

Similarity based on meaning.

Exact Match

Matching based on the presence of the same textual string or terms.

Meaning Match

Matching based on semantic correspondence even when wording differs.

Semantic Similarity vs Relevance

Semantic similarity measures closeness of meaning.

Relevance measures how well information satisfies a particular information need.

High Semantic Similarity
does not necessarily imply
High Relevance

The complete dependency developed across the first two chapters is now:

Text
↓
Tokenization
↓
Tokens
↓
Token IDs
↓
Numerical Representation
↓
Embeddings
↓
Vectors
↓
Embedding Space
↓
Semantic Relationships
↓
Similarity

We can now represent language numerically and compare those representations.

But an important question remains unresolved.

An embedding tells us where something is represented in a numerical space. It does not yet explain how a modern language model uses many such representations together to understand relationships between tokens inside a context.

That is the question that leads into the next chapter: Transformer Architecture.

The Single Most Important Idea from Chapter 2

Embeddings represent text as vectors in a learned numerical space, making semantic relationships mathematically comparable—even when the texts use different words.