Chapter 1 - From Text to Model Input

1. Text

Text is written language represented as characters.

Examples:

Hello

Persian restaurant

What is AI Visibility?

At this stage, we are talking about the text exactly as a user might type it.

Text can contain:

• letters
• numbers
• spaces
• punctuation
• symbols
• emojis
• different languages

For example:

Toronto restaurant — open until 10:30 PM 🍽️

This contains letters, spaces, punctuation, numbers, and an emoji.

To us, this is one sentence.

To a computer, however, the sentence must eventually be converted into numbers.

2. Character

A character is one individual written symbol.

For example, the word:

cat

contains three characters:

c
a
t

The text:

AI!

contains three characters:

A
I
!

Spaces also matter.

For example:

AI Search

contains:

A
I
space
S
e
a
r
c
h

Characters are the smallest written units we need to discuss at this stage.

But modern language systems usually do not process language one character at a time.

Instead, they usually work with larger units called tokens.

Before reaching tokens, however, we need to understand words and vocabularies.

3. Word

A word is a linguistic unit that humans naturally recognize.

Examples:

• restaurant
• lawyer
• Toronto
• search

A sentence can therefore be viewed as a sequence of words.

For example:

I need a restaurant.

can be separated approximately into:

I
need
a
restaurant

But there is a problem.

Language is much more complicated than simply splitting text at spaces.

Consider:

don't

Should this be one unit?

Or:

do + n't

What about:

• AI-powered
• GPT-5
• ostadbank.com

Or Persian text where word boundaries and written forms behave differently from English?

Because of these complications, AI systems generally do not simply treat every space-separated word as one unit.

This is one reason tokenization exists.

4. Vocabulary

A vocabulary is the collection of text units that a particular tokenizer knows how to represent directly.

Do not think of vocabulary here exactly like a human dictionary.

A tokenizer's vocabulary can contain:

• complete words
• parts of words
• punctuation
• symbols
• common character sequences
• sometimes combinations involving spaces

For example, an imaginary tokenizer vocabulary might contain units such as:

the
restaurant
ing
un
Toronto
.

Again, this is only an illustration. Real tokenizer vocabularies are much larger and their exact contents depend on the system.

A vocabulary allows the tokenizer to convert text into known units.

Those units are called tokens.

5. Token

A token is a unit of text that an AI language system processes.

A token is not necessarily a word.

This is one of the most important ideas in this chapter.

A token can represent:

• an entire word
• part of a word
• punctuation
• a number
• a symbol
• sometimes a combination involving spaces

For example, imagine the word:

unbelievable

A tokenizer might represent it approximately as:

un
believ
able

That would mean one human word became three tokens.

Another tokenizer might divide the same word differently.

So:

Word ≠ Token

A word may correspond to:

• one token
• several tokens

And sometimes a token may contain more than what we would normally think of as one simple word fragment.

6. Why Not Just Use Words?

You might reasonably ask:

Why not simply make every word a token?

The main problem is that languages contain an enormous number of possible words and word forms.

Consider English:

help
helps
helped
helping
helpful
unhelpful

If every possible word needed its own separate vocabulary entry, the vocabulary would become very large.

Now consider:

• names
• technical terms
• new brands
• URLs
• spelling variations
• multiple languages

A system will constantly encounter words that were uncommon or nonexistent when its tokenizer was designed.

Breaking words into smaller reusable pieces helps solve this problem.

7. Subword Tokenization

Modern systems commonly use subword tokenization.

A subword is a piece of a word.

Instead of requiring every possible word to exist in the vocabulary, the tokenizer can construct uncommon words from smaller pieces.

Imagine that the vocabulary contains:

AI
vis
ibility

Then:

AI visibility

might be represented using those known pieces.

Again, the exact split is only illustrative.

The principle is what matters:

Common patterns can have their own tokens, while unusual words can be represented using combinations of smaller tokens.

This gives the system flexibility.

8. Common Words vs. Rare Words

Frequently occurring text patterns are often represented efficiently.

Unusual text may require more tokens.

For example, a common English word might correspond to a single token.

An unusual:

• surname
• company name
• scientific term
• foreign-language expression
• strange URL

might be divided into several tokens.

This gives us an important idea:

Different strings of similar human length can require very different numbers of tokens.

For example:

restaurant

XQZRestaurant2027

may look similar in length to us, but the second one may require more tokens because its internal patterns are less common.

The exact result depends on the tokenizer being used.

9. Tokenization

Tokenization is the process of converting text into tokens.

Suppose the input is:

I like Persian food.

A purely illustrative tokenizer might produce:

I
 like
 Persian
 food
.

Another tokenizer could split the same sentence differently.

Therefore, there is no universal tokenization that every AI system must use.

Different systems can use different tokenizers.

The important process is:

Raw Text → Tokenization → Tokens

10. Tokenizer

A tokenizer is the component that performs tokenization.

Its job is to take text and convert it into the token units used by the system.

Conceptually:

I need a Persian restaurant.

goes into the tokenizer.

The tokenizer returns something like:

[token 1]
[token 2]
[token 3]
[token 4]
...

The tokenizer is not yet doing the kind of deep language understanding that we will study later.

At this stage, it is primarily converting text into a representation that the language system can process.

11. Token ID

The computer does not ultimately process the written token itself.

Each token in the vocabulary is assigned a number.

That number is called a token ID.

Imagine a very small fictional vocabulary:

If the tokenizer sees:

restaurant

it might convert it to:

826

So the process becomes:

Text → Tokens → Token IDs

For example:

Toronto restaurant.

might become something conceptually like:

[1942, 826, 15]

These numbers are arbitrary identifiers.

The fact that:

Toronto = 1942

would not mean Toronto has a numerical meaning of 1942.

It is simply an ID used to identify that token.

12. An Important Distinction: ID Is Not Meaning

This distinction will become very important in later chapters.

A token ID is simply an identifier.

Suppose:

cat = 271
dog = 893

You should not conclude:

Dog is mathematically larger than cat because 893 > 271.

The numbers themselves do not encode semantic meaning.

They are more like item numbers in a catalog.

For example:

Product #893

does not mean that product is somehow “greater” than Product #271.

Later, we will learn how the system converts these token IDs into numerical representations that actually contain useful information about relationships and meaning.

That will be the subject of the next chapter.

For now:

Token ID = identifier, not meaning.

13. Vocabulary Size

Vocabulary size means the number of distinct tokens available in a tokenizer's vocabulary.

Imagine a fictional tokenizer with:

50,000 tokens

Its vocabulary therefore contains 50,000 different token units.

A larger vocabulary can directly represent more text patterns.

But larger is not automatically better.

There are trade-offs involving:

• vocabulary size
• token efficiency
• language coverage
• computational design

We do not need to explore those engineering trade-offs yet.

For our purposes, remember:

The tokenizer has a predefined collection of token units, and each unit has a token ID.

14. Rare Tokens and Rare Text

An important practical consequence of tokenization is that uncommon language can sometimes require more tokens.

Imagine a familiar term:

restaurant

versus a fictional company name:

Qzervanix

A tokenizer may already contain a convenient token for “restaurant”.

But “Qzervanix” might need to be broken into several smaller pieces.

Something like:

Q
zer
van
ix

Again, this is illustrative rather than an actual tokenizer output.

This matters because unusual:

• brand names
• personal names
• URLs
• product codes
• technical terminology

can sometimes be represented less efficiently.

This does not mean the AI cannot understand an unusual brand.

It simply means that the text representation may use more tokens.

15. Token Count

The token count is the number of tokens produced from a piece of text.

Suppose:

I love Toronto.

becomes four tokens.

Its token count is:

4

A longer article might contain thousands of tokens.

Token count matters because AI systems have limits on how many tokens they can work with at one time.

That brings us to our next major concept:

context.

16. Context

In this course, context means the information currently available to the AI system while it is producing a response.

Suppose you ask:

What is the capital of Canada?

Your question becomes part of the context.

If you had previously said:

I'm preparing a geography quiz.

that earlier information may also be part of the context.

If the system has been given an external document, that document may also be part of the context.

So conceptually:

Context = information currently available to the system for this interaction

Later in the course, we will discover that context can include many different sources of information.

For now, we only need the basic idea.

17. Context Is Not the Same as Permanent Knowledge

This distinction is extremely important.

Imagine that you give the AI this sentence:

My fictional company, VeloraX, was founded in 2026.

The AI can use that information later in the same conversation if it remains available in its context.

For example:

When was VeloraX founded?

It can answer:

2026.

That does not necessarily mean the information has become part of the AI's permanent knowledge.

It may simply be available in the current context.

So:

Information in context ≠ information permanently learned by the system

We will study training and permanent learned knowledge later.

18. Context Window, Context Limit, and Context Length

These terms are closely related and are sometimes used inconsistently in technical writing. For this course, we will use them in a precise way.

Context Window

The context window is the total working space available to the model for one processing context.

Think of it as the container.

Context Limit

The context limit is the maximum size of that context window, measured in tokens.

If a model has a 128K context window, its context limit is approximately 128,000 tokens.

Context Length

In this course, context length will usually mean how many tokens are currently being used inside the context.

For example:

Context Window / Limit = 128K
Current Context Length = 32K

So the model can support up to about 128K tokens, but the current interaction is using only 32K.

One complication is that some articles use “context length” to mean the maximum supported length.

When you see the term outside this course, you must infer the intended meaning from context.

19. A Simple Analogy

Imagine a study desk.

• Context Window = the desk itself.
• Context Limit = the maximum amount that can fit on the desk.
• Context Length = how much is currently sitting on the desk.

This analogy is simple, but it is useful because it separates the container, the maximum capacity, and the current usage.

20. Input Context

The input context is the information provided to the system before it generates the next part of its response.

For example:

Explain tokenization.

is part of the input.

But the input may contain more than the visible user message.

Conceptually, the system may receive:

Instructions
+
Conversation
+
User Question
+
Additional Information

All of these can contribute to the input context.

We will study the different types of instructions and external information in later chapters.

21. Output Tokens

The system's generated response is also composed of tokens.

These are called output tokens.

Suppose you ask:

Explain AI in one paragraph.

Your question contributes input tokens.

The answer contributes output tokens.

Conceptually:

Input Tokens → Processing → Output Tokens

So when people discuss the token usage of an AI interaction, they may distinguish between:

• input tokens
• output tokens

22. Input vs. Output Tokens

Consider this simplified interaction:

Input

Explain tokenization.

Suppose that becomes:

4 tokens

Output

The AI generates a 300-token explanation.

Then:

Input tokens = 4
Output tokens = 300

The total processing involves both.

Real systems may also include other contextual information that is not visible in this simple example.

But the distinction is important:

Tokens do not only describe what the user writes. They also describe what the system generates.

23. Token Budget

A token budget is the amount of token capacity allocated to a particular purpose.

This is different from the context window itself.

Suppose a system has a context limit of 128K tokens. The application might decide to allocate that capacity approximately like this:

These numbers are purely illustrative.

The key distinction is:

Context Window = total available container

Context Limit = maximum size of that container

Context Length = how much is currently being used

Token Budget = how much capacity we allocate to a specific purpose

24. Context Allocation

Context allocation means deciding how the available context capacity is used.

Imagine a context window as a container.

Inside it we might want to place:

• instructions
• user messages
• previous messages
• useful source material
• room for the answer

If one component uses more space, less space may remain for another component.

This is a resource-allocation problem.

The exact architecture varies between systems.

The key idea is that available context is finite and different pieces of information compete for it.

25. Information Competition Inside the Context Window

This leads to a subtle but important principle:

More context is not automatically better context.

Imagine you ask:

What is the refund policy?

Now imagine the system receives 1,000 pages of company documents.

Only two paragraphs contain the refund policy.

The other pages discuss:

• hiring
• marketing
• company history
• technical documentation
• office rules

Giving the AI more text does not automatically make the answer better.

The important question becomes:

Which information should be placed into the context?

This idea will later lead us directly to Information Retrieval.

But we are not ready to study retrieval yet.

For now, just remember:

Limited context creates a selection problem.

26. Long Context

A system with a long context can process a relatively large number of tokens within one context window.

This can be useful for:

• long documents
• long conversations
• multiple sources
• large codebases
• research material

However:

Longer context ≠ perfect use of all information inside that context.

Simply placing information somewhere inside a very large context does not guarantee that every piece of information will be used equally well.

Later, we will distinguish between:

information being available

and

information being relevant and selected effectively.

That distinction becomes fundamental in search and retrieval.

27. Why Tokenization Matters for Long Context

Suppose two documents contain approximately the same number of human-readable characters.

Document A may tokenize efficiently.

Document B may contain:

• unusual terminology
• code
• URLs
• uncommon names
• mixed languages

and therefore require more tokens.

As a result, Document B may consume more of the context budget.

So the pipeline matters:

Text → Tokenizer → Token Count → Context Usage

This is why tokenization is not merely a technical detail.

It directly affects how much information can fit into the system's working context.

28. Why This Matters for Different Languages

Token efficiency can vary across languages.

A tokenizer may represent some languages more efficiently than others depending on how its vocabulary was constructed.

Therefore, two sentences with similar meaning and human-readable length may produce different token counts.

This is important when working with:

• English
• Persian
• Arabic
• Chinese
• multilingual documents

We do not need to study multilingual tokenizer engineering in this course.

The important principle is:

Human text length and token length are not identical measurements.

29. Why This Matters for Names and Brands

Suppose an AI system encounters:

Apple

That is an extremely common string.

Now compare it with a newly created brand:

Zerviqano

An unusual brand may be split into more pieces.

Again, this does not automatically create an AI Visibility problem.

But it teaches us an important rule:

The representation used by the AI system is not always identical to the linguistic units humans naturally perceive.

A human may immediately see one brand name.

The tokenizer may see several tokens.

Later, when we study entities, we will learn how an AI system can still treat multiple tokens as referring to one real-world thing.

But that concept belongs to a later chapter.

30. The Complete Pipeline So Far

We can now construct the first complete pipeline in this course.

Suppose the user writes:

Find a Persian restaurant in Toronto.

Step 1 — Raw Text

The system receives:

Find a Persian restaurant in Toronto.

Step 2 — Tokenization

The tokenizer divides the text into tokens.

Conceptually:

Find
a
Persian
restaurant
in
Toronto
.

The real tokenization may differ.

Step 3 — Token IDs

Each token maps to a numerical ID.

Conceptually:

[521, 42, 8912, 1734, 67, 4421, 13]

These numbers are fictional.

Step 4 — Input Context

Those token IDs become part of the information available to the system.

Step 5 — Context Window

The system can process them because they fit within its available context window.

We have therefore moved from:

Human-readable text

to:

machine-processable input identifiers

We have not yet explained how the system understands the meaning of “Persian restaurant”.

That is the next problem.

31. What We Have Not Explained Yet

At this point, an important question should occur to you.

Suppose:

restaurant = token ID 1734
cafe = token ID 9051

How does the system know that:

restaurant

and

cafe

are related concepts?

The token IDs themselves do not tell us that.

Remember:

1734
9051

are just identifiers.

They do not encode meaning by themselves.

So we now need another mechanism that can represent things such as:

• similarity
• relationships
• meaning

That is where vectors and embeddings enter the picture.

They are the subject of Chapter 2.

32. Mental Model

Keep this model in your head:

Human Side

Text

Find a Persian restaurant.

↓

Tokenizer

The text is divided into:

Tokens

↓

Each token is mapped to:

Token IDs

↓

Those IDs become part of:

Input Context

↓

The amount of information that can be considered is constrained by:

Context Window

This gives us:

Text → Tokenization → Tokens → Token IDs → Input Context → Context Window

There is still one major missing step:

How do token IDs become useful representations of meaning?

That is what we will solve next.

Chapter 1 Recap

You should now be comfortable with these concepts:

Text

Human-readable written information.

Character

An individual written symbol.

Word

A linguistic unit recognized naturally by humans.

Vocabulary

The set of token units available to a tokenizer.

Token

A unit of text processed by an AI language system.

Tokenization

The process of converting text into tokens.

Tokenizer

The component that performs tokenization.

Token ID

A numerical identifier assigned to a token.

Subword Tokenization

Representing words using smaller reusable word pieces.

Vocabulary Size

The number of distinct token units available.

Rare Text

Uncommon strings may require multiple smaller tokens.

Token Count

The number of tokens used to represent a piece of text.

Context

Information currently available to the system during an interaction.

Context Window

The total working space available to the model for a processing context.

Context Limit

The maximum size of the context window.

Context Length

How many tokens are currently being used in the context, in the terminology used in this course.

Input Context

Information provided before the next output is generated.

Output Tokens

Tokens produced by the system in its response.

Token Budget

The amount of token capacity allocated to a particular purpose.

Context Allocation

How available context capacity is distributed among different kinds of information.

Long Context

The ability to work with relatively large amounts of tokenized information.

The Single Most Important Idea from Chapter 1

An AI language system does not directly receive:

words and meaning.

It begins with a transformation:

Human Text → Tokens → Token IDs

and those tokens must fit within a limited:

Context Window

The next question is therefore:

How can numerical representations capture meaning and relationships between these tokens?

That is the starting point of Chapter 2 — Numerical Representation of Meaning.