1.1 Tokens & Tokenization
1.1 — Tokens & Tokenization
Learning objective: explain what a token is and why models think in tokens, not words or characters.
When you type a message to an AI model, the first thing that happens has nothing to do with meaning — it’s mechanical. Your text gets broken into tokens, the actual units the model reads and generates. A token is often a whole word (“cat”), sometimes a word fragment (“token” → “tok” + “en”), and sometimes a single character or punctuation mark. As a rough rule of thumb, 1 token ≈ 4 characters of English text, or about ¾ of a word.
What tokenization actually looks like
Most current models use a form of byte-pair encoding (BPE) or a close relative: a fixed vocabulary of tens of thousands of common substrings, built by starting from individual characters and repeatedly merging the most frequent adjacent pair until the vocabulary reaches its target size. Common whole words end up as single tokens; rare words end up split into recognizable pieces.
Here’s roughly what tokenizing a sentence looks like (illustrative — exact splits vary by tokenizer):
Input: "Tokenization isn't the same as counting words."
Tokens: ["Token", "ization", " isn", "'t", " the", " same", " as",
" counting", " words", "."]
Token count: 10 (word count: 7)
Notice three things in that example: the split doesn’t respect word boundaries cleanly (“Token” + “ization”), whitespace is often folded into the following token rather than treated separately, and punctuation gets its own token. This is why “1 token ≈ ¾ of a word” is an average, not a rule — a sentence full of uncommon words or code identifiers will tokenize far less efficiently than a sentence of common words.
Why does this matter to you as a builder?
- Cost: nearly every commercial model API charges per token, input and output combined. A prompt that “feels short” as a sentence can be expensive if it’s dense with rare tokens (unusual names, code, non-English text tokenize less efficiently).
- Context limits: every model has a maximum number of tokens it can hold in one request (covered in the next lesson). If you don’t think in tokens, you’ll blow through this limit without understanding why.
- Weird failure modes: models are sometimes bad at character-level tasks (like counting letters in a word, or reversing a string) precisely because they don’t see individual characters — they see tokens. This explains a class of “dumb mistake” that otherwise looks inexplicable. Asking a model “how many letter R’s are in ‘strawberry’” is, from the model’s perspective, closer to asking a human to count letters in a word shown only as a wax seal — the shape is recognizable, the internal letters aren’t directly visible as separate units.
A quick, concrete cost estimate
# Illustrative only — real pricing varies by provider and model tier.
INPUT_COST_PER_1K_TOKENS = 0.003 # example: $3 / million input tokens
OUTPUT_COST_PER_1K_TOKENS = 0.015 # example: $15 / million output tokens
def estimate_cost(input_tokens: int, output_tokens: int) -> float:
return (
(input_tokens / 1000) * INPUT_COST_PER_1K_TOKENS
+ (output_tokens / 1000) * OUTPUT_COST_PER_1K_TOKENS
)
# A support chatbot answering ~500 questions/day, ~800 input + 300 output
# tokens per exchange:
daily_cost = estimate_cost(800, 300) * 500
print(f"${daily_cost:.2f} / day") # illustrative order-of-magnitude figure
The exact numbers will be wrong for whatever model you actually use — the point of the exercise is the shape of the calculation: cost is a function of tokens in and tokens out, separately, and output tokens are usually priced higher than input tokens across most providers.
Different model families use different tokenizers, so the same text can split into a different number of tokens depending on the model. This is why token counts aren’t perfectly portable between providers.
Exercise
Take a paragraph of your own writing and a paragraph of code. Run both through a public tokenizer visualizer (e.g., a model provider’s tokenizer tool). Count tokens for each. Which one has a higher tokens-per-character ratio, and why do you think that is?