1.2 — Context Windows

Learning objective: explain what a context window is and how it constrains what a model “remembers” in a single conversation.

The context window is the maximum number of tokens a model can process in a single request — every system prompt, every prior message, every tool result, and the model’s own output all count against this same budget. Modern models range from tens of thousands to over a million tokens of context, but every window is finite.

Everything shares one budget

A useful mental model is a single container that every part of a request has to fit inside together:

pie showData
    title Example context window budget (100K tokens)
    "System prompt" : 2
    "Conversation history" : 35
    "Tool/RAG results" : 25
    "Reserved for model's response" : 20
    "Unused headroom" : 18

Note that the model’s own response comes out of the same budget — if you don’t reserve headroom for it, a request can fail simply because there’s no room left for an answer, even if the “conversation so far” fit comfortably.

Two things trip people up early on

  1. The context window is not memory across conversations. Unless a system explicitly stores and re-sends prior conversation history (or writes to some external store), the model has no idea what happened in a session that ended. “Memory” in agent products is almost always an engineering layer bolted on top of the model, not a native capability of the model itself.
  2. A full context window degrades performance before it errors out. Long before you hit the hard token limit, stuffing a context window with irrelevant information tends to dilute the model’s attention and produce worse answers — this is often called the “lost in the middle” effect. More context is not free; it has a real quality cost, not just a dollar cost.

This is exactly why Module 3’s lesson on RAG matters: instead of stuffing everything into context, you retrieve only what’s relevant for a given query.

Working through a token budget in code

def estimate_request_tokens(system_prompt_tokens, history_tokens,
                             tool_result_tokens, max_response_tokens,
                             context_window=100_000):
    used = system_prompt_tokens + history_tokens + tool_result_tokens
    headroom = context_window - used - max_response_tokens
    if headroom < 0:
        raise ValueError(
            f"Over budget by {-headroom} tokens — trim history or "
            f"tool results before calling the model."
        )
    return {"used": used, "reserved_for_response": max_response_tokens,
            "headroom": headroom}

estimate_request_tokens(
    system_prompt_tokens=500,
    history_tokens=1_600,      # 20 turns * 80 tokens
    tool_result_tokens=3_000,  # 3 calls * 1,000 tokens
    max_response_tokens=1_000,
)
# => {"used": 5100, "reserved_for_response": 1000, "headroom": 93900}

Even when the math clears the hard limit comfortably, that’s a check for “will this request work at all” — it says nothing about whether 5,100 tokens of history and tool results are the right 5,100 tokens for the model to be looking at. That’s a quality question, not a budget question, and it’s the one “lost in the middle” points at.

Exercise

Estimate the token budget for a hypothetical customer-support agent: a 500-token system prompt, a 20-turn conversation averaging 80 tokens per turn, and 3 tool calls returning 1,000 tokens each. How many tokens is that before the model even produces a new answer? What would you cut first if you needed to save space?

← Tokens & Tokenization · Back to Module 1 · Next: Sampling & Temperature →