Module 4 — Model Capabilities & Limitations

4.1 Hallucination: Why It Happens, How to Spot It

Learning objective: explain hallucination as a structural property of generation, and develop habits for catching it.

A model generates the next token based on learned statistical patterns, not by looking anything up (unless you’ve built retrieval in, per Module 3). When it doesn’t have reliable grounding for a fact, it doesn’t have an internal “confidence meter” it consults before answering — it produces the most plausible-sounding continuation regardless of whether that continuation is true. This is why hallucinations are often so convincing: they’re generated with the same fluency as correct answers, because fluency and correctness are produced by different parts of the process.

Practical habits that help:

  • Be most suspicious of specific, checkable claims — exact numbers, citations, quotes, API parameter names — since these are exactly the details a model is most likely to “fill in” plausibly rather than recall precisely.
  • Ask for sources, then actually verify them; a model can hallucinate a citation as fluently as it hallucinates a fact.
  • When correctness matters, prefer grounding the model in retrieved real documents (Module 3) over trusting parametric memory.

4.2 Bias and Training Data Effects

Learning objective: describe how training data composition shows up as bias in model output, and why this can’t be fully “prompted away.”

Models learn statistical patterns from their training data, and training data reflects the real world’s existing imbalances, historical patterns, and the specific sources it was drawn from. This surfaces as skewed assumptions in generated text (e.g., defaulting to a particular gender or nationality for an unspecified role), uneven quality across languages or dialects, and blind spots for topics underrepresented in training sources.

This matters practically, not just ethically: a hiring-assistant agent, a content moderation tool, or a customer-facing chatbot that inherits these patterns will produce systematically skewed behavior for some users. You can reduce the symptoms through system prompt instructions, few-shot examples, or output filtering, but you can’t fully eliminate the underlying tendency through prompting alone — it’s a property of the model, not just its instructions. Evaluation across diverse inputs (next lesson) is how you actually find where this shows up in your specific use case.

4.3 Basic Evaluation: How Do You Know If a Model Response Is “Good”?

Learning objective: describe basic approaches to evaluating model output quality beyond “it looked fine to me.”

“It looked fine when I tried it” is not evaluation — it’s one anecdote. Real evaluation means testing against a representative set of inputs, ideally including edge cases and adversarial ones, with some way to score output quality. Three common approaches, roughly in order of effort:

  1. Exact-match / rule-based checks — for tasks with a verifiable correct answer (does the extracted JSON have the right fields? does the code pass a test?). Cheap and reliable when applicable, but only applicable to a subset of tasks.
  2. Human review — a person reads outputs against a rubric. Gold standard for subjective quality, but slow and doesn’t scale to continuous monitoring.
  3. LLM-as-judge — using a separate model call to score outputs against criteria. Scales far better than human review, but the judge model has its own blind spots and biases, so treat its scores as a useful signal, not ground truth.

A minimal LLM-as-judge harness, scoring against an explicit rubric rather than a vague “is this good?”:

JUDGE_PROMPT = """You are grading an AI assistant's answer against a rubric.

Question: {question}
Answer to grade: {answer}

Rubric:
1. Is the answer factually grounded in the provided context (not invented)?
2. Does it directly address the question asked?
3. Is it free of unnecessary hedging or unnecessary length?

Respond as JSON: score_1
"""

def judge_response(question, answer, judge_llm_call):
    prompt = JUDGE_PROMPT.format(question=question, answer=answer)
    return judge_llm_call(prompt)  # returns parsed JSON per 2.4's structured-output lesson

Whatever method you use, build a small evaluation set before you need one — trying to evaluate a system only after something’s gone wrong in production means you’re improvising criteria under pressure.

4.4 Cost and Latency Tradeoffs Across Model Sizes

Learning objective: describe why larger/more capable models aren’t automatically the right choice, and how to reason about the cost/latency/quality tradeoff.

Model providers typically offer a range of models trading off capability against cost and speed — a small, fast, cheap model for straightforward tasks, and a larger, slower, more expensive one for tasks requiring deeper reasoning. Reaching for the most capable model by default is a common and expensive mistake: many tasks (classification, simple extraction, short rewrites) are handled just as well by a smaller model at a fraction of the cost and latency.

A practical approach is model tiering: route easy, high-volume requests to a cheap/fast model, and reserve the expensive model for requests that genuinely need its extra reasoning capability.

flowchart LR
    A["Incoming request"] --> B{"Simple / high-volume\ntask?"}
    B -->|yes| C["Small/fast model"]
    C --> D{"Confident\noutput?"}
    D -->|yes| E["Return result"]
    D -->|no / failed check| F["Escalate to\nlarger model"]
    B -->|no| F
    F --> E

This becomes directly relevant in Track 2 when agents make many model calls per task and cost/latency compound quickly.

Exercise

Pick 5 factual questions, at least 2 of which touch on obscure or highly specific details (exact statistics, niche technical parameters). Ask a model each question, and manually fact-check every claim against a reliable source. Document which answers were confidently wrong rather than hedged, and reflect on which “tells” (over-specificity, unusual confidence) might help you catch this in the future.

Quiz

  1. Why is fluency not evidence of correctness in model output?
  2. Give one way bias shows up in model output that a system prompt instruction can’t fully fix.
  3. Name the three evaluation approaches from this module and one tradeoff of each.
  4. Why might routing all requests to the most capable model be the wrong default?

Answer key: TODO — write once the module has been through at least one test learner.

← Back to Track 1 · Next: Capstone — Build a Simple RAG Chatbot →