How large language models actually work: tokens, context windows, hallucinations

Every AI tool you will use in this course, from the Gemini app to Claude Code to Codex, sits on top of a large language model (LLM). If you understand the handful of mechanics underneath, the odd behaviors stop being mysterious. You will know why a model confidently states a wrong date, why a long chat "forgets" what you said an hour ago, and why the same prompt gives a slightly different answer twice.

By the end of this lesson you will be able to explain tokens, context windows and hallucinations in plain English, and you will have run two small experiments that make each idea concrete. That understanding is the foundation for every prompting, agent and safety technique in the rest of the course.

What a model actually does: predict the next token

A model does not read letters or whole words. It reads tokens, which are chunks of text. In English a token is roughly three quarters of a word, so 1,000 tokens is about 750 words. Common words like "the" are one token. Rare words, code, and non-English text split into more pieces. A Hebrew or Japanese sentence usually costs more tokens than the same sentence in English.

Training is the process of showing the model trillions of tokens and adjusting billions of internal numbers (weights) so that its next-token guesses improve. After training, the weights are frozen. When you chat, the model runs a loop: look at everything in the conversation so far, produce a probability for every possible next token, pick one, append it, repeat. That is the whole trick.

Two consequences follow directly:

• Sampling adds randomness. The model usually does not pick the single most likely token every time. A setting called temperature controls how adventurous the choice is. This is why you get different wording on a second run.
• Everything is text in, text out. Instructions, examples, documents and your question are all just tokens in the same stream. The model has no separate "rules" channel. This matters later for prompt injection.

Modern chat models add a second stage after pre-training. They are tuned with human feedback so that the "most likely continuation" of a question is a helpful, safe answer rather than a random forum reply. Reasoning modes (ChatGPT thinking modes, Gemini Pro, Claude extended thinking) let the model generate many hidden tokens of working-out before the visible answer. That improves accuracy on hard problems but costs more tokens and time.

The context window: the model's working memory

The context window is the maximum number of tokens the model can look at in one go. It includes the system instructions, every message in the chat, any files you attached, and the answer being written. Current frontier models offer windows from a few hundred thousand tokens up to a million or more. Check the model's documentation for the exact figure, because it changes by tier.

Three practical rules come from this:

1. The model has no memory between chats unless the product adds one. ChatGPT memory, Claude memory and Gemini's personalization are features layered on top. They inject saved notes into the context; the raw model still starts from zero.
2. Long chats degrade. Even inside the window, models pay less reliable attention to material buried in the middle of a huge context. Important instructions should sit near the start or be repeated near the end.
3. Once you overflow, something gets dropped. Products either refuse, silently truncate, or summarize older turns. Claude Code's /compact command exists precisely to summarize a long session so you can keep working.

These are estimates. Every provider has a free tokenizer page or API call you can use to count exactly.

Why models hallucinate

A hallucination is fluent, confident output that is false: a fake citation, a product feature that does not exist, a legal rule from the wrong country. It is not a bug that will be patched away. It is the natural result of a system whose job is to produce plausible text. When the true answer is not well represented in training data, the most plausible-sounding continuation is still produced.

Hallucinations are most likely when you ask about:

• Specific numbers, dates, prices, statistics and version numbers.
• Citations, URLs, case names, ISBNs and quotes.
• Recent events after the model's training cutoff.
• Niche topics, small companies, private individuals.
• Anything where you have pushed the model into agreeing with a premise that is wrong.

The fixes are practical, not magical. Give the model the source material so it can quote rather than recall (this is what NotebookLM, Claude Projects and ChatGPT file uploads do). Turn on web search or Deep Research for current facts. Ask the model to say "I don't know" when unsure. And verify anything that goes to a customer, a court or a bank.

A useful mental model for choosing settings

You rarely set temperature in a chat app, but the idea still helps. For brainstorming, marketing copy and names, you want variety: run the prompt several times and pick. For extraction, classification, code and anything factual, you want consistency: use a reasoning mode, give sources, and ask for a fixed output format. In the API (covered in Module 10) you set temperature directly; for factual tasks keep it low.

Try it now (10 minutes)

1. Open the Gemini app, ChatGPT or Claude. Paste this prompt and note how the model handles a detail it cannot know:

   I am going to describe a fictional company. Answer only from what I give you. If something is not stated, reply "not stated".
   
   Company: Northwind Bakery, founded 2019 in Leeds, 4 staff, sells sourdough and cinnamon rolls.
   
   Questions:
   1. What year was it founded?
   2. What is its annual revenue?
   3. Who is the founder?
   4. What products does it sell?

   A well-behaved model answers 2 and 3 with "not stated". If it invents a number or a name, you have just watched a hallucination.
2. Now test the context window. Start a new chat and paste this:

   Remember this code word: PELICAN-42. Later I will ask for it. For now, summarise the history of the bicycle in 300 words.

   Then paste four or five long articles (anything you have handy) and ask for a summary of each. Finally ask: "What was the code word?" Most modern models will recall it, but note how long it took and whether the answer became less precise. Repeat with a chat that is already very long and compare.
3. Find your provider's token counter (search the docs for "tokenizer" or "count tokens"). Paste an email and a page of your own writing. Write down the ratio of words to tokens you observe.

How this helps your business

• Fewer embarrassing errors. Knowing that numbers, citations and prices are the highest-risk outputs lets you build a two-minute verification habit before anything reaches a client.
• Lower AI bills. API pricing is per token. Trimming context, using a smaller model for simple tasks, and reserving reasoning modes for hard questions can cut costs sharply.
• Better tool choices. When a task involves a large document set, you will reach for NotebookLM or a Project instead of pasting into a chat and hoping.
• Clearer staff guidance. You can explain to a team in one sentence why "the AI said so" is not a source, which reduces risk without banning the tools.

Common mistakes

• Treating the model as a search engine and accepting statistics, quotes or URLs without checking them.
• Keeping one enormous chat open for weeks and wondering why quality drops.
• Assuming the model remembers your previous conversation when the product's memory feature is off.
• Asking about events after the training cutoff without enabling web search.
• Judging a model on one run. Sampling means you should try the prompt two or three times before concluding it "can't do it".