Inside the Next-Token Machine

An engineer who treats a large language model as a pure black box can still ship a working feature. The trouble starts the moment that feature needs to be debugged, optimized, or explained to a skeptical stakeholder. Without a mental model of what is happening between the prompt and the response, every unexpected behavior looks equally mysterious, whether it is a cost spike, a factual error, or an inconsistent tone across runs. With a model of the mechanism, these problems sort themselves into distinct categories, each with its own likely cause and its own fix, rather than collapsing into a single vague complaint that the AI is being weird.

Consider a concrete scenario. A support-ticket summarizer that worked well in testing starts truncating its output mid-sentence in production, but only for a subset of customers. An engineer without a model of the system might suspect a bug in the application code first, and spend hours there. An engineer who understands the context window, the fixed budget of tokens a model can process in a single call, will instead check whether those particular tickets happen to be long threads that push the combined prompt and expected output past that budget. The fix, once identified, might be as simple as truncating older messages or summarizing in stages, but finding it quickly depends entirely on knowing the mechanism exists.

The same logic applies to cost. A team that assumes tokens roughly track words will budget incorrectly, because the tokenizer that turns text into tokens does not respect word boundaries the way a human reader does. A technical term, a rare name, or a string of digits can expand into several tokens where an everyday English word might collapse into one. Multiply that effect across millions of API calls and a mismatch between assumed and actual token counts becomes a real line item on an invoice, not a rounding error. The chapter on tokenization later in this book works through exactly how that happens and what to do about it.

Mental models also change what a team can evaluate before committing to a vendor or an architecture. A blog post announcing a new model release will typically cite its training compute, its parameter count, or a scaling comparison against a well-known baseline. Without any grounding in what a scaling law actually claims, those numbers read as marketing. With that grounding, an engineer can ask sharper questions: does a lower loss number on a benchmark actually predict better performance on this specific application, or does it only reflect improvement on the kind of text the benchmark was built from? That skepticism is a professional skill, not cynicism, and it only develops once the underlying mechanism is no longer a mystery.

There is also a risk management angle that matters especially for anything resembling production software. A model that occasionally invents facts, a behavior covered in detail in the chapter on hallucination, is not a defect to be patched away with a clever prompt, in the way a software bug can usually be patched away with a code change. It is closer to a structural property of how these systems generate text at all, which means it has to be designed around: validation layers, retrieval grounding, human review for high-stakes outputs. Engineers who understand why hallucination happens make calmer, better calibrated decisions about where to add those safeguards, instead of either over-trusting the model or distrusting it so much that it becomes useless.

Finally, a mental model compounds in value over time in a way that tricks and workarounds do not. A specific prompting trick that works around a specific model's quirk may stop working when that model gets replaced, which happens on a fast cycle in this field. An understanding of self-attention, embedding geometry, or the tradeoffs in decoding choices like temperature and top-p transfers across model versions and even across vendors, because those mechanisms are shared architecture, not implementation details specific to one product. That transferability is the real argument for spending the time on fundamentals rather than skimming documentation for the next quick fix.

A black-box API call can produce a working demo. Only a working mental model of the mechanism underneath it can produce a system you trust enough to put in front of real users.

None of this argues for becoming a machine learning researcher. The depth aimed at here is the depth an engineer needs to read a paper's abstract and methods section without getting lost, to ask a vendor a pointed question in a sales call, and to reason from first principles about why a system misbehaves instead of guessing. That is a specific, bounded, and achievable goal, and it is the one this book is built around from its first chapter to its last.

It also changes how a team communicates internally. A product manager who has never heard of a token count will not understand why adding a few extra sentences of instructions to every prompt noticeably raises the monthly bill. An engineer who can explain that relationship in plain terms, without resorting to jargon, becomes the person who translates between the technical reality of the system and the business decisions being made about it. That translation role is valuable precisely because so few people in most organizations can currently do it well.

The chapters that follow are sequenced so that each mechanism builds on the last, which mirrors how the system itself is actually constructed. Starting with next-token prediction establishes the single idea that everything else elaborates on. Moving through tokenization, embeddings, and attention builds the representation side of the model. Pretraining, scaling, and fine-tuning cover how a model gets built and shaped. Inference and hallucination cover what happens once that model is actually deployed and answering real prompts. Skipping around is possible, but the ordering is intentional and rewards a mostly linear read, at least on a first pass.

One more practical benefit deserves mention: interview and hiring conversations in this space increasingly assume exactly this kind of fluency. An engineer who can describe why a transformer block has both an attention sub-layer and a feed-forward sub-layer, or why residual connections matter for training stability, signals a level of seriousness that a list of API integrations on a resume does not. That signal matters whether the goal is landing a new role, leading a technical evaluation at a current job, or simply contributing more usefully to a team already building on these models.

The stakes of getting this wrong are not hypothetical. Teams have shipped products that confidently stated incorrect information to customers, that leaked more of a system prompt than intended because of a misunderstanding about context handling, or that burned unexpectedly large budgets because nobody modeled token growth across a multi-turn conversation. Each of these is traceable to a specific mechanism covered later in this book. Learning the mechanism in advance is considerably cheaper than learning it from an incident report.

The remainder of this introduction sets the practical terms for that learning: the background assumed, and the reading strategy that gets the most out of the chapters ahead.

The baseline assumption is comfort with Python at a working, professional level. Code examples throughout this book use Python because it is the common language of both application development against LLM APIs and the research code those APIs are built on. Reading a function definition, following a loop, and understanding basic data structures like lists and dictionaries is enough. Nothing here requires familiarity with a specific machine learning library in advance, since any library-specific code that appears will be explained inline rather than assumed as prior knowledge.

No prior machine learning background is required, and that point deserves to be taken literally rather than as a reassuring disclaimer. Terms like neural network, gradient, or loss function will all be introduced from scratch where they first matter, with enough explanation to use them correctly afterward. A reader who has taken a university course on the subject will find some early material familiar, and can move through those sections quickly, but a reader who has never opened a machine learning textbook will not be left behind either.

Some comfort with basic mathematics helps but does not need to be extensive. Vectors, the idea of multiplying and summing numbers arranged in a list, and probabilities expressed as numbers between 0 and 1 are the mathematical vocabulary used most often. Where a formula appears, such as the cross-entropy loss used to train these models, it will be walked through in plain language alongside the notation, so that the notation supports the explanation rather than replacing it. Nobody needs a background in calculus, linear algebra proofs, or statistics theory to follow any chapter in this book.

A development environment capable of running Python is useful for engineers who want to actually execute the code examples rather than just read them, though this is optional. The training loop walkthrough later in this book, for instance, is small enough to run on a laptop in simplified form, and seeing it execute and produce a loss number that decreases over iterations tends to cement the concept far more effectively than reading about it in isolation. A reader who prefers to read code as illustration rather than run it will still get the full conceptual content either way.

Practical experience calling an LLM API, such as sending a prompt and receiving a completion through a vendor's SDK, is helpful context but not a prerequisite. Readers who arrive having already built something on top of a model, even a small prototype, will recognize many of the behaviors described here immediately: inconsistent output length, occasional confident errors, costs that scale in surprising ways with conversation length. Those memories give the abstract mechanism something concrete to attach to. Readers without that experience yet will simply be building that intuition for the first time alongside the mechanism, which works just as well.

One thing genuinely not needed is any particular allegiance to a specific vendor or model family. The mechanisms covered here, next-token prediction, tokenization, attention, pretraining, fine-tuning, and inference-time decoding, are shared across essentially all modern large language models regardless of which company built them. A reader who primarily uses one particular hosted API, a locally run open model, or several different systems across different projects will find the content applies equally, because the architecture described is the common substrate underneath nearly all of them.

What does help, more than any specific technical prerequisite, is curiosity about mechanism over curiosity about results. It is easy to be impressed by what a model produces and stop there. Getting real value from this book means being willing to ask, repeatedly, some version of the question but how does it actually do that, and sitting with the answer even when it turns out to be less magical and more mechanical than expected. That mechanical reality, a system repeatedly predicting one token at a time, is the entire foundation the rest of the book is built on, and accepting it early makes every later chapter easier to absorb.

A short self-check can help calibrate readiness before diving in. The following quiz covers general comfort level rather than any content from the book itself.

If the self-check reveals gaps in Python fluency specifically, it is worth closing those before continuing, since code examples will not pause to explain basic language syntax. If the gaps are instead in machine learning or mathematics background, there is nothing to close in advance. Those gaps are precisely what the chapters ahead are designed to fill, in the order they are needed, without requiring a detour through outside material first.

Readers already working daily with LLM APIs should not assume their experience makes this book redundant. Using a system successfully and understanding why it behaves the way it does are different skills, and the gap between them is exactly where expensive production mistakes tend to live. Even an experienced API user is likely to find several moments across the chapters ahead where a long-standing assumption about model behavior turns out to be incomplete or slightly wrong once the actual mechanism is laid out.

With the prerequisites settled, the only remaining question is strategy: how to move through ten chapters covering a genuinely cumulative set of ideas without losing the thread partway through. The next section addresses that directly.

The chapters are ordered to mirror the actual lifecycle of a large language model, and that ordering is the single most important reading strategy to respect. The book opens with next-token prediction, the single mechanism that every other concept elaborates on or builds around. It then moves through tokenization, embeddings, self-attention, and the full transformer block, which together describe how a model represents and transforms text internally. From there it covers pretraining and scaling, which describe how a model gets built in the first place, followed by fine-tuning and alignment, which describe how a built model gets shaped toward following instructions. The final chapters cover inference, the moment-to-moment behavior of a deployed model, and hallucination, the most consequential failure mode engineers need to design around.

Reading in order matters more here than in many technical books, because later chapters genuinely depend on earlier ones rather than merely referencing them for color. The chapter on self-attention, for instance, assumes a reader already understands that tokens have been converted into vectors through an embedding lookup table, a concept introduced two chapters earlier. Jumping straight to self-attention without that foundation means encountering vector operations on an abstraction that has not yet been explained, which turns a clear explanation into a confusing one through no fault of the material itself.

That said, the book is also built so each chapter can serve as a reference later, once a first full read is complete. An engineer who needs a refresher on how KV cache reuse works before a specific optimization project does not need to reread the whole book to get there. The chapter on inference stands on its own well enough for that kind of targeted lookup, provided the foundational vocabulary from earlier chapters, terms like token, embedding, and attention weight, is already familiar from the first pass through.

Each chapter follows a consistent internal shape worth knowing in advance. It opens with the mechanism explained conceptually, often through a concrete example rather than abstract description first. It then moves into enough technical and sometimes mathematical detail to make the mechanism precise rather than vague. It closes by connecting the mechanism back to an engineering consequence: a cost implication, a design tradeoff, or a common misunderstanding worth correcting. Reading for that structure, rather than trying to memorize every detail on a first pass, tends to work better than treating each paragraph as equally weighted.

Code examples are meant to be read actively rather than skimmed past. A short Python snippet illustrating how a training loop updates weights after computing cross-entropy loss communicates something that a paragraph of prose alone cannot fully convey, namely the exact sequence of operations involved. Where time allows, actually running these snippets, even with toy data and a tiny model, builds intuition faster than reading them silently. Where time does not allow, reading them slowly enough to trace the flow of data through each line still delivers most of the benefit.

The following steps outline a practical reading plan for engineers working through this material alongside a full-time job, where reading in long uninterrupted stretches is rarely realistic.

For readers who want to prioritize based on an immediate work need rather than reading the book cover to cover first, some targeted shortcuts exist, though they come with a caveat. An engineer debugging unexpected API costs right now might jump straight to the chapter on tokens and tokenizers, or the chapter on inference covering context windows. An engineer evaluating a vendor's claims about a new model release might jump to the chapter on scaling laws. These shortcuts work reasonably well for urgent, narrow questions, but they will occasionally reference a term, like attention weight or residual connection, that a chapter earlier in the sequence defines more carefully.

The definitions, examples, and callout boxes scattered through each chapter are there specifically to support this kind of nonlinear use without requiring a full linear read first. A definition callout introducing a term like hallucination taxonomy is written to stand alone, so a reader arriving at that chapter out of sequence is not completely lost even without having read everything before it. But the deepest and most durable understanding still comes from reading start to finish at least once, letting each mechanism accumulate on top of the last the way it was designed to.

A note on pacing: this is not a book meant to be rushed through in a single weekend, despite how approachable the writing aims to be. The ideas are cumulative enough that a steady pace, one chapter every few days, tends to produce better retention than an intense but compressed binge. There is no test at the end and no deadline attached to finishing, so there is little cost to taking the material at whatever speed actually allows it to stick, chapter by chapter, mechanism by mechanism, until the full machine comes into focus.

What follows next is the first mechanism itself: the single, repeated act of predicting one token at a time, out of which every other behavior described in this book eventually emerges.

A language model's training objective can be stated in one sentence: given a sequence of tokens, predict the token that comes next. That is the entire task definition handed to the network during pretraining, and it is deceptively simple to say out loud. There is no instruction-following objective baked into this stage, no notion of correctness beyond matching what plausible human-written text tends to contain, and no explicit representation of facts, logic, or intent. The model simply learns, over an enormous number of examples, which token tends to follow which sequence of prior tokens, and it adjusts its internal parameters to make its guesses line up with what actually appears in real text.

This single-step framing is what lets the same mechanism scale up to apparently complex behavior. Writing a paragraph is not one decision, it is a chain of next-token decisions, each one conditioned on everything produced so far, including the model's own earlier output. This is called autoregressive generation: the model generates one token, appends it to the input, and then predicts the next token given that slightly longer sequence. A five-hundred-word essay is, mechanically, five hundred or more repeated applications of the exact same next-token step. Nothing about the process changes between predicting the second word of a sentence and predicting the five-hundredth.

Consider how this plays out concretely. Suppose the input so far is The weather today is. The model does not retrieve a weather report. It estimates, from patterns learned across its training text, that words like sunny, cloudy, cold, or nice are plausible continuations, while a token like purple or quickly is far less likely to follow grammatically and semantically. It picks one, appends it, and repeats the process for the next position, now conditioning on The weather today is sunny. Each step is local, but the accumulated chain is what produces a coherent sentence.

Precision matters here: when a model appears to work through a math problem or a logical puzzle, there is no hidden reasoning engine running separately from the token stream. The appearance of reasoning emerges because the model has learned, from training data full of worked examples, that laying out intermediate steps as text tends to make the final answer's tokens more predictable. Much of the visible reasoning happens in the sequence of tokens, not behind it, although interpretability research has found that models also carry out some intermediate steps inside their internal computations. This is one reason prompting a model to think step by step can change its output: it changes the sequence being conditioned on, which changes what the next-token step considers plausible.

This also explains a common point of confusion for engineers coming from traditional software, where a function either has a defined behavior for an input or it does not. A next-token predictor never fails to produce a next token. It always has a distribution ready, even for inputs that are nonsensical, out of scope, or outside anything resembling its training data. There is no concept of I don't know how to continue this built into the raw mechanism itself. Handling uncertainty, refusing to answer, or flagging low confidence are all behaviors that have to be deliberately trained or engineered on top of the base prediction task, a point that resurfaces directly when this book gets to hallucination.

The output of a single prediction step is not a single chosen token. It is a probability distribution over the vocabulary, meaning the model assigns a probability to every single token in its vocabulary, from the most likely candidate down to the least likely. If the vocabulary contains, say, several tens of thousands of entries, the model produces that many probabilities at every single step, one per possible token, and they all sum to one. A separate decoding process, covered in the chapter on sampling, temperature, and context windows, then decides how to turn that distribution into an actual chosen token.

Mathematically, this distribution is produced by a function commonly called the softmax, which converts a set of raw scores, often called logits, into valid probabilities. Given raw scores z1, z2, ..., zV for each of the V tokens in the vocabulary, the probability assigned to token i is:

P(token = i) = e^z_i / Σ_j=1^V e^z_j

This formula guarantees two useful properties. Every probability is positive, since an exponential is always positive no matter the sign of the input score, and all the probabilities sum exactly to one, since the denominator is just the sum of all the same exponentials used in the numerators. That makes the output a well-formed probability distribution that downstream code can sample from, rank, or threshold, rather than an arbitrary list of numbers.

A worked example makes this concrete. Suppose the vocabulary were tiny, containing only four tokens: cat, dog, run, and jump, and after processing some input the model produces raw scores of 2.0, 1.0, 0.1, and 0.1 respectively. Running these through the softmax formula produces probabilities of roughly 0.60 for cat, 0.22 for dog, and about 0.09 each for run and jump. Real vocabularies are far larger, but the mechanics scale up unchanged: one score per token, converted into one probability per token, all summing to one.

Here is a minimal illustration of that calculation in Python, using only plain lists so the mechanics are visible without any library hiding the steps:

import math

logits = {"cat": 2.0, "dog": 1.0, "run": 0.1, "jump": 0.1}
exp_scores = {token: math.exp(score) for token, score in logits.items()}
total = sum(exp_scores.values())
probabilities = {token: value / total for token, value in exp_scores.items()}

for token, p in probabilities.items():
    print(f"{token}: {p:.3f}")

The size of the vocabulary matters practically, not just mathematically. A model with a vocabulary of tens of thousands of tokens must compute, store, and normalize that many scores at every single position in every sequence it generates. This is a real computational cost, and it is one reason vocabulary design (covered in the chapter on byte-pair encoding) is not a cosmetic choice. A larger vocabulary can shorten sequences by packing more meaning per token, but it also makes this one prediction step more expensive, since the softmax has to run over a larger set of candidates each time.

One subtlety worth flagging here, without going into full depth, is that the raw probability distribution is rarely used exactly as produced. Settings like temperature reshape the distribution before a token is chosen, and top-p / nucleus sampling restricts the choice to a subset of likely tokens. Those mechanisms belong to the chapter on inference and decoding, but it matters to know now that the distribution described above is a starting point, not the final word. The model's job ends at producing this distribution faithfully; what happens next is a separate, configurable decision.

The key takeaway from this section is simple to state but easy to forget in practice: a language model never directly outputs text. It outputs a ranked set of odds over every token it knows about, one step at a time, and a decoding procedure built on top of that is what turns those odds into the words appearing on screen. Confusing the model's distribution with the final decoded text is a common source of misunderstanding when reading papers that discuss things like entropy of the output distribution, calibration, or sampling diversity.

A model trained only on next-token prediction over raw internet-scale text does not naturally behave like a helpful assistant. Trained purely on that objective, it tends to continue whatever pattern the input resembles. Given a question, it might continue with another question, mimicking a list of exam questions found during training, rather than answering the first one. This gap between predicts plausible continuations of text and answers my question helpfully is exactly the gap that later stages of training exist to close, and understanding that gap is what makes chat behavior legible rather than magical.

GPT-3 is a useful reference point for this stage of the story. It was an autoregressive language model trained at large scale and evaluated by giving it prompts with zero, one, or a handful of examples embedded directly in the text, rather than through any separate instruction-following mechanism.^[1] In other words, its capability to follow a pattern shown to it in the prompt, called few-shot prompting, emerged from the same next-token mechanism already described, with no separate reasoning or instruction-parsing component added. The model simply predicted plausible continuations of a prompt that happened to contain a worked pattern, and the continuation happened to extend that pattern correctly.

That capability, impressive as it was, still left a gap between completes patterns well and reliably follows an instruction phrased as a request. A raw next-token model given the prompt Write a polite email declining a meeting might just as easily continue with more instructions of the same style, since that is a plausible pattern in its training data, rather than actually writing the email. Closing that gap is the job of the later training stages, specifically supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), both covered in depth in the chapter on fine-tuning and alignment. Those stages do not replace next-token prediction with something else. They retrain the same prediction mechanism on curated examples where the plausible continuation of an instruction actually is a direct, helpful response.

It helps to think of this as reshaping which continuations the model treats as likely, rather than adding a new capability from scratch. Before this shaping, given Explain photosynthesis simply, a plausible continuation in raw training-style text might be another unrelated sentence, a list of similar questions, or a tangential discussion. After fine-tuning on examples where instructions are followed by direct, well-formed answers, the model's learned distribution shifts so that an actual explanation becomes the high-probability continuation. The underlying mechanism, predicting one token at a time from a probability distribution, never changes. What changes is which continuations the training process has made likely.

This reframing also explains why a chat interface feels like a back-and-forth conversation even though the underlying model is still just predicting one token at a time. Chat systems built on top of these models typically format the conversation history, including the system instructions, the user's messages, and the model's own prior replies, into a single long sequence of tokens before every single generation step. The model then predicts the next token for that whole sequence, exactly as described earlier in this chapter, with the conversation structure living entirely in how the text is formatted, not in any special conversational mode inside the model itself.

The steps below summarize that layered picture, from the raw mechanism up to the behavior a user actually experiences in a chat application.

Seeing this pipeline laid out makes it clear that nothing about a chat reply happens all at once. The fluency of a long, well-structured answer is an emergent property of thousands of individually narrow decisions, each one a probability distribution computed over exactly the same vocabulary, each one shaped by training to make helpful continuations more likely than unhelpful ones. There is no moment where the model decides on the whole answer in advance; the structure that feels planned is a byproduct of patterns learned during fine-tuning, expressed one token at a time.

This also clarifies why things like formatting instructions, example outputs, and system prompts are such effective engineering levers. Each one changes the token sequence the model conditions on, which changes the probability distribution at every subsequent step, which changes which continuation looks most plausible. There is no separate configuration layer being adjusted behind the scenes. Prompt engineering works precisely because it operates directly on the one mechanism this entire chapter has described: a model that looks at the tokens so far and produces a distribution over what comes next, applied again and again until the output is complete.

Keeping that single mechanism in view is what makes the rest of this book legible. Tokenization determines what the raw material of that sequence actually looks like, embeddings determine how each token gets represented numerically before prediction happens, self-attention and the transformer block determine how the model computes its score for each candidate token, and pretraining, scaling, and alignment determine which continuations end up looking plausible in the first place. Every one of those pieces exists to serve, refine, or reshape the same one-step guessing game introduced at the start of this chapter, and every chapter ahead builds directly on that fact.

A token is simply a unit of text that a tokenizer decides to treat as one piece. That piece might be a whole common word like cat, a fragment of a longer word like ing at the end of running, a single punctuation mark, a space bundled with the next word, or even a single byte that does not correspond to any readable character on its own. There is no fixed rule that says a token equals a word. Tokenizers used by modern language models very often split uncommon words into two or more pieces, while keeping frequent words, and even frequent short phrases, as a single piece.

Once text has been split into pieces, each piece gets mapped to an integer called a token ID. The tokenizer keeps a fixed list, its vocabulary, pairing every possible piece with exactly one ID. The word the might map to ID 262, the fragment tion might map to ID 341, and so on. This mapping is static: it was decided once, during tokenizer construction, and does not change at inference time. When a prompt comes in, the tokenizer splits it into pieces from its vocabulary, applying the merge rules it learned, and returns a sequence of these IDs. Those IDs, not the original string, become the input the model works with.

Consider an uncommon name like Szczepanski. A tokenizer trained mostly on English text has probably never seen this exact string often enough to give it one dedicated ID. Instead it breaks it into smaller recognized fragments, perhaps something like Sz, cz, ep, an, ski, each with its own ID. Five or six tokens for one name is common. Compare that to a frequent English word like the, which almost always gets exactly one token. This unevenness is normal and expected, not a bug in the tokenizer.

Spacing and capitalization matter more than most engineers expect. Many tokenizers attach the leading space to the following word as part of a single token, so " dog" (with a leading space) and "dog" (without one) can be two entirely different tokens with two different IDs. A capitalized "Dog" may be different again. This explains why reformatting a prompt, adding stray whitespace, or changing capitalization can shift token counts in ways that look arbitrary from the outside but follow directly from how the vocabulary was built.

A token is whatever the tokenizer's vocabulary says it is, nothing more and nothing less. Word boundaries, grammar, and meaning never enter into that decision directly.

Tokenization also explains why language models sometimes struggle with letter-level tasks, such as counting the letters in a word or reversing a string character by character. If a word like strawberry gets represented as two or three subword tokens rather than individual letters, the model's internal view of that word is a short list of chunk identifiers, not a list of letters. Asking it to reason about individual letters means asking it to work with information that its tokenization does not expose directly. This is a direct, practical consequence of subword tokenization, not a separate mystery about model reasoning.

Different model providers and different model families use different tokenizers, often with different vocabularies and different rules for splitting text. A prompt tokenized one way for one model might split differently for another model, even if both are large modern language models. This matters for anyone switching providers or comparing token-based pricing across APIs: the same English sentence can cost a measurably different number of tokens depending on which tokenizer processes it, since vocabulary size and training data shape exactly which chunks earned their own dedicated ID.

A short glossary helps keep these pieces straight when reading papers or API documentation:

• Token: one unit of text as defined by a specific tokenizer's vocabulary.
• Token ID: the integer a tokenizer assigns to a given token, used internally by the model.
• Tokenizer: the program and vocabulary that convert text to token IDs and back again.
• Vocabulary: the fixed list of all tokens a specific tokenizer can produce.

None of this touches on what the model does with those IDs once it has them, how it turns an ID into a vector, or how it decides what comes next. Those are separate questions handled by the embedding lookup table and by next-token prediction. What matters here is narrower and more mechanical: text goes in, a fixed set of rules chops it into pieces, and those pieces become the integers that everything downstream depends on.

Byte-pair encoding (BPE) is the most common algorithm used to build a subword vocabulary for modern language models. Tokenization converts text into discrete token IDs, and tokenization methods include subword approaches such as byte-pair encoding.^[1] The idea behind BPE did not originate in machine learning at all. It started as a simple data-compression trick: repeatedly find the most frequent adjacent pair of symbols in a body of text and replace that pair with a new symbol that represents it. Applied to language model vocabularies, this same repeated-merging process produces a set of subword chunks that balances two competing goals: a small enough vocabulary to keep the model manageable, and large enough chunks that common words do not get shredded into single characters.

The construction process starts from the smallest possible pieces, individual characters or bytes, and builds upward. Here is the general procedure, stated as a sequence of steps that gets repeated until the target vocabulary size is reached:

Walking through a tiny toy example makes the mechanism concrete. Suppose the entire training text consists only of the words low, lower, lowest, repeated many times, each split into characters with a marker for word boundaries. Early on, the pair l and o appears constantly, so it merges into lo. Next, lo and w appears constantly, merging into low. Later passes might merge e and r into er, since that pair shows up often in lower. After enough merges, low becomes a single token in its own right, while rarer endings like est might stay as two or three smaller pieces, because the training data contained fewer examples to justify merging them further.

This explains, at a mechanical level, why common English words usually become a single token while rare words, names, and words in low-resource languages often fragment into several. A merge only gets added to the vocabulary if the pair it represents appeared frequently enough in the training corpus to be worth a dedicated slot. Frequent patterns across the training data earn merges early and often, producing compact single-token representations. Rare patterns never accumulate enough frequency to earn a merge, so they stay broken into smaller, more generic pieces, each costing a separate slot in the eventual token sequence sent to the model.

A short, runnable sketch helps make the merge step precise rather than abstract. This simplified version ignores word boundaries and efficiency concerns that a production tokenizer would handle carefully, but it shows the core loop:

from collections import Counter

def get_pair_counts(tokens):
    pairs = Counter()
    for i in range(len(tokens) - 1):
        pairs[(tokens[i], tokens[i + 1])] += 1
    return pairs

def merge_pair(tokens, pair, new_symbol):
    merged = []
    i = 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
            merged.append(new_symbol)
            i += 2
        else:
            merged.append(tokens[i])
            i += 1
    return merged

tokens = list("lowlowerlowest")
for step in range(3):
    pairs = get_pair_counts(tokens)
    best_pair = pairs.most_common(1)[0][0]
    new_symbol = "".join(best_pair)
    tokens = merge_pair(tokens, best_pair, new_symbol)
    print(f"Merge {step + 1}: {best_pair} -> {new_symbol}")
    print(tokens)

Running that loop on a larger, realistic corpus (billions of words rather than one short string) for many thousands of merge steps is exactly how a production BPE vocabulary gets built before any model training begins. The final vocabulary size, often tens of thousands of entries, is chosen in advance as a design decision. A larger vocabulary means fewer tokens per sentence on average, since more complete words and common phrases earn their own slot, but it also means a larger embedding lookup table and a heavier final prediction layer, since the model has to assign a probability to every vocabulary entry at each step. A smaller vocabulary shrinks those layers but forces more words to fragment into multiple tokens, lengthening every sequence.

Byte-level BPE, the variant used by several widely deployed tokenizers, starts the merging process from raw bytes rather than Unicode characters. This detail matters because it guarantees that absolutely any input, including unusual symbols, emoji, or text in scripts the training data barely covered, can always be represented, since any sequence of bytes can be built from the base byte vocabulary even without a single merge covering it directly. The tradeoff is that genuinely unfamiliar scripts or symbols end up broken into many individual byte-level tokens rather than a few meaningful chunks, which inflates token counts sharply for that kind of text compared to the common languages and formats the tokenizer's merges were optimized around.

It is worth separating two layers that sometimes get confused in casual explanations. One layer is the algorithm: the repeated counting and merging process described above, which runs once to build a fixed vocabulary. The other layer is the tokenizer itself as it operates afterward: a lookup procedure that takes new text, splits it into tokens from that fixed vocabulary, and returns a sequence of IDs. Once vocabulary construction finishes, tokenizing new text at inference time is fast: it does not repeat any merge counting, it just applies the previously learned merge rules in order to split new input the same way the training corpus was split.

Other subword methods exist alongside byte-pair encoding, including approaches that build a vocabulary by directly optimizing the likelihood of the training data rather than by greedy pairwise merging. These alternative methods pursue the same underlying goal: a vocabulary that keeps common patterns compact while still being able to represent anything. The exact algorithm a given model's tokenizer uses rarely matters for everyday engineering work. What matters is the shared outcome across all of them: text becomes a sequence of subword chunks chosen for frequency, not for meaning or for matching human intuitions about where one word ends and another begins.

Once the mechanics of byte-pair encoding are clear, the practical consequences for building applications follow directly. Most commercial language model APIs charge per token, counting both the tokens sent in the prompt and the tokens generated in the response. A prompt that looks short in characters can still cost more tokens than expected if it contains rare words, unusual formatting, code snippets with dense symbols, or text in a language underrepresented in the tokenizer's training data. Estimating cost by word count alone, a habit carried over from ordinary writing tools, routinely misleads engineers who have not yet internalized how uneven subword splitting can be.

Context windows add a second, related constraint. Every model accepts only a fixed maximum number of tokens across the prompt and the generated response combined, a limit covered in detail when discussing context windows directly. The point relevant here is that this limit is counted in tokens, not characters or words, so the same maximum budget buys noticeably less room when the text is dense with rare words, code, or non-English scripts that fragment heavily. An engineer budgeting for a long document to summarize needs to measure that document in tokens using the actual tokenizer for the target model, not estimate loosely from its character count.

Formatting choices that look cosmetic can shift token counts more than engineers expect. Adding extra whitespace, switching between tabs and spaces in code, changing capitalization, or inserting markdown syntax like asterisks and backticks can all change how a tokenizer's merge rules split the surrounding text, since the tokenizer may have learned a merge for the capitalized or spaced version that does not apply to the alternate form. The table below summarizes common patterns that tend to inflate token counts, alongside the kind of text that tends to stay compact.

A concrete workflow habit solves most of the surprises described above before they become production incidents. Rather than guessing at token counts from character length, run representative sample prompts, including real user input and real documents, through the actual tokenizer for the model being used, and inspect the resulting token count directly. Most providers publish a tokenizer library or an online tool for exactly this purpose. Doing this once during development, and again whenever the prompt template or target model changes, catches cost and context-window surprises long before they show up as a production bill or a truncated response.

Latency is a second practical consequence that often gets overlooked next to cost. Generating a response token by token means that longer expected outputs take proportionally longer to produce, and this measurement of length is in tokens, not words. Two prompts that ask for answers of similar apparent length in English can still produce responses of very different token counts if one answer naturally uses more rare words, more punctuation, or formatting like bullet points and code blocks that the tokenizer splits less efficiently. Engineers tuning response time budgets benefit from thinking in tokens from the start rather than converting from an intuitive word estimate afterward.

Multilingual applications deserve particular attention, since tokenizer vocabularies are built from training corpora that are rarely balanced evenly across every language. A vocabulary trained predominantly on English and a handful of other high-resource languages will typically represent those languages compactly, while text in a lower-resource language fragments into many more, often byte-level, tokens for the same amount of meaning. This has a direct, double-sided business consequence: serving users in that language costs more per equivalent sentence, and the same context window holds less content before truncation, both of which should factor into capacity planning and pricing assumptions for genuinely multilingual products.

A useful mental test, once these mechanics are clear, is to ask of any surprising model behavior: would this look different if the input were tokenized differently? Odd results on letter-counting tasks, inconsistent handling of rare names, unexpected costs for seemingly short prompts, and context windows that fill up faster than expected on code-heavy input all trace back, at least partly, to how byte-pair encoding chose to split that specific text. None of this requires retraining a tokenizer or reading research papers on merge algorithms. It requires remembering, every time text crosses into an API call, that the model never sees characters or words. It sees a list of token IDs, chosen by a fixed vocabulary built long before that particular prompt was ever written.

Mechanically, an embedding lookup table is nothing more exotic than a big matrix. If the vocabulary has V entries and each embedding vector has d numbers (its dimensionality), the table is a matrix of shape V × d. Looking up the embedding for token ID i just means reading row i out of that matrix. There is no formula, no computation beyond indexing. The entire intelligence of this layer lives in the specific numbers that end up in each row, and those numbers are learned, not designed by hand, during training.

At the start of training, the table is typically filled with small random numbers, so every token starts out meaning nothing in particular and sitting near the center of the space by pure chance. As training proceeds, the standard pretraining process adjusts these numbers through gradient descent, driven by the next-token prediction objective described under autoregressive pretraining. A token's row only moves when that token actually shows up in training data, and it moves in directions that help the model predict surrounding tokens more accurately. Over many passes through a large corpus, tokens that tend to appear in similar contexts drift toward similar regions of the space, purely as a side effect of minimizing prediction error.

It is worth sketching this in code to make the lookup concrete rather than abstract. A minimal PyTorch-style embedding layer looks like this:

import torch
import torch.nn as nn

vocab_size = 50000
embedding_dim = 768

embedding_table = nn.Embedding(vocab_size, embedding_dim)

token_ids = torch.tensor([42871, 9003, 17])
vectors = embedding_table(token_ids)
print(vectors.shape)  # torch.Size([3, 768])

That single line, embedding_table(token_ids), is doing exactly the row-lookup described above: for each ID in the input, it pulls out the corresponding row of the 50000 × 768 matrix. Everything that follows in the transformer, the query, key, and value vectors inside self-attention, the transformations inside each feed-forward layer, starts from these numbers per token. Changing the embedding dimension changes how much room each token has to encode nuance, but it also changes how many parameters the table holds and how much compute every downstream layer needs, since every subsequent matrix has to match that width.

A second, less obvious detail matters for anyone reading model architecture diagrams closely. Many transformer implementations reuse the same embedding table at the output end of the network, a technique sometimes called weight tying, where the final layer that turns the model's internal vector back into a probability distribution over vocabulary reuses the embedding matrix rather than learning a separate one. This is an implementation choice that saves parameters and often helps training stability, but it is not universal across every model, so it is best treated as something to check rather than assume when reading a specific architecture's documentation.

One practical consequence follows directly from the table being indexed by token ID rather than by word: a tokenizer's vocabulary decisions, covered under byte-pair encoding, directly shape what the embedding table has to represent. If a word gets split into three subword tokens, there is no single embedding for that word, only three embeddings that have to be combined by later layers into something useful. This is why a model can seem to understand a common word instantly but hesitate or behave oddly on a rare or foreign one that fragments into unfamiliar pieces, each with its own imperfectly trained row in the table.

Rare tokens expose the weak point of this whole scheme clearly. A token that appeared only a handful of times in training data gets very few gradient updates, so its row in the embedding table may end up close to its random initialization, essentially undertrained. This matters directly for application builders: a rare product name, an unusual technical term, or a token from a low-resource language may sit in an uninformative part of the vector space, and any downstream computation that relies on it, including attention and similarity search, inherits that weakness. There is no way to detect this from outside the model except by testing behavior on the specific inputs that matter for a use case.

A table this size, often hundreds of millions of parameters just for the embedding layer alone in a large vocabulary model, is also one of the first places engineers look when trying to shrink a model for deployment. Techniques like quantization, which reduce the precision of each number stored in the table, or vocabulary pruning, which drops rarely used tokens, trade off model size against the fidelity of these representations. None of that detail is necessary to use an LLM API, but it explains why documentation sometimes mentions embedding-layer size separately from total parameter count, since this one table can be a meaningful share of a smaller model's footprint.

Once every token has a vector, those vectors live in a geometric space, and geometric relationships between them can be measured. The most common measurement is cosine similarity, which looks at the angle between two vectors rather than their raw distance, and ranges from -1 (pointing in opposite directions) to 1 (pointing in the exact same direction).

cos(θ) = (a · b) / (‖a‖ ‖b‖)

where a · b is the dot product of the two vectors and ‖a‖, ‖b‖ are their lengths. A cosine similarity close to 1 means two tokens, or two pieces of text once their vectors are pooled, point in nearly the same direction in the embedding space, and in practice this often correlates with some notion of topical or semantic closeness.

The famous illustration of this geometry, repeated in countless introductory talks, is that the vector arithmetic king - man + woman lands close to the vector for queen in certain trained embedding spaces. This kind of result is genuinely striking and shows that training on next-token prediction does organize the space along directions that loosely track human-recognizable relationships like gender or plurality. But it is an empirical observation about certain trained spaces, not a guaranteed property of every embedding table, and it should not be generalized into a belief that the space encodes a clean, reliable algebra of meaning for arbitrary analogies.

This is exactly where application builders tend to overreach. A high cosine similarity between a user's query and a document chunk in a retrieval system is a useful signal, not a proof of relevance. Two pieces of text can be grammatically and topically similar while meaning opposite things, for instance the contract can be terminated with notice and the contract cannot be terminated with notice may land close together in vector space because they share almost all the same words, even though their meanings are nearly opposite. A retrieval pipeline that trusts similarity scores blindly will happily hand a negated clause to a model as if it supported the opposite claim.

Dimensionality adds another layer of nuance worth internalizing. An embedding space with many dimensions is not something a human can visualize directly, and intuitions built on two or three dimensional geometry do not always transfer cleanly. High-dimensional spaces have counterintuitive properties, for instance most pairs of random high-dimensional vectors turn out to be roughly orthogonal (near-zero cosine similarity) by default, so a similarity score has to be interpreted relative to the typical scores seen for unrelated pairs in that specific space, not against some universal threshold like 0.5 that happens to sound reasonable.

It also matters that the embedding produced inside a full language model is not the same object as the embedding produced by a dedicated embedding model trained for search or clustering. The internal embedding layer inside a generative transformer is optimized purely to help next-token prediction, with no explicit training signal pushing semantically related texts toward each other. A dedicated embedding model, by contrast, is usually trained with a contrastive objective that directly rewards pulling related pairs close together and pushing unrelated pairs apart. Confusing the two, for instance by grabbing an internal activation from a chat model and expecting it to behave like a purpose-built embedding, often produces disappointing retrieval results.

Context further complicates any notion of a single, fixed meaning per token. The raw embedding lookup gives every occurrence of a given token ID the exact same starting vector, regardless of surrounding words, which is why a word like bank starts identically whether the sentence is about rivers or finance. It is only self-attention, layered on top of this initial embedding, that lets the representation shift based on context, producing a context-dependent representation by the time it reaches later transformer blocks. So the static embedding table is best understood as a starting point, a rough and context-free coordinate, not the model's final word on what a token means in a given sentence.

The matrix below lays out this distinction along two axes that matter for engineering decisions: how much a given similarity signal depends on surface wording versus deeper relationships, and how much it holds up with or without surrounding context. It is meant as a quick gut check, not a precise measurement, for deciding how much weight to put on a similarity score in a particular pipeline.

None of this means embedding geometry is unreliable in general, only that it is a statistical artifact of training data and objective, not a verified semantic ground truth. It is genuinely useful for clustering similar support tickets, powering approximate search, or detecting near-duplicate content, precisely the kinds of tasks where an approximate, probabilistic signal is good enough. The failure mode to avoid is treating a similarity score as equivalent to a human judgment of meaning, correctness, or logical relationship, since the vector space was never trained to guarantee any of those things.

Most engineers meet embeddings not inside a transformer's internals but through a standalone embedding API, used to build a retrieval system, a recommendation feature, or a semantic deduplication pipeline. The typical workflow turns a batch of text chunks into vectors, stores those vectors in a specialized index, and later encodes an incoming query into the same space to find the stored chunks whose vectors are closest to it. Getting value out of this pattern without getting burned by its failure modes takes some deliberate design choices, which the steps below lay out as a practical checklist.

Chunk size deserves extra attention because it directly interacts with the geometry discussed earlier. A single sentence pulled out of a longer argument can embed in a way that looks similar to a completely different claim, simply because negation or qualification happened in a neighboring sentence. Chunking too coarsely has the opposite problem: a long chunk covering several distinct subtopics ends up with a vector that is a blurry average of all of them, matching weakly against almost everything and strongly against nothing. There is no universal correct chunk size, since it depends on document structure and query style, which is why testing retrieval quality on representative queries matters more than copying a default value from a tutorial.

Dimensionality choices also carry real tradeoffs in production systems. A higher-dimensional embedding can, in principle, encode finer distinctions, but it also costs more to store and compare at scale, since a vector database holding millions of chunks multiplies that storage and comparison cost directly by the vector's length. Many embedding APIs now offer a way to truncate a vector to a shorter length while preserving most of its usefulness, a technique sometimes called Matryoshka representation, letting engineers trade a small amount of retrieval quality for substantial storage and speed savings. The right choice depends on the scale of the index and the latency budget of the application, not on always picking the largest available dimension.

A related and often underestimated risk is embedding drift across model versions. If an embedding model is updated, even subtly, the new version's vectors are not guaranteed to be comparable to the old version's vectors, because the geometry of the space itself can shift during retraining. An index built with an older embedding model compared against queries encoded with a newer one will produce scores that look plausible but carry no real meaning, since the two sets of vectors were never trained to share a coordinate system. Any system that depends on embedding similarity needs a clear policy for re-embedding stored content whenever the embedding model changes, treated as a required migration step rather than an optional cleanup task.

Privacy and sensitivity considerations also follow from how embeddings are built. Because a vector is a lossy but information-rich compression of its source text, embeddings of sensitive documents, such as medical notes or personal messages, can sometimes be partially reconstructed or at least linked back to their source text through careful analysis, particularly if an attacker has access to the embedding model and can query it. Treating a vector database as inherently anonymous or safe to store without access controls is a mistake; it deserves roughly the same handling discipline as the original text it was derived from, including encryption at rest and scoped access permissions.

Clustering is another common application that benefits from the same caution applied earlier to similarity scores. Grouping documents or support tickets by embedding similarity, often with an algorithm like k-means applied to the vectors, can surface genuinely useful structure, such as a cluster of billing complaints or a cluster of feature requests. But a cluster boundary drawn purely from vector geometry reflects surface and topical patterns learned during training, not a guaranteed alignment with the business categories an application actually cares about. It is good practice to review cluster samples manually before shipping an automated categorization feature built this way, since a geometrically tight cluster can still mix unrelated content that happened to share vocabulary.

A useful discipline across all these applications is to think of an embedding as an index, not as an answer. It narrows a large collection of candidates down to a short, plausible list far faster than any exhaustive comparison could, which is exactly why it is valuable for search and retrieval at scale. What it does not do is verify truth, confirm logical consistency, or guarantee that the narrowed list is complete or correctly ranked for a specific need. Systems that rely on embeddings for a critical decision, legal document review or medical triage being clear examples, need a verification layer on top, built from rules, a second model pass, or human review, rather than trusting the vector space's geometry as a final judgment.

Finally, it is worth remembering that the embedding lookup table inside a generative model and a standalone embedding API, though built from the same underlying idea, serve genuinely different engineering purposes and deserve different scrutiny. The internal table is invisible plumbing that makes self-attention and the rest of the transformer possible at all, and an application builder rarely interacts with it directly. The external embedding API is a tool that gets chosen, tested, and maintained like any other dependency, with its own versioning risks, cost tradeoffs, and failure modes. Keeping that distinction clear is what lets an engineer read a paper describing internal model geometry and a vendor's documentation for an embeddings endpoint without conflating the two into a single vague idea of the embedding thing.

Start with the embedding for each token in a sequence, a vector produced by the embedding lookup table described in the previous chapter. Self-attention does not use that embedding directly for mixing information. Instead, it transforms each embedding into three separate vectors by multiplying it by three learned weight matrices, producing a query vector, a key vector, and a value vector for every token position. These three matrices are parameters the model learns during pretraining, just like the embedding table itself, and they stay fixed once training ends.

A useful analogy is a library search. Imagine every token announces what it is looking for (its query), what it has to offer to others who are searching (its key), and the actual content it would hand over if asked (its value). The token for "bank" in "fished along the bank" sends out a query roughly meaning "what nearby word tells me what kind of bank this is?" Every other token in the sentence, including "fished," exposes a key that can be compared against that query. The comparison decides how much attention "bank" should pay to "fished" when building its new representation.

Concretely, each of the three vectors comes from a separate linear projection of the same input embedding:

q_i = W_Q x_i, k_i = W_K x_i, v_i = W_V x_i

Here xi is the embedding of the token at position i, and WQ, WK, WV are the learned weight matrices for queries, keys, and values respectively. Every position in the sequence gets its own qi, ki, and vi, computed the same way with the same shared matrices. This is why self-attention scales cleanly to sequences of different lengths: the same three matrices apply to every position, whether the sequence has ten tokens or ten thousand.

It helps to separate what each vector is for, since conflating them is a common source of confusion when first reading transformer code or papers.

• Query vector: represents what the current token is looking for in the rest of the sequence.
• Key vector: represents what a token has to offer, used to match against other tokens' queries.
• Value vector: represents the actual content passed along once a match between query and key is found useful.

The split between key and value matters because the thing a token is best known for and the thing it hands over do not have to be the same shape of information. A token might have a key that makes it easy to find (its grammatical role, for instance) while its value carries different content (its meaning). Keeping these as separate learned projections gives the model more flexibility than if it tried to reuse one vector for both jobs.

In code, a minimal version of producing these three sets of vectors for a batch of token embeddings looks like this, using NumPy for clarity rather than a production deep learning library:

import numpy as np

# x: shape (sequence_length, embedding_dim)
# W_q, W_k, W_v: shape (embedding_dim, head_dim)

def project(x, W_q, W_k, W_v):
    queries = x @ W_q
    keys = x @ W_k
    values = x @ W_v
    return queries, keys, values

Note that W_q, W_k, and W_v are learned during pretraining through the same gradient descent process that shapes every other parameter in the network, not hand-designed. Before training, these matrices start out close to random, so early in training the queries and keys carry no useful signal at all. Over many steps of the training loop, the model gradually shapes them so that semantically or grammatically relevant tokens produce queries and keys that align well, a process entirely driven by the cross-entropy loss on next-token prediction rather than any explicit instruction to "find the subject of the sentence" or "track pronoun references."

It is worth pausing on scale here. A real transformer block does not compute a single query, key, and value per token. It typically splits the embedding dimension into several parallel attention heads, each with its own smaller WQ, WK, WV matrices, so that different heads can specialize in different kinds of relationships, such as tracking subject-verb agreement in one head and coreference in another. That multi-head detail belongs to the fuller discussion of the transformer block structure, but the single-head mechanics described here are exactly what each head repeats, just with its own private set of matrices.

A natural question from an engineering standpoint is why not just compare raw embeddings directly instead of projecting them into separate query and key spaces first. The answer is flexibility: raw embeddings are shaped by the training objective to carry many kinds of information at once, including meaning, position hints, and statistical regularities. Projecting into a dedicated query space and a dedicated key space lets the model learn a narrower, task-specific notion of similarity for the purpose of attention, separate from whatever else the embedding needs to encode for other parts of the network.

One more property worth internalizing before moving to the arithmetic: queries, keys, and values are all recomputed at every transformer block in the stack, not just once at the bottom. Each block receives the output representations from the block below it, and projects those into a fresh set of query, key, and value vectors for its own attention sub-layer. This means the notion of "what bank refers to" can keep sharpening block by block, with early blocks picking up coarse signals and later blocks refining a more specific context-dependent representation.

Finally, note that nothing about queries, keys, or values inherently ties them to a particular token's identity once they are computed. A key vector for "bank" in a river sentence and a key vector for "bank" in a finance sentence will differ substantially, because the embedding feeding into WK already carries some positional and contextual texture from earlier processing, and the projection itself is sensitive to small differences in that input. This is the seed of the context-dependence that self-attention is built to produce, and the next section shows exactly how that seed grows into a full context-dependent representation.

With queries, keys, and values in hand, the actual attention computation follows a fixed recipe applied at every position in the sequence. For a given token, take its query vector and compare it against the key vector of every other token in the sequence, including itself, using a dot product. A larger dot product means the query and key align more strongly, which the model interprets as a stronger signal that this other token is relevant to the current one. This comparison happens for all pairs of positions at once, which is why attention computation is often described through matrix multiplication rather than a loop over pairs.

Those raw dot products, called attention scores, are not yet usable as weights because they can be any real number, including negative numbers, and their scale depends on the dimensionality of the vectors involved. Two corrections are applied. First, the scores are divided by the square root of the key dimension, a scaling step that keeps the values in a numerically stable range regardless of vector size. Second, the scaled scores are passed through a softmax function, which converts them into a set of positive numbers that sum to one for each query position. The full computation is usually written as:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Here Q, K, and V stack the query, key, and value vectors for every position into matrices, and dk is the dimensionality of the key vectors. The output of the softmax is the set of attention weights: one weight per pair of positions, telling you how much a given token's new representation should draw on every other token's value vector. Multiplying those weights by the value matrix V produces a weighted average of all the value vectors in the sequence, and that weighted average becomes the new, context-dependent representation for the query position.

A small worked example makes this concrete. Picture a three-token sequence: "river," "near," "bank." Suppose, after projection, the query vector for "bank" happens to have a high dot product with the key vector for "river" (because the model has learned that certain kinds of nearby nouns are informative about word sense), a moderate dot product with the key for "near," and a low dot product with its own key. After scaling and softmax, this might produce attention weights like 0.65 for "river," 0.25 for "near," and 0.10 for "bank" itself. The new representation for "bank" is then 0.65 times the value vector for "river," plus 0.25 times the value vector for "near," plus 0.10 times the value vector for "bank," all summed together.

The table below shows how this plays out for the same token in two different sentences, illustrating why the output representation for "bank" diverges even though the starting embedding is identical in both cases.

These numbers are illustrative, not measured from a real model, but they capture the key idea: the attention weights redirect most of the new representation's content toward whichever nearby token carries the disambiguating information, river-related words in one case and finance-related words in the other. The final vector for "bank" after this step is no longer just "bank in isolation." It is a blend that leans heavily on its neighbors, which is exactly the property the embedding lookup table alone could never provide.

It helps to walk through the computation as a discrete procedure, since many engineers meet this first as a block of matrix algebra in a paper without seeing the step-by-step flow.

Translating this into NumPy makes the shapes explicit and matches exactly what libraries like PyTorch do under the hood, minus the automatic differentiation needed for training:

import numpy as np

def softmax(x, axis=-1):
    x = x - np.max(x, axis=axis, keepdims=True)
    e = np.exp(x)
    return e / np.sum(e, axis=axis, keepdims=True)

def self_attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)
    weights = softmax(scores, axis=-1)
    return weights @ V, weights

Running this on a small batch of vectors shows the mechanism directly: feed in three or four token embeddings, project them with random matrices, and inspect the returned weights matrix. Each row sums to one, and each row's largest entry shows which other token that position is drawing most heavily from. This small experiment is a useful sanity check whenever debugging why a fine-tuned or prompted model behaves unexpectedly on a specific input, since it makes the abstract "attention" idea into a concrete matrix an engineer can print and read.

One detail matters for anyone building with autoregressive models: during next-token prediction, a token must never attend to tokens that come after it in the sequence, since that would let the model peek at the answer it is supposed to predict. This is enforced with a causal mask, which sets the attention score for any future position to negative infinity before the softmax step, forcing its weight to zero. Without this mask, training would quietly leak future tokens into each prediction, making the model look accurate during training while being useless at actual generation, since at generation time those future tokens simply do not exist yet.

With one pass of this computation complete, every token's embedding has been replaced by a new vector that reflects some portion of the full sequence, weighted by learned relevance. That output then typically passes through additional sub-layers inside the same transformer block, and the whole thing repeats across many stacked blocks, each time recomputing queries, keys, and values from the previous block's output and sharpening the representation further. The result, after many layers, is what lets a model distinguish "bank" the riverbank from "bank" the financial institution purely from context, with no separate lookup or rule ever written by hand.

Because attention weights are numbers between zero and one that sum neatly to one per query position, they look like an obvious candidate for explaining why a model produced a given output. It is tempting to print the weights for a question-answering example, see that the model attended most heavily to a particular phrase in the passage, and conclude that this phrase is "why" the model gave its answer. Some interpretability tools do visualize attention this way, and the visualizations can be genuinely informative for spotting gross patterns, such as a model fixating on the wrong part of a long document. But treating attention weights as a faithful causal explanation of model reasoning goes further than the evidence supports.

The first problem is that attention is only one piece of a transformer block, not the whole computation. After the attention sub-layer produces its weighted combination of value vectors, that output still passes through a feed-forward layer, gets combined with the original input through a residual connection, and is adjusted by layer normalization, all of which can substantially transform the information before it reaches the next block. A high attention weight on one token says something about how values get mixed at that specific sub-layer, but it does not describe what happens afterward, and a model with dozens of stacked blocks performs this sequence of transformations dozens of times.

The second problem is that most transformer blocks use several attention heads in parallel, each with its own separate query, key, and value projections, and each head can attend to completely different things. One head might track grammatical agreement, another might track topic continuity, and a third might show no interpretable pattern at all to a human observer. Averaging or picking one head's weights to display in a visualization throws away the fact that the model's behavior emerges from the combination of all heads across all layers, not from any single head's pattern. A striking attention map from one head can be an artifact of that head's narrow role rather than a summary of the model's overall reasoning.

Research directly testing whether attention weights explain model predictions has found mixed and often discouraging results. Studies have shown that it is sometimes possible to substantially alter a model's attention weights, through adversarial manipulation or simple retraining, while leaving the model's output prediction nearly unchanged, which undercuts the idea that the displayed weights are uniquely responsible for that output. Other studies have shown the reverse: that very different attention patterns can produce identical predictions, suggesting multiple valid "explanations" coexist, which is itself a sign that no single pattern deserves to be called the explanation. The field of interpretability research continues to debate exactly how much weight, so to speak, should be put on attention visualizations.

A practical example makes the risk concrete. Suppose an application highlights the words in a contract that an LLM "focused on" when flagging a clause as risky, based on attention weights from one layer and one head of the model. A reviewer might reasonably treat that highlight as the model's justification and stop digging further. But if the actual decision emerged from a different head entirely, or from a combination spread across several layers, the highlighted words could be misleading rather than clarifying, giving false confidence in a conclusion the model did not actually reach for the stated reason. This matters more as LLM-based tools get used in higher-stakes review workflows, such as legal or medical document triage.

None of this means attention weights are useless for engineering work. They remain a reasonable first place to look when debugging why a model seems to ignore a key piece of context in a long prompt, since a pattern where attention is spread thin or focused entirely on the start of the input can point toward a real issue with how a prompt is structured or how the context window is being used. They are also the basis for some genuinely useful techniques, such as diagnosing position bias in long documents or spotting when a retrieval-augmented pipeline's retrieved passage is getting ignored in favor of the model's own prior knowledge.

The more defensible framing is to treat attention weights as one observable signal among many, useful for forming hypotheses rather than drawing firm conclusions. If an attention pattern suggests the model is ignoring a crucial fact, that is worth testing directly, for instance by removing the fact from the prompt and checking whether the output actually changes. A measured behavioral test, where you alter the input and observe the output, is almost always stronger evidence of what the model relied on than inspecting internal weights, because it tests the system end to end rather than one intermediate stage of it.

The query, key, and value mechanism described here is only the attention sub-layer of a single transformer block. The next chapter picks up immediately after this point, showing how that sub-layer's output is combined with a feed-forward layer, residual connections, and layer normalization, and how many such blocks stack together to form the deep network that actually does the work of turning a prompt into a coherent response.

A transformer block takes in a sequence of vectors, one per token position, and returns a sequence of vectors of the same shape. Inside, the input first passes through the self-attention sub-layer, where each position builds a new representation by combining information from query, key, and value vectors across the sequence. That mechanism is explained in depth in the chapter on self-attention, so here it is treated as a function: it takes the block's input and produces an attention output of the same size, one vector per position, each one now shaped by its neighbors rather than standing alone.

The second sub-layer is the feed-forward layer, sometimes called the position-wise feed-forward network. It takes each position's vector, one at a time, and pushes it through a small two-layer neural network, typically expanding the vector to a larger size, applying a nonlinearity, then projecting it back down to the original size. Unlike attention, this step does not let positions talk to each other at all. Each token's vector is processed independently, using the same learned weights at every position. If attention is where positions exchange information, the feed-forward layer is where each position, now enriched with that borrowed context, gets to think on its own.

It helps to picture this as a two-stage pipeline per block: a mixing stage and a thinking stage. A simplified version of the feed-forward step, ignoring batching and real dimensions, looks like this in code:

def feed_forward(x, w1, b1, w2, b2):
    # x: vector for a single token position
    hidden = relu(x @ w1 + b1)   # expand and apply nonlinearity
    output = hidden @ w2 + b2    # project back to original size
    return output

Applied across a whole sequence, this function runs separately on every position's vector, with the same w1, b1, w2, b2 reused each time. That weight sharing across positions is deliberate: it keeps the parameter count from growing with sequence length, and it means the feed-forward layer learns a general transformation that applies no matter where a token sits in the sequence. Most of a transformer's parameters live here, not in the attention mechanism, since the expansion inside the feed-forward layer typically involves a hidden size several times larger than the model's working dimension.

Both sub-layers, attention and feed-forward, are wrapped in a residual connection. Instead of letting a sub-layer's output replace the input, the block adds the sub-layer's output back onto its own input: output = x + sublayer(x). This single addition solves a problem that made early deep networks hard to train. When many layers are stacked, gradients computed during training have to flow backward through every layer, and without a shortcut path, they tend to shrink or blow up as they pass through each one. The residual connection gives gradients a direct path straight back to earlier layers, bypassing the sub-layer's transformation entirely, which keeps training stable even when the network is dozens of layers deep.

The other piece of bookkeeping is layer normalization. As vectors pass through sub-layer after sub-layer, their scale can drift, growing or shrinking in ways that make training unstable. Layer normalization rescales each vector so its values have a consistent mean and spread, applied independently at each position. In the original 2017 design, normalization was applied after each residual addition, a pattern often called post-norm. Many later models apply it before each sub-layer instead, called pre-norm, which tends to make very deep stacks easier to train. Either way, the job is the same: keep the numbers flowing through the network in a predictable range.

Putting attention, feed-forward, residuals, and normalization together, one full transformer block can be written, in the pre-norm style common in many later models, as a short sequence of steps.

def transformer_block(x, attn_params, ff_params):
    # self-attention sub-layer with residual connection
    normed = layer_norm(x)
    x = x + self_attention(normed, attn_params)

    # feed-forward sub-layer with residual connection
    normed = layer_norm(x)
    x = x + feed_forward(normed, *ff_params)

    return x

Reading this function closely reveals the whole chapter's argument in miniature. The residual addition (x + ...) appears twice, once around attention and once around the feed-forward layer, which is why residual connections are sometimes described as the architecture's safety net. layer_norm appears twice as well, keeping the vectors well-scaled before each sub-layer does its work. Nothing here changes the shape of x. A block takes in a sequence of vectors and hands back a sequence of vectors of exactly the same shape, which is precisely what allows the next block to accept it as input.

That shape-preserving property is not an accident. It is the reason transformer blocks can be stacked at all. If a block changed the dimensionality of its input, stacking twelve or ninety-six of them would require twelve or ninety-six different designs, each matched to the shape coming from the one before. Instead, a single block design, with its own independent set of learned weights, can be repeated as many times as the architecture calls for, with each copy doing the same two-stage transformation on a representation that keeps getting refined.

A common point of confusion is treating the feed-forward layer as an afterthought compared to attention, since attention gets most of the conceptual spotlight. In practice, the feed-forward layer carries a large share of the network's capacity and, by some analyses, a large share of what the model has memorized from training, since it is where position-independent patterns get stored in dense weight matrices^[1]. Attention decides what information to gather; the feed-forward layer is closer to where that information gets turned into a useful transformation.

A short checklist captures the four ingredients covered in this section and the one job each one does inside a block.

• Self-attention sub-layer: lets each position gather context-dependent information from the rest of the sequence.
• Feed-forward layer: processes each position's enriched vector independently, with shared weights across positions.
• Residual connection: adds a sub-layer's input back to its output, giving gradients a direct path through depth.
• Layer normalization: rescales vectors to a consistent range so deep stacks stay numerically stable.

None of these four pieces does anything useful by itself at depth. Attention without residuals would see gradients vanish after a handful of layers. Feed-forward layers without normalization would see activations drift until training diverges. The transformer block earns its name by combining all four into a single repeatable unit, one that can be copied dozens of times without redesigning anything, which is exactly the property the next section builds on.

A full transformer-based language model is, structurally, a long chain of identical transformer blocks, each one feeding its output straight into the next as input. The sequence of vectors enters at the bottom after the embedding lookup table has turned token IDs into vectors, passes through block one, then block two, and so on, until it reaches the final block near the output. Each block has its own independently learned weights for attention and the feed-forward layer, so block five is not doing the same computation as block twelve even though they share the same internal design. What they share is shape, not content.

Why stack blocks at all, rather than building one very wide block that does everything in a single pass? The answer has to do with what each layer can realistically learn. A single layer of attention plus feed-forward processing can capture relatively local or shallow patterns, such as which nearby words modify which others. Stacking blocks lets the network build increasingly abstract representations, where early blocks might resolve local relationships, middle blocks start combining those into phrase-level or clause-level structure, and later blocks assemble something closer to a representation of meaning useful for predicting the next token. This is a rough intuition, not a guaranteed division of labor the network is verified to follow.

Depth here plays a role similar to depth in other kinds of neural networks: convolutional networks used for image recognition stack many layers so that early layers detect edges and textures while later layers detect whole objects, combining the simpler features detected earlier. Transformer depth works on a similar principle applied to sequences of tokens instead of grids of pixels. Each additional block gets a chance to recombine and refine what the previous block produced, which is why models described in papers are often characterized by their number of blocks (sometimes called layers) as a headline architectural detail, alongside their working vector size.

That pyramid is a simplification worth treating carefully. Researchers who probe individual layers of trained models do find evidence that different depths tend to encode different kinds of information, but the boundaries are fuzzy, the pattern is not identical across every model, and no layer is doing only one clean job. Treat the picture as a teaching aid for why depth matters, not as a literal map you could use to predict what block seven of a specific model is doing.

Stacking also compounds the two stabilizing mechanisms from the previous section in a way that matters more as depth grows. A single transformer block's residual connections keep gradients flowing through that one block. Stack ninety-six blocks, and gradients during training have to flow backward through all ninety-six, each with its own residual shortcut. Without those shortcuts at every block, a signal trying to update the earliest blocks based on an error measured at the output would have to pass through dozens of nonlinear transformations, shrinking or distorting along the way. The residual path gives it something closer to a straight line back.

Layer normalization compounds similarly: a small numerical drift introduced by one block, left unchecked, would multiply across every subsequent block it passes through. Normalizing after (or before, in the pre-norm style) every single block resets that drift at every stage, rather than letting ninety-six blocks' worth of drift accumulate before the first correction. This is part of why increasing depth is not simply a matter of copying and pasting more blocks into a model definition. Without the stabilizing design covered in the previous section already in place, adding depth would make training harder rather than more capable.

In practical terms, depth is one of two main levers architects use to make a transformer bigger, the other being width (the size of the vectors each block operates on, and correspondingly the size of the feed-forward layer's expansion). The chapter on scaling laws covers how researchers study the tradeoffs between making a model deeper, wider, or trained on more data, so this chapter leaves that comparison there. What matters here is narrower: depth is achieved by stacking the exact block design covered in the previous section, unchanged in structure, repeated as many times as the architecture specifies.

A concrete way to see the stacking in code is to wrap the single-block function from the previous section in a loop over a list of per-block parameters.

def transformer_stack(x, blocks):
    for attn_params, ff_params in blocks:
        x = transformer_block(x, attn_params, ff_params)
    return x

This loop is, structurally, most of what separates a small language model from a large one in terms of raw architecture. A model with twelve entries in blocks and a model with ninety-six entries in blocks can share the exact same transformer_block function and differ only in how many times the loop runs and how large each block's internal vectors are. The complexity that produces strikingly different capabilities between small and large models is not hidden in some exotic extra mechanism. It emerges from how many times this same repeatable unit gets stacked and how large each copy is, trained on how much data.

One practical consequence for engineers working with model APIs: the number of layers in a model is rarely something you choose or tune directly, since it is fixed at training time by whoever built the model. But understanding that depth is literally a count of repeated blocks helps make sense of release notes and model cards that describe architecture in terms of layer counts and hidden sizes. When a provider describes a smaller or larger variant of the same model family, the difference is frequently exactly this: fewer or more copies of the same block, at a smaller or larger width, rather than a fundamentally different design.

It also explains why inference cost scales roughly with depth. Every token generated has to pass forward through every block in the stack, in sequence, since each block depends on the output of the one before it. A model with twice as many blocks does roughly twice as much computation per token, all else equal. This connects directly to the engineering tradeoffs covered in the chapter on inference, where context window size and the KV cache interact with exactly this per-block, per-token cost to determine how fast and how expensive generation is in practice.

Finally, depth interacts with the residual stream in a way worth naming explicitly. Because every block adds its output onto the running vector rather than replacing it, you can think of the sequence of vectors flowing through the stack as a single running total, continuously updated block by block. Each block reads from that running total, computes something using attention and the feed-forward layer, and adds its contribution back in. By the time the running total reaches the final block, it carries additive contributions from every block that came before it, layered on top of the original embedding. That cumulative structure is a direct consequence of the residual design covered earlier, now visible at the scale of the whole network rather than one block.

The original transformer, introduced in the 2017 paper, was built for machine translation, and its design reflects that task directly. It used an encoder stack, built from the block design covered in this chapter, to process the source sentence, and a separate decoder stack to generate the translated sentence one token at a time, attending both to its own previous outputs and to the encoder's representations^[2]. The decoder blocks included an extra attention sub-layer beyond the self-attention and feed-forward layers already described, specifically to attend over the encoder's output, a mechanism called cross-attention.

Most of today's large language models, including the kind this book focuses on, use only the decoder half of that original design, often called a decoder-only architecture. There is no separate encoder processing a source sentence, because the task is not translation but next-token prediction over a single stream of text. Each block in a decoder-only model contains self-attention and a feed-forward layer, wrapped in residual connections and layer normalization, the same four-part structure described above, with no cross-attention sub-layer at all. The lineage from the 2017 paper is visible in every block, even though the overall shape of the network has narrowed to one stack instead of two.

One detail worth being specific about, since it shapes how self-attention behaves inside a decoder-only block: the attention mechanism in a decoder is restricted so that a given position can only attend to itself and to positions before it, never to positions later in the sequence. This restriction, often implemented with a mask that blocks forward-looking connections, is what keeps next-token prediction honest. Without it, a block generating a prediction for position five could peek at the actual token sitting at position six, which would make the training objective meaningless since the model would simply be copying an answer it was never meant to see yet.

The table below summarizes how the original 2017 design maps onto the decoder-only architecture most current language models use, to make the lineage and the differences concrete rather than abstract.

What the table makes clear is that the core block design, attention plus feed-forward wrapped in residuals and layer normalization, survived essentially unchanged across a major shift in application, from translating sentences to predicting tokens across general text. That durability is unusual in a field that otherwise moves quickly. Most of the visible evolution in language models since 2017 has come from scale, training data, and the fine-tuning and alignment techniques covered later in this book, rather than from replacing the transformer block with something structurally different.

That said, the block has not stood perfectly still. The pre-norm versus post-norm choice mentioned earlier is one widely adopted refinement, generally improving training stability at greater depth compared with the original post-norm placement. Other refinements touch the feed-forward layer's internal nonlinearity, the exact form of layer normalization, and how attention is computed efficiently at long sequence lengths, but these are adjustments to the block's internals, not departures from its four-part structure. An engineer who understands attention, feed-forward layers, residuals, and normalization from this chapter can read the architecture section of most model papers published since 2017 and recognize nearly everything described.

A short timeline situates the transformer block relative to the broader arc this book follows, without repeating the detail owned by later chapters.

Seeing the block in this longer arc also clarifies what the rest of this book does and does not add to the picture. Pretraining, covered next, is about what objective trains the weights inside each block's attention and feed-forward layers. Scaling laws are about how many blocks, and how wide, tend to pay off given a training budget. Fine-tuning and alignment do not touch the block's structure at all. They continue adjusting the same weights, inside the same attention and feed-forward layers, toward a different objective than raw next-token prediction on general text.

This is a useful filter when reading claims about new model architectures. A paper describing a genuinely new attention variant, a different feed-forward activation function, or a new normalization scheme is proposing a refinement to one piece of the block's four-part structure. A paper describing a model with more parameters, more layers, or more training tokens is operating one level up, in the territory of scaling laws, using the same block design underneath. Very few papers propose discarding the block entirely, and the ones that do tend to get compared directly against it as the established baseline, precisely because of how well it has held up since 2017.

For an engineer building on top of model APIs rather than training models from scratch, none of this changes what calls look like or what parameters are exposed. But it changes what a term like 24-layer model or decoder-only architecture means when it shows up in a technical report or a provider's documentation. It is a direct, countable reference to how many times the same block has been stacked, built from a design whose core logic, self-attention gathering context, a feed-forward layer processing it, residual connections preserving gradient flow, and layer normalization keeping the numbers well-behaved, has carried a research field for years without a structural replacement emerging.

The next chapter turns from structure to training, asking what objective actually shapes the weights sitting inside every attention and feed-forward layer in every block of the stack. The architecture covered here is the vessel. Pretraining is what fills it.

Pretraining is the stage where a transformer learns the statistics of language from huge amounts of unlabeled text, before any instruction-following behavior is added. The task given to the model during pretraining is deceptively narrow: look at a sequence of tokens and predict the next one. This is called autoregressive generation when the model is producing text, and the matching training setup, where the model is trained to predict each token from the ones before it, is what this book calls autoregressive pretraining. Despite the narrowness of the task, predicting the next token well over a huge, varied corpus forces the network to pick up grammar, facts, reasoning patterns, and style, because all of those things help predict what comes next.

The elegance of this setup is that it turns ordinary text into a self-labeling dataset. Take any passage, for example a paragraph from a novel, a function from an open-source repository, or a paragraph from a news article. Split it into tokens, and at every position in that sequence there is a "correct answer" sitting right there: the token that actually comes next in the real text. No human has to annotate anything. A training example is simply a chunk of text, and the target at each position is just the next token in that same chunk, shifted by one. This is why pretraining can scale to a volume of text that would be impossible to label by hand.

Concretely, imagine the sentence the cat sat on the mat tokenized into six tokens, roughly one per word for a simple case like this. The model is handed the first token, the, and asked to predict the second, cat. Then it is handed the cat and asked to predict sat. Then the cat sat predicting on, and so on, until the end of the sequence. Each of these is a separate prediction opportunity, and a well-implemented training loop computes all of them for a sequence in a single forward pass, rather than looping position by position, which is part of why transformers train efficiently on long sequences.

That single forward pass works because of a constraint baked into the architecture: each position can only attend to itself and the positions before it, never the ones after. This is sometimes called causal masking, and it guarantees that the prediction for position i never cheats by looking at the real token that comes at position i or later. Without that constraint, the model could trivially "predict" the next token by just copying it from later in the sequence, which would make the loss meaningless. The attention mechanism covered in the self-attention chapter is adapted for pretraining precisely by adding this forward-looking blindfold.

It helps to be concrete about scale here without pinning down specific published numbers. A single training run processes an enormous corpus, often assembled from web pages, books, code repositories, and other text sources, broken into fixed-length chunks that fit the model's context window. Each chunk yields one next-token prediction per position, so a modest batch of sequences can generate many thousands of individual prediction tasks in one step. This is why pretraining is almost always described in terms of total tokens processed rather than total documents, since the unit of learning is the token-level prediction, not the document as a whole.

A useful mental shortcut is to think of autoregressive pretraining as an extremely large, extremely repetitive fill-in-the-blank exercise. The blank is always the very next token, never a token in the middle of a sentence and never a whole missing phrase. That is different from some other kinds of self-supervised training used in other fields, where a chunk in the middle of the input is masked and the model fills it in using context from both sides. Decoder-only language models of the kind this book focuses on stick to the left-to-right, next-token version, which happens to match how text is generated one token at a time at inference.

The practical upshot for an engineer reading a model card or a paper is that pretrained on text data almost always means exactly this setup: next-token prediction across a huge, usually web-scale, corpus of raw text, with no human-written labels beyond the text that was already there. Recognizing that lets you reason about what a pretrained model is actually good at out of the box. It has absorbed patterns of language and, incidentally, a great deal of factual and procedural knowledge, but it has not yet been taught to follow instructions or prefer one kind of answer over another, which is the job of the fine-tuning stage covered later in this book.

One subtlety worth sitting with is that the model never "sees" words or sentences as such during this process; it sees token IDs, consistent with how the tokenizer chapter describes subword units. A single pretraining example might include a token that is a whole common word, a token that is only a fragment of a rarer word, and a token that is a single punctuation mark, all mixed together in one sequence. The next-token objective applies uniformly to all of them: predict the ID of the next token, whatever kind of text fragment it happens to represent. This is one reason token counts and word counts diverge, since the model is being scored on tokens, not words.

"Raw text" in practice is more varied than the phrase suggests. Pretraining corpora typically mix genres deliberately, blending encyclopedic writing, conversational forums, source code, and long-form books, because each genre teaches the model different regularities. Code teaches strict syntactic structure and long-range dependencies between a variable's definition and its later use. Conversational text teaches informal phrasing and turn-taking patterns. This diversity is part of why a single pretrained model ends up competent across such a wide range of tasks, since the underlying objective rewards any text pattern, structured or not, that helps predict the next token.

A common misconception worth correcting directly is that the model is memorizing specific sentences in order to reproduce them later. For most of a large corpus, that is not what minimizing the loss rewards. Memorizing one exact passage helps predict the next token for that one passage, but it does nothing for the next million passages the model will also be trained on. The optimization process favors weights that generalize, that is, weights encoding patterns which transfer across many different sequences, because those patterns reduce the average loss over the whole dataset, not just one sequence.

That said, exact memorization does happen for some content, especially text that appears many times across the training corpus or text that is highly distinctive and hard to predict from general patterns alone. This is a real engineering consideration for anyone deploying a model in a setting where verbatim reproduction of training text would be a problem, and it is one of the reasons data curation and filtering are taken seriously before a pretraining run begins. The next-token objective does not distinguish between learning a general grammatical pattern and memorizing a rare string; it simply rewards whatever reduces the prediction error, and sometimes memorization is the cheapest way to do that for unusual snippets.

Stepping back, autoregressive pretraining is best understood as a very large-scale, very repetitive supervised learning problem, where the supervision signal is extracted for free from ordinary text rather than hand-labeled. The next two subchapters pick up exactly where this leaves off: first by making precise what it means to "measure how wrong" a next-token prediction is, using cross-entropy loss, and then by walking through the mechanical steps of a training loop in Python that actually performs this prediction-and-correction cycle.

Once the model predicts the next token, there has to be a number that captures how good or bad that prediction was, something the training loop can use to decide which direction to push every weight. That number is the cross-entropy loss, and understanding it in plain terms removes a lot of the mystery around phrases like minimizing the loss that show up constantly in papers and blog posts. At a high level, cross-entropy measures how surprised the model was by the actual next token, given the probability distribution over the vocabulary it had just produced.

Recall from the chapter on next-token prediction that the model's final layer produces a probability distribution over vocabulary, one number for every possible token ID, all summing to one. Suppose the real next token in the training text was cat, and the model's output distribution assigned a probability of 0.7 to cat. Cross-entropy loss for that single prediction is defined as the negative logarithm of the probability assigned to the correct token.

loss = -log(p(correct token))

Plugging in p = 0.7 gives a small positive loss, because the natural logarithm of a number close to one is close to zero, and the negative sign flips it to a small positive value. Now suppose instead the model had been badly wrong and assigned only 0.01 probability to the correct token cat, perhaps because it predicted something like dog with high confidence instead. The negative log of 0.01 is much larger than the negative log of 0.7, so the loss for that prediction is much higher. This is the essential shape of cross-entropy: confident correct predictions get a loss near zero, and confident wrong predictions get punished heavily, growing without bound as the assigned probability approaches zero.

That asymptotic behavior is deliberate and important. A model that is only mildly unsure, say it splits probability 0.4 and 0.3 between two plausible next tokens, is not punished nearly as hard as a model that is confidently wrong, assigning 0.01 to the right answer while putting 0.95 somewhere else. Cross-entropy loss specifically penalizes overconfident mistakes far more than honest uncertainty, which is a useful property for a training signal, because it discourages the model from committing too hard to a guess when the text genuinely does not disambiguate the next token.

In practice, a training batch contains many sequences, and each sequence has many token positions, each with its own next-token prediction and its own loss value. The number reported as "the loss" for a training step is almost always the average of all these individual cross-entropy values across every position in the batch. Averaging matters because batch sizes and sequence lengths vary across experiments, and an average loss per token is comparable across those variations in a way that a raw summed loss is not.

It is worth connecting this back to the probability distribution over vocabulary directly, since cross-entropy only makes sense in terms of it. The training signal, meaning the feedback used to update the weights, comes entirely from comparing the model's predicted distribution to the one true next token at each position. There is no separate "correctness checker" or external judge involved at this stage; the correct answer is simply read off the training text itself, and the loss is purely a function of how much probability mass the model put on that one correct token.

A helpful way to build intuition is to imagine the cross-entropy loss as a measure of surprise, borrowed from information theory, where rare or unexpected outcomes carry more "surprisal" than expected ones. If the model is well-calibrated, meaning its probability estimates genuinely reflect how predictable each next token really is given the context, then its average surprise, and therefore its average loss, will be low. This is why a lower loss on held-out text is used as the standard way to compare pretrained models of different sizes or trained on different amounts of data, a practice central to the scaling studies covered in a later chapter.

One common point of confusion is mixing up cross-entropy loss with accuracy. Accuracy would ask a binary question: did the model's single highest-probability guess match the correct token, yes or no? Cross-entropy asks a more nuanced question: how much probability mass did the model put specifically on the correct token, regardless of whether that token happened to be the single most likely guess. A model can have reasonable accuracy while still having a fairly high loss, if it tends to spread probability thinly across several plausible candidates even when it eventually guesses right, and a model can have low loss while occasionally being wrong on genuinely ambiguous positions where even a very well-trained model should not be fully confident.

Perplexity is a related number that often shows up alongside loss values in papers, and it is worth demystifying briefly since the term appears constantly. Perplexity is simply the exponential of the average cross-entropy loss, and it can be read loosely as the effective number of roughly equally likely choices the model is weighing at each position. A perplexity near one would mean the model is almost always certain of the exact next token, while a higher perplexity means the model is distributing its guesses across a wider effective set of candidates. Lower loss always corresponds to lower perplexity, so the two measures tell the same story in different units.

Minimizing cross-entropy loss carries specific guarantees, and specific blind spots, that run through a recurring theme of this book. Driving the average loss down across a training corpus makes the model better at matching the statistical patterns of that corpus, including its grammar, common phrasing, and the kinds of facts that show up frequently and consistently in the data. It does not by itself guarantee the model will be truthful, helpful in a conversational sense, or safe to deploy without further shaping, since the raw objective never distinguishes between useful knowledge and confidently repeated misinformation in the data; that distinction is left to fine-tuning and alignment, covered later.

A worked numerical example helps cement the relationship between probability and loss before moving to code. Suppose the vocabulary has only four tokens for simplicity, and the model's output distribution for a given position is p(a)=0.5, p(b)=0.3, p(c)=0.15, p(d)=0.05, and the real next token in the training text is b. The loss for this one position is -log(0.3), which is roughly 1.20 in natural log units. If instead the correct token had been d, the loss would be -log(0.05), roughly 3.00, a much larger penalty reflecting how unlikely the model thought that outcome was.

Holding this worked example in mind makes the next subchapter's code much easier to follow, because the training loop is, at its heart, nothing more than repeating this same calculation across vast numbers of positions, then using the size and direction of the resulting loss to decide how to adjust every weight in the network. The loss number is the single scalar that ties the entire enormous computation graph, embeddings, attention, feed-forward layers, and all, back to one simple question at every step: how probable did the model think the true next token was.

With the task defined as autoregressive next-token prediction and the scoring defined as cross-entropy loss, the remaining piece is the mechanical process that uses that loss to actually change the weights. This process is called the training loop, and despite the scale of real-world pretraining runs, the loop itself follows a short, repeatable pattern that fits comfortably in a few dozen lines of code. The steps below focus on structure and intent rather than a runnable, fully optimized implementation, since the goal is to recognize this pattern when reading real training code, not to reproduce industrial-scale infrastructure.

Translating that into code makes the pattern concrete. The snippet below uses PyTorch-style pseudocode, close enough to real PyTorch that it would run with minor adjustments, to show the loop end to end for a single training step, assuming a model object that already implements the transformer architecture from the previous chapters.

import torch
import torch.nn.functional as F

def training_step(model, optimizer, batch_token_ids):
    # batch_token_ids has shape (batch_size, sequence_length)
    inputs = batch_token_ids[:, :-1]   # all tokens except the last
    targets = batch_token_ids[:, 1:]   # all tokens except the first

    # forward pass: logits has shape (batch_size, seq_len - 1, vocab_size)
    logits = model(inputs)

    # reshape so cross_entropy can compare each position independently
    logits_flat = logits.reshape(-1, logits.size(-1))
    targets_flat = targets.reshape(-1)

    loss = F.cross_entropy(logits_flat, targets_flat)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    return loss.item()

Walking through this line by line connects every earlier concept to a concrete line of code. The split between inputs and targets is exactly the shifted-by-one pattern described in the first subchapter: input position zero pairs with target position zero, which is actually the token originally at position one in the raw sequence. The call model(inputs) runs the full stack of transformer blocks from the previous chapter, ending in logits, raw scores that get turned into the probability distribution over vocabulary through a softmax step typically folded inside F.cross_entropy itself for numerical stability, so it is not written out separately here.

The reshaping step deserves a moment of attention, since it trips up a lot of newcomers reading real training code for the first time. F.cross_entropy in PyTorch is simplest to call with a two-dimensional tensor of predictions and a one-dimensional tensor of target class indices, not the three-dimensional batch-by-sequence-by-vocabulary shape that naturally comes out of a transformer. Flattening the batch and sequence dimensions together just means treating every single token position, across every sequence in the batch, as one independent classification problem: given this position's context, which of the vocabulary's token IDs is correct. That reframing is exactly the worked example from the previous subchapter, just applied simultaneously across thousands of positions instead of one.

loss.backward() is where backpropagation happens, the algorithm that computes, for every single weight in the network, how much a tiny change in that weight would change the final loss value. This is standard gradient-based optimization, not something specific to language models, but it is worth naming explicitly because it is the step that actually makes learning possible. Without it, there would be no way to know which direction to move millions or billions of individual numbers to make the next-token predictions better.

optimizer.step() then applies those computed gradients, nudging each weight by a small amount in the direction that should reduce the loss, with the exact amount controlled by a learning rate and by the specifics of whichever optimization algorithm is in use, commonly a variant called Adam for transformer training. The size of that nudge matters enormously in practice. Too large, and training can become unstable, with the loss spiking instead of decreasing. Too small, and training crawls, requiring far more steps to reach a useful loss value, which directly translates into more compute cost for a pretraining run.

optimizer.zero_grad() is easy to overlook but important: gradients accumulate by default in frameworks like PyTorch, so without explicitly clearing them before each backward pass, the gradients from the previous training step would mix into the current one, corrupting the update. This is a common source of subtle bugs for engineers writing their first training loop from scratch, and it is worth checking for explicitly if a custom loop produces a loss curve that behaves strangely.

Scaling this single-step function into a full pretraining run is mostly a matter of repetition and data logistics rather than new concepts. A real training script wraps this training_step function in an outer loop that keeps pulling fresh batches from a data loader, which itself streams tokenized text from the pretraining corpus, cycles through the entire dataset potentially multiple times, and periodically saves checkpoints of the model's weights so that an interrupted run can be resumed. Logging the loss value at every step, or every handful of steps, produces the loss curves that show up constantly in papers, almost always trending downward and flattening as training progresses.

Reading a loss curve well is itself a practical skill worth building. A healthy pretraining run shows a loss that starts relatively high, since the model begins with randomly initialized weights and therefore near-random predictions, then drops quickly in the earliest steps as the model picks up the most common statistical patterns in the corpus, such as frequent word pairs and basic grammar. After that initial drop, the loss keeps decreasing but at a slower and slower rate, a pattern that connects directly to the diminishing-returns shape that shows up in the empirical scaling studies covered in the next chapter. A flat or rising loss partway through a run is a signal worth investigating, whether that points to a learning rate problem, a bad batch of corrupted data, or a numerical instability in the optimizer.

Stepping back from the code, it is worth recognizing how much of modern engineering work around large language models sits on top of this exact loop, scaled up by orders of magnitude in data, model size, and hardware, but conceptually unchanged. Understanding this loop demystifies statements in papers like trained for a fixed token budget or loss plateaued after a certain number of steps, since those are just descriptions of this same predict-score-adjust cycle running at a scale that fills data centers rather than a single laptop. The next chapter picks up exactly this thread, examining what happens to the loss curve as model size and data size grow together, and what those empirical scaling trends can and cannot promise about the resulting model's behavior.

A scaling-law chart almost always uses log-log axes, meaning both the horizontal axis (compute, parameters, or tokens) and the vertical axis (loss) are plotted on logarithmic scales rather than linear ones. This choice is not cosmetic. A relationship of the form loss ≈ A · x-α, where x is compute or size and α is a small positive exponent, becomes a straight line once both axes are logarithmic, because taking the logarithm of both sides turns the power law into a linear equation. Seeing a straight line on a log-log plot is the visual signature of a power law, and it is the reason these charts look so clean compared to almost anything else in machine learning.

Concretely, suppose doubling the training compute reduces loss by a fixed fraction each time, rather than by a fixed amount. That is the signature of a power law: constant percentage improvements per doubling, not constant absolute improvements. Early doublings buy large visible gains because the model moves off a poor starting point, while later doublings buy the same percentage improvement but it looks smaller on the page because the baseline loss is already low. This is exactly why diminishing returns are often misread as a plateau: the curve has not leveled off, the improvements are still there, but they are shrinking proportionally and require ever larger compute increases to notice.

The chart below sketches this pattern for a hypothetical training setup, holding tokens and architecture choices roughly constant while compute grows across a wide range. The shape, steep at first and flattening on a linear scale, is the same information as a straight line on a log-log plot, just viewed differently.

Three quantities show up again and again in these studies, and it is worth keeping them straight. Model size is usually measured in number of parameters, essentially the count of learnable weights spread across the embedding table, attention projections, and feed-forward layers inside each transformer block. Dataset size is measured in training tokens, the same unit produced by the tokenizer and consumed one at a time during the training loop described earlier. Compute combines both of these, since the floating-point operations spent on training scale roughly with the product of parameters and tokens processed. Scaling laws describe how loss responds when one or more of these three move.

A useful mental habit when reading a scaling-law chart from a paper or blog post is to ask three questions before trusting the conclusion drawn from it. First, which quantity is varying on the x-axis, and which two are held fixed or allowed to vary alongside it. Second, what is the loss actually measuring: held-out cross-entropy on a specific kind of text, or some downstream benchmark score, since the two do not always move together. Third, over what range of scale was the curve actually fit, since extrapolating a straight line far beyond the measured range is an assumption, not a finding. Skipping these questions is how scaling claims get overstated in casual conversation.

It is also useful to notice what a scaling law is silent about. The curve reports an average loss over a broad mixture of text, which is not the same as performance on a narrow task a particular application cares about, such as extracting structured data from invoices or writing correct code in a specific language. Two models can sit at the same point on the aggregate loss curve and still differ meaningfully on a task that matters to a specific product. A loss curve is a strong signal about the overall quality of next-token prediction across the training distribution, not a certificate for any individual capability.

The following callout distills the reading habit into something checkable before trusting a scaling claim from a paper or marketing page.

These trends emerged from running many training jobs at different scales and fitting curves to the results, not from a theoretical derivation of how neural networks must behave. That empirical grounding is part of why the findings have been so influential: they describe what was actually observed across a wide sweep of model sizes, dataset sizes, and compute budgets, rather than a prediction from first principles^[2]. It also explains why later studies, run with different data mixes or architectural tweaks, sometimes found somewhat different exponents or constants. The qualitative shape, a power law with diminishing but persistent returns, has proven durable even as the exact numbers have shifted.

A common mistake among engineers new to this material is treating the exponent in a scaling law as a universal constant, something like the speed of light for language models. It is closer to a measured property of a particular family of training runs, under a particular tokenizer, data mixture, and architecture. Change any of those and the curve can shift, sometimes noticeably. Treat published scaling exponents as a guide for reasoning about trends and planning experiments, not as a number to plug into a spreadsheet and expect an exact prediction from.

Pretraining loss and post-fine-tuning or alignment metrics are distinct targets. The curves discussed here describe the autoregressive pretraining phase, where the objective is still pure next-token prediction over a broad corpus. Once a model goes through supervised fine-tuning and reinforcement learning from human feedback, its behavior on instructions and preferences depends on choices made in that later stage too, and those choices do not follow the same clean power-law curves that pretraining loss does.

One more subtlety: loss is continuous and smooth, but capabilities often look discontinuous. A model might show no measurable ability to perform multi-step arithmetic until it crosses some scale threshold, then suddenly perform it reliably, even though its aggregate loss curve looks perfectly smooth the entire time. This gap between smooth loss and seemingly jumpy capabilities is an active area of debate, and part of why scaling laws, while useful for planning compute budgets, are a poor tool for predicting exactly which skills a model will or will not have at a given size.

Put together, reading a scaling trend well means treating it as a compass for budgeting experiments and compute, not a crystal ball for predicting a specific model's behavior on a specific task. The next section turns this reading skill into an actual decision: given a fixed compute budget, how should it be divided between making the model bigger and feeding it more data.

For several years after the earliest scaling-law work, a common practice in the field was to make models as large as a compute budget would permit and then train each one on a comparatively modest, roughly fixed number of tokens. The logic seemed sound given the curves available at the time: if loss falls as model size grows, pour the budget into parameters and keep the token count manageable so training finishes in a reasonable amount of time. Several influential large language models were trained this way, with parameter counts growing much faster than token counts across successive model generations.

The Chinchilla study revisited this practice by running a much more systematic sweep: training many models of different sizes, each on different amounts of data, and carefully tracking which combination achieved the lowest loss for a given fixed compute budget. The headline finding reshaped how the field thinks about building large models. Under its compute-optimal training analysis, the study found that model size and the number of training tokens should scale together, roughly in step with one another, rather than growing model size much faster than data^[3]. In plain terms, many of the largest models trained before this study were undertrained relative to their size. They had plenty of parameters but had not seen enough text to use that capacity efficiently.

The practical consequence was striking: a noticeably smaller model, trained on substantially more tokens for the same total compute budget, could achieve lower loss than a much larger model trained on fewer tokens. This gave rise to the shorthand now known as the Chinchilla ratio, an approximate proportion between model parameters and training tokens that tends to make the most of a given compute budget. Engineers now use this ratio as a sanity check: if a model size is proposed without a commensurately scaled training dataset, it is worth asking whether that plan is compute-optimal or simply inherited from older conventions.

A useful way to picture the trade-off is as a fixed budget split two ways. Imagine the training compute as a fixed amount of money, and two things to buy with it: a bigger model (more parameters, more cost per token processed) or more data (more tokens, more passes needed through the network to see them). Spend it all on a bigger model and fewer tokens, and it finishes training quickly but leaves performance on the table, because the extra parameters are not matched with enough examples to be fully useful. Spend it all on a huge token count and a small model, and the model saturates, meaning it runs out of capacity to use the extra data productively. Compute-optimal training sits at the balance point between these two failure modes.

The table below sketches, in illustrative terms, how this trade-off plays out across three hypothetical training plans that all consume roughly the same total compute budget.

It is worth walking through what this looks like as an engineering decision rather than an abstract curve. Suppose a team has a fixed compute budget and is deciding between two plans: Plan A trains a larger model on a smaller slice of available data, finishing sooner, while Plan B trains a smaller model on a much larger slice of the same data, taking about the same total compute because the extra tokens processed balance out the smaller per-token cost. A compute-optimal analysis, in the spirit of the Chinchilla findings, would favor whichever plan actually lands closer to the matched parameter-to-token ratio for that compute budget, which in many realistic cases turns out to be the smaller, more thoroughly trained model rather than the larger, lightly trained one.

This reasoning also explains a pattern visible across the industry after the Chinchilla study's publication: subsequent models from several labs shifted toward comparatively smaller parameter counts paired with much larger training datasets, chasing the compute-optimal point rather than raw parameter counts as a headline number. A smaller compute-optimal model also tends to be cheaper to run at inference time, since inference cost scales with parameters processed per token generated. That makes the compute-optimal trade-off attractive for reasons beyond training efficiency: it can also reduce the cost of every single API call made against the deployed model afterward.

None of this means parameter count stopped mattering. A compute-optimal small model trained on vast data can still be outperformed in absolute terms by a much larger model trained compute-optimally with a proportionally larger token count too, since the budget itself, not just the ratio, sets the ceiling on achievable loss. The Chinchilla ratio answers the question of how to spend a fixed budget well, not the separate question of how large a budget to commit to in the first place. Both decisions matter, and conflating them is a common misreading of the finding.

Here is a short worked sketch to make the ratio concrete, using illustrative round numbers rather than any published measurement. Suppose a compute budget supports training a model of a given parameter count on a proportionally scaled number of tokens, and doubling the parameter count while keeping the token count fixed would require roughly double the compute just to maintain the same per-token exposure, let alone add more tokens to match the larger model. A compute-optimal plan would instead split that doubled budget between a moderately larger model and a moderately larger token count, rather than spending it entirely on size. The exact split depends on the fitted exponents from a scaling study, but the directional lesson, that tokens need to grow roughly alongside parameters, is the one engineers should carry forward.

The steps below summarize how a team might apply this thinking when scoping a training run, without needing to rederive the underlying curves from scratch.

Even this careful planning process is still an estimate, not a precise formula. Real training runs face practical constraints that pure scaling curves do not capture: licensing and quality limits on available data, hardware memory limits that cap feasible parameter counts, and diminishing data quality once the easiest, highest-quality sources are exhausted and lower-quality sources get folded in to hit a larger token target. The Chinchilla ratio is a strong prior for planning, not a substitute for running the smaller validation experiments described in the steps above.

The single most important caveat about scaling laws is also the simplest: they describe trends measured within a particular setup, and nothing guarantees those trends hold exactly once key assumptions change. A curve fit using one data mixture, one tokenizer, and one architecture family does not automatically transfer to a different mixture, a different vocabulary, or a different architectural choice. Treating a scaling exponent as a universal physical constant that applies to every model, every dataset, and every objective is a misreading of what these studies actually show, and it is one of the more common mistakes made when citing this research casually.

A second limit concerns what the loss metric itself measures. Cross-entropy loss, introduced in the previous chapter, captures how well the model predicts the next token across a broad sample of held-out text. It is an excellent proxy for overall language modeling quality, but it is an average, and averages hide variance. A model can improve its aggregate loss while staying flat, or even getting slightly worse, on a narrower task that matters for a particular application, such as following a specific output format or avoiding a specific kind of factual error. Scaling laws say almost nothing directly about these narrower, task-specific behaviors.

This disconnect between smooth loss curves and uneven downstream performance is one reason benchmark results can feel surprising even when training followed the scaling script closely. A team might train a compute-optimal model exactly as the curves recommend, watch loss fall exactly as predicted, and still find that a downstream evaluation relevant to their product barely moved, or moved in an unexpected direction. This is not a contradiction of scaling laws, it is a reminder that loss and downstream capability are correlated but not identical measurements, and the gap between them widens for narrower or more specialized tasks.

A third limit involves the data side of the equation more directly. Scaling laws for dataset size generally assume that additional tokens are roughly as useful, on average, as the tokens already used, which works reasonably well early on when plentiful high-quality text is available. As a training target pushes toward using far more tokens, maintaining that same average quality becomes harder, since the easiest, cleanest sources get exhausted and lower-quality or more repetitive sources fill the gap. A scaling curve fit on earlier, higher-quality data does not automatically predict what happens when later increments of data are noisier or more repetitive, a limitation that compute-optimal planning has to account for separately from the pure ratio.

Scaling laws also say nothing about safety, factual reliability, or alignment with instructions, each of which depends on later stages of training covered in upcoming chapters. A larger, compute-optimally trained model is not automatically a model that hallucinates less, follows instructions more faithfully, or better reflects any particular set of values. Those properties are shaped substantially by fine-tuning and reinforcement learning from human feedback, applied after pretraining concludes, and they follow their own dynamics rather than inheriting a clean power-law relationship with scale. A team hoping that simply scaling up will quietly fix a behavioral problem is usually disappointed.

Another boundary worth naming explicitly: scaling studies are typically run under controlled, well-resourced conditions, with careful data curation and significant engineering effort devoted to stable training. A smaller team working with messier data, less hardware, or less time to tune hyperparameters should expect more noise and less clean adherence to the published curves. This does not make scaling laws useless outside a handful of well-funded labs, it just means the curves describe a best case under favorable conditions, and real-world deviations from that best case are the norm rather than the exception.

There is also a forecasting trap worth naming directly: extrapolating a fitted curve far beyond the range of scale it was actually measured on. A line that fits beautifully across several orders of magnitude of compute can, in principle, bend, flatten, or even reverse direction outside that range, for reasons that include running out of high-quality data, hitting hardware or optimization limits, or encountering some structural change in how the model represents information at very different scales. Nothing in the mathematics of a power-law fit guarantees it continues indefinitely, which is why the most careful scaling papers are explicit about the range over which their curves were actually tested.

Put together, these limits argue for a specific, disciplined posture toward scaling-law claims rather than blanket skepticism or blanket trust. Use published curves to plan compute budgets and sanity-check the rough split between model size and training tokens, since that is exactly the question they were designed to answer. Do not use them to predict a specific downstream capability, to assume a problem like hallucination will shrink automatically with scale, or to extrapolate confidently far past the measured range.

None of this diminishes why scaling laws earned their place as a standard planning tool. They replaced guesswork with measurement, gave teams a principled way to size training runs before spending months of compute, and corrected a real inefficiency in how early large models split their budgets between parameters and data. The discipline they require, reading curves carefully, respecting their measured range, and never mistaking a loss trend for a promise about specific downstream behavior, is exactly the kind of grounded skepticism that carries forward into the next stage of the pipeline, where fine-tuning and alignment take a pretrained model and shape it toward following instructions.

Fine-tuning means taking a model that has already gone through pretraining and continuing to train it, usually on a smaller, more targeted dataset, with a different goal in mind. The weights are not reset. The model keeps everything it learned about grammar, facts, and style during pretraining, and the fine-tuning stage nudges those weights toward a narrower kind of behavior. This matters practically: fine-tuning a model that already predicts tokens well is far cheaper than training one from scratch, because the model is not learning language from zero, it is learning a new way to use language it already has.

The most direct form of fine-tuning is supervised fine-tuning (SFT). Here, the training data is no longer raw internet text but a curated set of prompt-and-response pairs, where each response was written or approved by a human to demonstrate the desired behavior. If the goal is an assistant that answers questions directly instead of continuing them with more questions, the SFT dataset consists of many examples of exactly that: a question, followed by a direct, well-formed answer. The model is trained with the same underlying mechanics as pretraining, predicting the next token and adjusting weights by cross-entropy loss, but now the training text comes from a different source and the model's output distribution shifts to look like those demonstrations.

Picture a pretrained model that, given the prompt Explain what a stack overflow is, might continue with a list of unrelated forum questions, because that is what often follows similar text on the open internet. After supervised fine-tuning on demonstrations of direct explanations, the same prompt produces a clear, self-contained answer, because the model has seen many examples of that pattern and the next-token predictions that write questions-as-continuations have been pushed down in probability relative to the next-token predictions that write direct answers.

Building an SFT dataset is itself an engineering and labor task, and the quality of that dataset directly limits the quality of the fine-tuned model. A sloppy dataset with inconsistent answer lengths, inconsistent tone, or factual errors will teach the model those inconsistencies just as reliably as it teaches good behavior. Teams building SFT datasets typically write detailed guidelines for human writers or reviewers, covering things like preferred answer length, how to handle requests for medical or legal information, and what tone to use. This is the first place where value-laden choices enter the pipeline, since someone has to decide what a good answer looks like before a single example is written.

SFT alone has a real limitation: it only teaches the model to imitate the exact demonstrations it was shown. It cannot, by itself, tell the model that one plausible answer is better than another slightly different plausible answer unless both appear explicitly in the training data. If a model generates two decent responses to the same prompt and one is clearly more helpful than the other, SFT has no mechanism to express that preference, because its training signal is just match this demonstration, not this output is better than that one. That gap is what the next subchapter's technique, RLHF, is designed to close.

It helps to be concrete about scale and cost here. Pretraining datasets are enormous and largely unlabeled, drawn from broad sources of text. SFT datasets, by contrast, are much smaller and expensive per example, because each one typically requires a human to write or carefully vet a response. This tradeoff is normal across machine learning: pretraining gets breadth cheaply, fine-tuning buys precision at a much higher cost per example. An engineering team building a narrower assistant (say, one that only answers questions about a specific internal codebase) might fine-tune on a few thousand carefully written examples rather than attempting anything close to the scale of pretraining data.

Fine-tuning is not limited to instruction-following. The same mechanism (continue training a pretrained model on a narrower dataset) is used for things like making a model better at a specific domain, such as legal or medical text, or teaching it a particular output format, such as structured JSON. SFT for instruction-following and conversational behavior is a special case of fine-tuning, and the one most relevant to the chat assistants readers are likely building with or on top of.

One practical consequence for engineers using commercial APIs: when a provider offers a fine-tuning option, it usually means supplying your own prompt-and-response examples, which the provider uses to continue training a version of their base model for your specific use case. This is the same mechanism described here, just packaged as a service. The quality guidance above applies directly: a fine-tuning job built on inconsistent or low-quality examples will produce a model that reproduces those flaws, often in ways that are hard to notice until it is already in production.

A common mistake among teams new to fine-tuning is assuming it can fix a model that is fundamentally missing knowledge. If a pretrained model never encountered information about a narrow domain during pretraining, fine-tuning on a handful of example conversations about that domain will not reliably inject that knowledge. Fine-tuning reshapes behavior and style far more effectively than it adds new facts, because the number of examples is small relative to the scale needed to shift a model's underlying knowledge. For injecting fresh or proprietary knowledge, retrieval-based approaches that supply relevant text inside the prompt are usually a better fit than fine-tuning, though that is a separate engineering decision outside the scope of this chapter.

Another common mistake is conflating fine-tuning with prompting. Prompting changes what you ask a model at inference time, leaving the weights untouched. Fine-tuning changes the weights themselves through additional training. The two are not competitors so much as tools with different costs and different kinds of leverage. Prompting is cheap, fast to iterate, and reversible in seconds. Fine-tuning requires a dataset, a training run, and produces a new set of weights that behaves differently by default, without needing the same instructions repeated in every prompt. Many production systems use both: a fine-tuned model for a stable baseline behavior, combined with prompting for task-specific adjustments on top.

The distinction between pretraining and fine-tuning also clarifies why open-source base models and chat-tuned variants of the same architecture can behave so differently. A base model released after only pretraining will often continue text in ways that feel strange or unhelpful for conversational use, simply completing patterns from its training data. A chat-tuned release of the same architecture, after SFT and often RLHF, will answer questions directly and follow instructions far more reliably. Both share the same underlying transformer block structure and the same parameter count, but their behavior differs enormously because of what training stages were applied after pretraining finished.

Finally, it is worth remembering that fine-tuning is reversible in the sense that it does not erase pretraining knowledge entirely, but it can degrade capabilities that were not represented in the fine-tuning data, an effect sometimes called catastrophic forgetting in the broader machine learning literature. A model fine-tuned heavily on customer support transcripts, for example, might become noticeably worse at creative writing or code generation if the fine-tuning process pushes its weights too far from the broad distribution learned during pretraining. Engineers running their own fine-tuning jobs should test for this kind of regression on tasks outside the fine-tuning dataset, not just on the target task itself.

The InstructGPT procedure is the clearest documented example of combining supervised fine-tuning with reinforcement learning from human feedback, and it is worth walking through its stages in order. The procedure used supervised fine-tuning on human-written demonstrations followed by reinforcement learning from human feedback based on ranked outputs.^[1] That two-sentence summary hides three distinct stages, each with its own data and its own training signal, and each worth understanding separately because they solve different problems.

The first stage is exactly the SFT process described above: a pretrained model is fine-tuned on a dataset of prompts paired with human-written demonstration answers. This produces what is often called an SFT model, a model that already follows instructions reasonably well but has not yet been refined using comparative feedback. The second stage introduces something new: human labelers are shown multiple outputs the SFT model generated for the same prompt, and instead of writing a new answer, they rank those outputs from best to worst. This ranking data is called preference data, and it captures comparative judgments rather than single correct answers.

Preference data is used to train a separate model called a reward model. The reward model's job is to take a prompt and a candidate response and output a single score estimating how much a human would prefer that response. It is trained on the ranking data from the labelers: given two responses to the same prompt where humans ranked one above the other, the reward model is adjusted so that it assigns a higher score to the preferred response. Once trained, the reward model can score any new response to any prompt without needing a human in the loop for every evaluation, which is what makes the next stage computationally feasible.

The third stage uses the reward model inside a reinforcement learning loop to further adjust the SFT model's weights, which is the actual RLHF step. The model generates a response to a prompt, the reward model scores that response, and the model's weights are updated to make higher-scoring responses more likely in the future. This is a fundamentally different training signal than the next-token prediction cross-entropy loss used in pretraining and SFT. Instead of matching a target token sequence, the model is being nudged toward whatever the reward model considers preferable, which indirectly encodes the aggregated preferences of the human labelers who ranked the original outputs.

The timeline below lays out these stages in sequence, which is useful for keeping straight which dataset and which training signal belongs to which step.

Why go through the trouble of training a separate reward model rather than just having labelers rate every output directly during training? Because reinforcement learning requires an enormous number of feedback signals to train effectively, far more than human labelers could realistically provide one-by-one in real time. The reward model acts as a learned stand-in for a human judge, scalable to millions of generated responses during the reinforcement learning stage, while only the smaller initial ranking dataset required direct human labor. This is the practical reason the pipeline has three stages rather than trying to do reinforcement learning directly against live human raters.

It is useful to walk through what the preference data collection actually looks like in practice, since it is easy to picture it too abstractly. Labelers are shown a prompt, such as a question about tax deductions, along with several candidate answers generated by the SFT model. They might rank one answer highest because it correctly states a caveat about needing to consult a professional, rank another lower because it states something confidently that is likely wrong, and rank a third lowest because it refuses to answer at all when answering would have been reasonable. Every one of those judgments reflects a stance on what good behavior looks like, encoded by the specific labelers and the guidelines they were given.

The steps below summarize the practical workflow an engineering team would follow if building an analogous pipeline for a narrower, in-house use case, such as aligning a model to a company's support policies.

The InstructGPT researchers reported that human-feedback fine-tuning improved preference ratings on their evaluated prompt distribution compared to the earlier pretrained and SFT-only models, while also reporting that the resulting models still made mistakes.^[2] That combination, measurable improvement alongside persistent error, is the honest summary of what RLHF accomplishes. It shifts the probability mass of a model's outputs toward what human raters preferred on the prompts it was trained and evaluated on. It does not produce a model that is reliably correct, nor does it generalize perfectly to every prompt a user might later send, especially ones far outside the style or topic distribution of the training and evaluation prompts.

An important technical detail for engineers to carry forward: RLHF is a genuinely different kind of training from pretraining and SFT, because it optimizes against a learned proxy (the reward model) rather than a fixed, known-correct target token sequence. This creates a known risk called reward hacking, where the model finds ways to score highly on the reward model's scale without actually being more helpful. A model might learn, for instance, that longer or more confident-sounding responses tend to score well, and start padding answers or asserting things with unwarranted confidence as a side effect of optimizing against the learned reward signal rather than genuine helpfulness.

Because reward hacking is a real risk, production alignment pipelines typically include safeguards beyond the three stages described above, such as penalizing the reinforcement-learning-tuned model if its outputs drift too far from the SFT model's distribution. This is usually done by adding a term to the training objective that discourages large deviations, keeping the final model anchored to the broadly sensible behavior learned during SFT while still nudging it toward the reward model's preferences. Readers who look closely at technical reports describing RLHF pipelines will often see this kind of anchoring term mentioned, and it is the direct engineering answer to the reward-hacking risk just described.

Taken together, these three stages explain a pattern engineers frequently notice when comparing models: a model can be very good at following clear, well-specified instructions (a direct product of SFT) while still occasionally producing confidently wrong information or inconsistent judgment calls on edge cases (a reflection of the limits of reward modeling and the finite scope of the human feedback it was trained on). Neither SFT nor RLHF changes the underlying fact that the model is still, at the mechanical level, predicting the next token in a sequence. Alignment reshapes which sequences are likely, it does not replace the underlying generation mechanism described in earlier chapters on next-token prediction and self-attention.

Every stage of the pipeline just described depends on a human judgment somewhere: someone wrote the demonstration answers, someone wrote the labeler guidelines, someone ranked the candidate outputs. None of those judgments are neutral. They reflect the language, culture, legal environment, and institutional priorities of whoever produced them. This is the sense in which alignment is not a purely technical operation. The reward model is often described in papers and documentation as learning human preferences, but it would be more precise, if less convenient, to call it a model of the preferences of a specific, finite group of labelers working from a specific set of written instructions.

Consider a concrete example to make this less abstract. Suppose labeler guidelines instruct reviewers to rank responses that express uncertainty about contested political topics higher than responses that take a firm stance. That single instruction will shape the reward model to systematically favor hedged, neutral-sounding answers to political prompts across the entire deployed system, for every user, regardless of whether that user wanted a hedge or a direct opinion. The instruction was presumably written with good intentions, perhaps to avoid the appearance of the model pushing a political agenda, but it is nonetheless a policy choice embedded into the statistics of the model's outputs, not a discovery about what is objectively correct.

The matrix below maps where value judgments enter the pipeline versus where they do not, across each stage from SFT through RLHF.

This matters for engineers for a very practical reason: when a model refuses a request, hedges excessively, or confidently takes a stance you didn't expect, that behavior usually traces back to a specific training choice made somewhere in this pipeline, not a bug in the ordinary sense. Reporting it as a bug to a vendor sometimes leads to an adjustment, but often the behavior is working exactly as the alignment process intended, reflecting a deliberate choice about what the deployer wanted the model to do in that situation. Distinguishing between the model is broken and the model is aligned to priorities I don't share is a useful habit when debugging unexpected model behavior in production.

This also explains why different vendors' chat models, even ones built on similarly capable base architectures, can give noticeably different answers to the same sensitive prompt. One model might answer a medical question directly with appropriate caveats, while another refuses and suggests consulting a professional instead. Neither choice is more technically correct than the other in any absolute sense. Each reflects a different set of labeler guidelines, risk tolerances, and legal considerations baked into that vendor's SFT demonstrations and reward model training. Engineers building on top of multiple model providers should expect and test for this kind of divergence rather than assuming consistent behavior across vendors.

There is also a question of representativeness within a single pipeline. The group of people who write demonstrations and rank outputs is necessarily a small sample relative to the eventual global user base of a deployed model. Their judgments, however carefully guided, cannot represent every cultural context, language, or value system a model will encounter in deployment. A model aligned primarily using feedback from one linguistic or cultural group may handle prompts rooted in a different context less gracefully, not because the underlying language capability is missing, but because the alignment stage trained it toward a narrower slice of preferences than the full range of users it now serves.

None of this means alignment work is misguided or should be abandoned. Pretrained models without any fine-tuning are genuinely harder and less safe to deploy directly, since they lack any mechanism for refusing clearly harmful requests or producing consistently useful answers to straightforward questions. The point is narrower and more useful for engineering practice: alignment reduces certain kinds of bad behavior and increases helpfulness on the distribution of prompts and values it was trained against, while leaving open, unresolved questions about prompts and values outside that distribution. Treating an aligned model as a neutral oracle, rather than a system tuned to a particular set of guidelines, is the mistake to avoid.

For practical engineering purposes, this suggests a few habits worth adopting when working with aligned models in an application. First, read a vendor's usage policies and any published guidelines about model behavior, since they often explain the reasoning behind refusals or stylistic defaults that otherwise look arbitrary. Second, when a model's default behavior does not fit an application's needs (too cautious, too verbose, too hedgy), consider whether targeted prompting or a narrower fine-tuning pass on top of the vendor's model can adjust that behavior within acceptable limits, rather than assuming the behavior is fixed. Third, when evaluating a model for a sensitive application, test it against the specific population and context it will actually serve, rather than relying solely on a vendor's general-purpose alignment having already covered that case.

It is also worth separating two different things that often get bundled together under the word alignment in casual conversation: alignment to instructions (does the model do what it is asked) and alignment to values (does the model's behavior match some notion of ethical or safe conduct). The pipeline described in this chapter, SFT followed by RLHF, touches both, since demonstration data teaches instruction-following while labeler guidelines and reward model training encode value judgments about refusals, caution, and tone. Keeping these two threads distinct helps when diagnosing a model's behavior: a model that fails to follow a clear instruction has a different kind of problem than a model that follows the instruction but applies a value judgment you disagree with.

The broader lesson for anyone building on top of these models is that the line between technical limitation and policy decision is blurrier in alignment than in almost any other part of the pipeline covered in this book. Tokenization, attention, and scaling laws are governed by mathematics and measured trends that behave the same regardless of who is running the code. Alignment is governed partly by mathematics, specifically the training mechanics of SFT and RLHF, and partly by a set of written guidelines that someone decided on, for reasons that may or may not be visible to the engineer calling the API. Carrying that distinction forward is good preparation for the next chapter's look at how decoding choices, made entirely at inference time with no retraining involved, give an engineer a different and more direct kind of control over a model's output.

Recall that for each position in a sequence, the model produces a probability distribution over vocabulary, a score for every possible next token given everything generated so far. If the vocabulary has tens of thousands of entries, that is tens of thousands of numbers between 0 and 1 that sum to 1. The simplest possible way to pick the next token is to take the single highest-scoring one every time. That strategy is called greedy decoding, and it is deterministic: the same prompt always produces the same output, because there is no randomness anywhere in the process.

Greedy decoding sounds safe, but it produces text with a specific flavor. Because it always grabs the most probable token, it tends to repeat common phrasings, fall into loops on longer generations, and avoid anything that reads as creative or varied. For tasks like extracting a date from an invoice or classifying a support ticket, this predictability is exactly what you want: the same input should give the same output every time. For open-ended tasks, like writing variations on a tagline or brainstorming names, greedy decoding feels flat and samey across repeated calls.

Sampling replaces that deterministic argmax with a random draw from the probability distribution itself. A token with a 40% probability gets chosen about 40% of the time, a token with 2% probability gets chosen rarely but occasionally, and so on. This single change reintroduces the variety that greedy decoding strips out, because the model can occasionally pick a less likely but still plausible continuation instead of the single most predictable one. The cost is that outputs become non-deterministic: the same prompt, called twice, can produce two different answers.

Temperature is a knob that reshapes the distribution before sampling happens, rather than a replacement for sampling. Mechanically, the model produces raw scores called logits, which get converted into probabilities through a function that divides each logit by the temperature value before normalizing. A temperature of 1 leaves the distribution as trained. A temperature below 1 sharpens the distribution, pushing more probability mass onto the already-likely tokens and making the output closer to greedy decoding. A temperature above 1 flattens the distribution, giving lower-probability tokens more of a chance and producing more varied, sometimes stranger, output.

P(token_i) = e^{logit_i / T} / Σ_j e^{logit_j / T}

Here T is the temperature. As T approaches 0, the distribution collapses onto the single highest-logit token, recovering greedy decoding as a limiting case. As T grows large, the distribution flattens toward uniform, where every token in the vocabulary becomes roughly equally likely regardless of how implausible it seemed to the model. In practice, a temperature near 0 is used for factual, deterministic tasks, a middle value is common for general conversational use, and higher values are reserved for deliberately creative or exploratory generation where variety outweighs coherence.

Temperature alone has a weakness: it rescales every token's probability, including the ones far down in the tail that were never good options in the first place. At a higher temperature, a model might occasionally sample a token that was almost irrelevant to the context, just because flattening gave it enough of a boost to get picked by chance. Top-p sampling, also called nucleus sampling, addresses this by restricting the candidate pool before sampling. Instead of considering the entire vocabulary, it sorts tokens by probability, then keeps adding them to a shortlist until their combined probability reaches the threshold p, discarding everything else.

If p is set to 0.9, the model builds the smallest set of top-ranked tokens whose probabilities add up to 90%, then samples only from that set, renormalizing their probabilities to sum to 1. When the model is confident and one or two tokens dominate the distribution, the nucleus is small and the output stays close to deterministic. When the model is uncertain and probability is spread across many plausible tokens, the nucleus grows to include more candidates, allowing more variety exactly in the situations where multiple continuations are genuinely reasonable. This adapts to context in a way that a fixed temperature cannot, since it reacts to how peaked or flat the actual distribution is at each step rather than applying a constant transformation.

Temperature and top-p are usually combined, not chosen as alternatives. A typical configuration might apply temperature to reshape the distribution, then apply top-p to prune the tail before sampling. Different API providers expose these as separate parameters, and some documentation recommends adjusting one or the other but not both at once, since their effects compound in ways that can be hard to reason about together. A reasonable default for most production use is a moderate temperature with a top-p setting that keeps the tail restricted, adjusted only after observing real outputs rather than guessed from first principles.

Consider a customer support bot built on top of an LLM. If it runs at a high temperature with no top-p restriction, it might occasionally generate a confident-sounding but fabricated policy detail, because the sampling step pulled in a low-probability token that happened to start a plausible-sounding but wrong sentence. The fix is rarely to blame the model's knowledge. Often the fastest fix is to lower the temperature for that specific call, since factual consistency usually matters more than linguistic variety in that context, and the sampling settings are the first and cheapest thing to check before touching the prompt or the underlying model.

Decoding settings should be matched to the job, not set once globally. The table below sketches how three common tasks pull toward different ends of the sampling spectrum. Extraction and classification tasks want determinism and reproducibility. Conversational assistants want a bit of natural variation without drifting into incoherence. Creative writing tools want enough randomness that repeated calls feel genuinely different from each other.

It helps to remember that sampling settings only ever restrict or reshape what the model already considered plausible at that step. Sensible temperature and top-p settings rarely surface a token the model assigned essentially zero probability to, and no value can inject knowledge the model never had. These parameters govern how the model expresses its existing uncertainty, not what it knows. That distinction will matter again in the discussion of hallucination, since decoding choices are one contributing cause among several, not the root cause by itself.

One more practical note: because sampling is a separate step from the model's forward pass, you can often experiment with temperature and top-p for free, without any retraining or even a different model, since the same forward pass naturally produces one distribution per position that different sampling settings consume differently. This makes decoding settings one of the cheapest and fastest levers available when an application's output quality feels off, worth testing before reaching for a different model or a longer prompt.

Every transformer-based model has a limit on how many tokens it can consider at once, called the context window. This includes everything: the system instructions, the conversation history, any documents pasted into the prompt, and the tokens the model is in the process of generating. Once that combined total reaches the model's limit, something has to give. Depending on the implementation, older tokens get truncated, the request gets rejected outright, or the application has to actively manage what stays in view. There is no universal size for this window; it varies by model family and by specific release, so always check the documentation for whatever model is actually in use rather than assuming a number.

It helps to picture the context window as a sliding frame rather than a bucket that just fills up. At each generation step, the model attends across every token currently inside that frame, through the self-attention mechanism, to decide what comes next. A token outside the frame does not exist as far as that forward pass is concerned, no matter how important it was three exchanges ago in a chat conversation. This is why a long-running chatbot conversation can appear to "forget" something a user said early on: it was never forgotten by any model weight, it simply fell outside the window that got sent to the model on this particular call.

Because token counts rarely match word counts (a consequence of byte-pair encoding, covered when discussing tokenizers), engineers routinely underestimate how quickly a context window fills up. A long legal document, a chat history with many back-and-forth turns, or a codebase pasted in for review can consume a surprising share of the available window before the model has generated a single token of response. A practical habit is to count tokens programmatically before sending a request, using the same tokenizer the target model uses, rather than estimating from word or character counts, which can be off substantially depending on the text's language and structure.

The context window creates direct engineering tradeoffs, not just an abstract ceiling. A larger context window lets an application stuff in more background, whole documents, long chat histories, extensive few-shot examples, without any retrieval or summarization step. But a longer prompt also means more tokens to process on every single call, since the self-attention computation scales with sequence length, which raises both latency and the per-call cost that most providers bill by the token. A model with a huge window is not automatically the right choice if the application only ever needs a few hundred tokens of context; the window size and the typical workload need to be matched deliberately.

When an application's needs genuinely exceed what fits in a single call, three common strategies kick in, each with its own cost. Truncation simply drops the oldest or least relevant content, which is cheap but risks losing information the model actually needed. Summarization periodically compresses older conversation turns into a shorter summary that gets fed back in as context, trading some detail for a much smaller token footprint. Retrieval stores information externally, often as embeddings in a vector store, and pulls back only the most relevant chunks for a given query rather than holding everything in the live context at once.

The decision tree below sketches a rough path for picking among these strategies based on what is actually overflowing the window. None of these strategies is universally correct; the right choice depends on whether the overflow comes from a single long document, an accumulating chat history, or a large and growing knowledge base that the model needs to draw on selectively.

Context window limits also interact with cost in a way worth making explicit with numbers. If an application sends a long conversation history on every single turn, even though the model only needs the last two or three turns to respond sensibly, it is paying to reprocess thousands of redundant tokens on every turn. This is wasteful not just financially but in latency, since the self-attention computation must run over every token in the window on every call, including ones that contributed nothing useful to the response. Trimming history to what is actually needed is often a bigger cost lever than switching to a cheaper model.

A second subtlety is that a model's performance does not always stay uniform across its entire advertised window. Some models show a measurable drop in how well they use information placed in the middle of a very long context, compared to information near the beginning or the end, an effect sometimes called the "lost in the middle" pattern in evaluation literature. This means doubling a context window does not automatically double how much useful information an application can reliably rely on; testing with realistic, full-length prompts is the only way to know how a specific model actually behaves at length, rather than trusting the advertised maximum as a performance guarantee.

A concrete example: an application that lets users chat with a long onboarding manual might be tempted to paste the entire manual into every request alongside the chat history. For a short manual this works fine. For a long one, the manual alone can consume most of the context window, leaving little room for conversation history or for the model's own response, and performance on specific questions may suffer if the relevant passage ends up buried in the middle of a very long prompt. A retrieval step that pulls only the few paragraphs relevant to the current question is usually more reliable, and considerably cheaper, than relying on brute-force context stuffing.

Context window size has also grown substantially across model generations, and it is tempting to treat "bigger window" as straightforwardly better. It often is useful, but it is not a substitute for good context management. A model with a huge window still benefits from receiving a focused, well-organized prompt rather than an unstructured dump of everything that might conceivably be relevant, both because focused prompts are cheaper to run and because models generally perform more reliably when the genuinely relevant information is easy to find rather than diluted among irrelevant text.

One frequent mistake worth naming directly: assuming that because a model's documented window is large, an application can safely ignore token budgeting altogether. In practice, almost every production LLM system benefits from some explicit accounting of how many tokens are being spent on system instructions, history, retrieved context, and the expected length of the response, because unbounded growth in any of these eventually triggers either a hard failure when the window is exceeded or a silent and expensive overuse of tokens that nobody budgeted for. Treating the context window as a finite, costed resource from the start avoids both failure modes.

The context window interacts with sampling settings too, in one easy-to-miss way: longer outputs give more opportunities for a non-zero temperature to introduce drift, since each sampled token becomes part of the context that conditions every subsequent token. A short, high-temperature response has fewer chances to wander off course than a long one, which is one reason long-form generation sometimes benefits from a slightly lower temperature than a short, single-turn answer would.

Generation happens one token at a time. The model predicts a token, that token gets appended to the sequence, and the whole updated sequence is fed back in to predict the next one. This is incremental generation, the mechanical process underlying every chat response you have ever seen stream across a screen. Without any optimization, this naive approach would mean that generating the tenth token requires reprocessing all nine tokens that came before it from scratch, generating the hundredth token means reprocessing ninety-nine, and so on, with the redundant work growing larger at every single step of a long response.

That redundancy comes directly from how self-attention works. At every transformer block, every position computes a query vector, a key vector, and a value vector, and attention combines them to build a context-dependent representation. Crucially, the key and value vectors for a given token, at a given layer, depend only on that token and the tokens before it, not on anything generated afterward. Once a token's key and value vectors have been computed, they never change no matter what gets generated later, which means recomputing them on every subsequent step is pure wasted work.

The KV cache exploits exactly this fact. Instead of recomputing key and value vectors for the entire sequence at every step, the model stores the key and value tensors it already computed for earlier tokens and simply reuses them, computing fresh key and value vectors only for the newest token.^[1] The new token's query vector then attends over the full set of cached keys and values, old and new combined, exactly as if the whole sequence had been reprocessed, but without actually redoing any of the earlier computation. The output is mathematically identical to the naive approach, it is purely a computational shortcut, not an approximation.

The practical effect on speed is substantial, and it grows with the length of the sequence. Without caching, the amount of repeated work at step n scales with the sequence length so far, meaning total work across a full generation grows substantially with the output length. With caching, each new step only requires computing attention for one new query against the existing cached keys and values, which keeps the per-step cost far more manageable even as responses get long. This is a major reason that modern LLM inference, including the kind that powers responsive chat interfaces, is fast enough to feel usable for interactive use at all.

It helps to walk through what actually happens in memory during a single generation call. When a request first arrives, the model processes the entire prompt in one pass, called the prefill step, computing and caching key and value vectors for every prompt token across every layer. From that point forward, the model enters the decode step: it generates one token, computes that one token's key and value vectors, appends them to the cache, and repeats. The prefill step does the bulk of the up-front work; the decode step is what makes a model feel like it is "thinking out loud" one token at a time.

# Simplified sketch of incremental generation with a KV cache
# (illustrative pseudocode, not a real model's internals)

def generate(prompt_tokens, max_new_tokens):
    cache = {}  # holds key/value tensors per layer, grows each step
    # Prefill: process the whole prompt once, populate the cache
    logits, cache = model.forward(prompt_tokens, kv_cache=cache)
    output_tokens = []

    for _ in range(max_new_tokens):
        next_token = sample(logits)          # temperature / top-p applied here
        output_tokens.append(next_token)

        if next_token == EOS_TOKEN:
            break

        # Decode step: only the new token is processed,
        # reusing cached keys and values for everything before it
        logits, cache = model.forward([next_token], kv_cache=cache)

    return output_tokens

Notice in that sketch that the decode loop passes only [next_token] into the forward call, a single token, rather than the entire growing sequence. That single line is the entire payoff of the KV cache. Without it, that call would need to pass the full list of all tokens generated so far, every single step, and the model would silently redo work it had already done on every previous call. The cache turns an operation that would otherwise grow more expensive every step into one that processes only the new token at each step, whose attention over the cached keys and values still grows with the sequence length, at the price of memory needed to store the growing cache itself.

That last tradeoff deserves attention, because the KV cache is not free. Storing keys and values for every token, at every layer, in a potentially long context window consumes memory, and that memory requirement grows directly with the length of the sequence currently being processed. For a single short chat exchange this is negligible. For an application running many concurrent long conversations, each with a large context window in active use, KV cache memory can become a real constraint on how many requests a server can handle at once, which is one reason providers may charge differently, or impose different concurrency limits, for very long context usage.

This has a direct implication for how applications should think about conversation history. Each new message in an ongoing chat means the cache keeps growing, and keeping a long cache resident takes up memory on whatever server is serving that request, memory that cannot be used for other concurrent requests during that time. This is a second, independent reason beyond cost and attention quality for trimming or summarizing old conversation turns rather than letting history grow without bound, since the KV cache makes a long context expensive not just in tokens billed but in server memory consumed for the duration of the request.

A common point of confusion is thinking of the KV cache as something a model architecture either has or lacks, as if it were a design choice baked into the weights. It is not part of the trained model at all. It is purely an inference-time engineering optimization implemented in the serving code around the model, something a framework or inference server does regardless of which specific transformer weights are being run through it. Two different applications serving the exact same model could implement caching differently, with different memory tradeoffs, and a user would never be able to tell from the output alone, since the cache changes speed and memory use, not the actual probability distribution the model produces.

The checklist below summarizes practical questions worth asking when a generation pipeline feels slower or more expensive than expected, tying sampling, context window management, and caching together.

Taken together, sampling, the context window, and the KV cache form the practical machinery that sits between a trained model and the text an application actually receives. None of them touch the weights produced by pretraining, fine-tuning, or alignment. They decide how a fixed distribution gets turned into a choice, how much of the conversation the model can actually see at any one moment, and how efficiently that choice gets computed as the response grows. An engineer who understands these three levers can often fix a slow, expensive, or oddly behaving LLM feature without retraining anything or switching providers, simply by adjusting the settings that were available all along.

The word hallucination is itself borrowed uneasily from psychology, where it describes a perception with no external stimulus. Applied to a language model, it is a metaphor stretched to cover any output that is fluent but ungrounded in fact or in the given context. Survey work on the topic documents multiple competing definitions and categorization schemes across the research community, rather than one settled taxonomy that every paper uses the same way.^[1] That matters for engineers because it means a benchmark claiming to measure hallucination rate is measuring a specific operational definition, not an agreed-upon universal quantity, and two benchmarks can disagree sharply while both being internally valid.

One useful split, even without claiming it is the only correct one, separates intrinsic hallucination from extrinsic hallucination. Intrinsic hallucination contradicts information already present in the input, such as a summary that reverses a fact stated in the source document it was asked to summarize. Extrinsic hallucination adds information that cannot be verified against the input at all, whether or not it happens to be true, such as a summary that introduces a date never mentioned anywhere in the source. The first is a grounding failure relative to context; the second is closer to confabulation, filling a gap with something plausible-sounding.

A second useful split separates factuality hallucination from faithfulness hallucination. Factuality hallucination is a conflict with real-world facts, like naming the wrong year for a historical event. Faithfulness hallucination is a conflict with instructions or provided context, like answering a question the user did not ask or ignoring a constraint stated in the prompt. A model can be faithful but wrong (it stuck to the document but the document itself was outdated) or unfaithful but accidentally correct (it ignored the prompt but stated a true fact). Treating all of these as one undifferentiated category makes evaluation and debugging much harder than it needs to be.

Concrete examples make the distinctions tangible. Consider a model asked to summarize a contract that states a deadline of a certain month, and the model's summary states a different month: that is intrinsic, faithfulness-type hallucination, because the error contradicts the given input directly. Now consider a model asked an open factual question with no supporting document, like who won a particular award in a particular year, and it states a confident but incorrect name: that is extrinsic, factuality-type hallucination, because there is no local source to contradict, only the external world. A third case: a model asked to write a product description invents a technical specification that sounds right but was never in any spec sheet provided. That is also extrinsic, but closer to confabulation than to factual error, since no ground truth was ever in play.

The callout below captures this grid for quick reference when triaging a real failure.

Why does this taxonomy debate matter for someone shipping a product rather than writing a survey paper? Because the fix for intrinsic, faithfulness-type errors is often architectural: give the model better access to the source and check the output against it, a pattern often called retrieval-augmented generation. The fix for extrinsic, factuality-type errors is often different: restrict the domain of claims the model is allowed to make unsupervised, or route open factual questions to an external lookup rather than trusting generation alone. Conflating the two leads to mismatched solutions, like adding more retrieval to a system whose real problem is that it invents technical specifications nobody ever asked it to source.

Another dimension worth separating is severity. A hallucinated middle initial in a biography is annoying. A hallucinated legal precedent cited in a court filing is a professional and legal crisis.^[2] Research surveys of the field catalog this spread of severity alongside the spread of causes and categories, reinforcing that no single number or definition captures what matters across every application. An engineering team should define, for its own domain, which categories of error are tolerable annoyances and which are unacceptable failures, rather than importing a generic hallucination rate metric from an unrelated benchmark and assuming it transfers.

It also helps to notice what hallucination is not. It is not the same as a model being uncertain and saying so, nor the same as a model declining to answer. Those are arguably the opposite of hallucination: a model expressing calibrated doubt instead of confabulating confidently. Unfortunately, current language models are trained in ways that make confident-sounding, fluent text the default output style, regardless of the underlying uncertainty driving the token choices, which is exactly why hallucinated answers so often read as authoritative as correct ones. That stylistic uniformity, independent of the taxonomy question, is one of the most dangerous properties of the whole phenomenon, and it sets up the causal discussion in the next section.

Finally, resist the temptation to treat hallucination as a problem that will simply vanish as models improve. Larger and more capable models do shift the mix of errors, often reducing simple factual slips while leaving subtler confabulations intact, but the underlying mechanism producing hallucination, discussed next, is tied to how these models are trained and sampled, not to their raw scale. The chapter on scaling laws already warned that empirical trends in loss do not guarantee improvement on every capability or every deployment, and hallucination is one of the clearest places that warning applies.

Four broad sources of hallucination show up repeatedly across the research and in production systems: problems in the training data, irreducible statistical uncertainty, a mismatch between the training objective and the goal of truthfulness, and choices made at decoding time. None of these is the single cause. They compound, and a given wrong answer often has more than one of them contributing at once.

Data causes start with the simple fact that pretraining corpora, built from large-scale scraped and curated text, inevitably contain errors, outdated information, and outright fiction presented as nonfiction somewhere in the mix. A model trained to predict the next token across that data absorbs the statistical patterns of all of it, including the parts that were wrong when written or have since become wrong. If a particular false claim appeared frequently and consistently across many documents during training, the model may reproduce it with real confidence, simply because the pattern was well represented in its training signal. There is also a coverage problem: domains with sparse or low-quality text in the training data (niche technical fields, rapidly changing current events, specific small organizations) are exactly where a model's internal knowledge is thinnest, and thin knowledge plus a fluent generation style is a recipe for confident invention.

Uncertainty causes are more structural. The model's entire output mechanism is a probability distribution over vocabulary at each step, and when the true answer is genuinely unknown or ambiguous given the context, that distribution does not collapse to a clean I don't know token. It spreads probability mass across several plausible continuations, and the sampling procedure then picks one of them, right or wrong, with the same fluent phrasing either way. Consider asking a model for the population of a small town with limited web coverage: the model has seen a handful of numbers in similar contexts during training, none necessarily correct for that specific town, and it will pick one and state it as fact because stating numbers fluently is what next-token prediction trained it to do. The uncertainty was there internally, in the spread of the distribution, but nothing in standard decoding surfaces that uncertainty to the user.

Objective-mismatch causes go deeper than data quality. The training objective behind pretraining, minimizing cross-entropy loss on next-token prediction, rewards the model for assigning high probability to whatever token actually follows in the training text. It does not directly reward the model for having a true belief about the world, nor does it penalize fluent confabulation more than fluent truth, provided both are statistically plausible continuations. Instruction tuning and RLHF, covered in the chapter on fine-tuning and alignment, add a further layer of pressure that can make this worse in a specific way: human raters and reward models used in RLHF tend to prefer confident, complete-sounding answers over hedged or uncertain ones, so the alignment stage can inadvertently train the model to sound more certain than its internal knowledge actually supports. This is sometimes called a mismatch between the training signal and truthfulness as a goal: nothing in the pipeline directly optimizes for say this only if you are actually confident it's true.

Decoding causes sit on top of all of the above, at the moment text is actually generated. Sampling settings such as temperature, covered in the chapter on inference, directly affect how often a model strays from its highest-probability continuation into less likely, and more error-prone, territory. A higher temperature flattens the distribution, making the model more likely to pick a less probable, less supported token purely for variety, which raises hallucination risk in exchange for more creative-sounding output. Long generations compound this risk further: each new token is generated conditioned on everything before it, so an early hallucinated claim becomes part of the context for every token that follows, and the model tends to continue coherently elaborating on its own earlier mistake rather than noticing and correcting it. This compounding effect explains why hallucinations often appear as entire fabricated paragraphs, complete with convincing supporting detail, rather than as isolated wrong facts.

The table below summarizes these four causes side by side, with a representative example and a general direction for mitigation, which the next section develops into concrete engineering steps.

It is useful to see these four causes interact in a single worked scenario. Suppose an application asks a model to describe the technical specifications of a specific product model that was only briefly covered in a handful of low-quality web pages during training (a data cause: thin, possibly unreliable coverage). The model's internal distribution over possible specification values is genuinely spread across several plausible numbers (an uncertainty cause). Because the RLHF stage rewarded confident, complete-sounding answers during training, the model has learned to state one of those numbers outright rather than hedge (an objective-mismatch cause). Finally, because the application runs at a moderate temperature for more natural-sounding prose, the specific number chosen that run may not even be the model's single most probable guess (a decoding cause). The result: a confident, fluent, wrong specification, with no single point in the pipeline that obviously caused it, because all four layers contributed.

This layered picture also explains why hallucination resists simple fixes. Scaling up model size and training data, the subject of the earlier chapter on scaling laws, tends to reduce data-driven and uncertainty-driven errors somewhat, since a larger model trained on more text has seen more correct information and built sharper distributions over many common facts. But it does little to fix the objective-mismatch problem, since the reward signal during alignment still tends to favor confident answers, and it does nothing at all to fix decoding-time choices made by whoever deploys the model. This is exactly why newer, larger, more capable models still hallucinate, sometimes on different kinds of questions than older models but rarely at a rate of zero.

One more structural point deserves attention: self-consistency. Because text is generated one token at a time, with each token conditioned on everything generated so far, a model has no built-in mechanism to go back and revise an earlier claim once later tokens depend on it. A human writer can draft a paragraph, notice a factual slip, and rewrite it before anyone sees it. A standard autoregressive model, absent an explicit mechanism like self-critique prompting or output verification, commits to each token as it goes and builds everything afterward on top of that commitment, right or wrong. This is a direct consequence of the generation mechanism described in the chapter on next-token prediction, not an incidental implementation detail, and it means hallucination is not merely a knowledge gap but partly a structural property of how these models write at all.

A final subtlety worth naming is the difference between a model not knowing and a model not being asked correctly. Ambiguous or underspecified prompts invite exactly the kind of confident filling-in-the-gaps behavior that produces hallucination, because the model must resolve the ambiguity somehow to produce any output at all, and resolving it silently, without flagging the ambiguity, is the path of least resistance given typical training. A prompt that asks what was the revenue of the company last year without specifying which company, which year, or which revenue figure (gross, net, by which accounting standard) all but guarantees an answer built on unstated assumptions the user never sees. This particular cause is the most directly addressable one by whoever is writing the prompts, which leads naturally into the mitigations covered next.

Given that hallucination cannot be eliminated at the model level, the practical task for an application builder is to design systems that reduce its frequency, limit its severity when it does occur, and catch it before it reaches a user wherever possible. This is closer to building a reliability system around an unreliable component than it is to debugging a single root cause, and the right mindset borrows more from defensive software engineering than from pure machine learning tuning.

The single highest-leverage mitigation for factuality hallucination is grounding the model in retrieved, checkable source material, commonly implemented as retrieval-augmented generation. Instead of asking a model to answer a factual question purely from its internal, frozen training knowledge, the application first retrieves relevant documents from a trusted, current source (a knowledge base, a document store, a live API) and includes that text directly in the prompt, instructing the model to answer only using the provided material. This converts an extrinsic factuality problem into an intrinsic faithfulness problem, which is both easier to verify (did the answer actually match the provided text) and easier to constrain through prompting. It does not fully solve intrinsic hallucination, since a model can still misread or contradict the provided context, but it dramatically shrinks the space of possible errors compared to unconstrained recall.

Lowering temperature for factual tasks is a cheap, immediate decoding-level lever. Since a higher temperature increases the chance of sampling a less probable, less supported continuation, dropping it toward the conservative end for any task where accuracy matters more than variety (data extraction, structured lookups, citation generation) reduces one specific, well-understood source of risk at essentially no engineering cost. This should be paired with the understanding, from the chapter on inference, that temperature is just one decoding control among several, and that top-p or similar settings can interact with it in ways worth testing empirically rather than assuming.

Structuring output to make verification cheap is another direct, implementable step. Asking a model to cite which part of a provided source document supports each claim, or to output structured fields (a claim, a confidence label, a source reference) rather than free-flowing prose, makes it far easier for an automated check or a human reviewer to catch a fabrication before it propagates. A system that asks for unstructured prose and trusts it outright has no natural seam at which to insert a check. A system that asks for a claim paired with its supporting quote has a built-in seam: a simple string-match or embedding-similarity check can flag cases where the claim and the quoted support do not actually line up.

Prompting for calibrated uncertainty directly targets the objective-mismatch cause. Explicit instructions such as if the provided context does not contain the answer, say that explicitly rather than guessing push against the default training pressure toward confident, complete-sounding answers. This does not make the underlying model genuinely more calibrated internally, but it shifts the observable behavior at the margins, and it is nearly free to implement. It pairs well with structured output: a required confidence or source found field forces the model to make an explicit commitment about whether grounding exists, which is harder to silently skip than a free-text hedge buried in a paragraph.

Keeping generations short and decomposed addresses the compounding problem described earlier. Rather than asking for one long, free-form answer where an early hallucinated claim can seed an entire elaborated paragraph, breaking a complex task into smaller steps, each independently checkable, limits how far a single error can propagate before it is caught. This is the same idea behind decomposing a large function into smaller, independently tested units in ordinary software engineering, applied to generation instead of code.

Independent verification passes add a second line of defense once an answer has been generated. This can mean a second model call that is specifically asked to check the first output against the source material (sometimes called self-critique or a verifier pass), or a traditional rule-based check for the specific failure modes most common in a given domain, such as validating that every cited reference actually appears in the provided document set. Verification passes add latency and cost, so they are best reserved for the highest-stakes outputs in a pipeline rather than applied uniformly everywhere, which is itself a design decision worth making deliberately rather than by default.

The steps below collect these mitigations into a rough order of implementation, from cheapest to most involved, for a team retrofitting hallucination defenses onto an existing LLM-powered feature.

Evaluation deserves its own attention, because a mitigation that is never measured is a mitigation nobody can be sure is working. Building a small, domain-specific test set of prompts where the correct answer is known, run periodically against the production configuration, catches regressions introduced by a model upgrade, a prompt change, or a decoding setting adjustment, far more reliably than relying on anecdotal user reports. This test set should specifically include cases from each taxonomy category discussed earlier (intrinsic and extrinsic, faithfulness and factuality) rather than a single undifferentiated pile of questions, since a model's error rate often varies substantially across those categories.

None of these mitigations should be sold internally, or to end users, as making hallucination impossible. The honest framing is that an LLM-based system carries a residual, non-zero risk of confident, fluent, wrong output, and the engineering job is to push that risk down, make its remaining instances cheaper to catch, and set user expectations accordingly. A product that promises zero hallucination is making a claim the underlying mechanism cannot support. A product that is honest about the risk, and visibly engineered to catch and limit it, earns more durable trust from the people who rely on it.

The deeper lesson connects back to the very first idea in this book: an LLM is a next-token predictor, and everything built on top of it, from alignment to retrieval to verification passes, is an attempt to steer and constrain that underlying mechanism toward outcomes a person actually wants. Hallucination is not a glitch sitting awkwardly outside that mechanism. It is a direct, structural consequence of it, and understanding that is what lets an engineer stop hoping it will go away and start designing systems that work well despite it.

Start from the beginning and walk the path a single prompt takes, because that walk is the shortest summary of the whole book. Text arrives and a tokenizer, built through byte-pair encoding, breaks it into tokens that may be whole words, fragments, or punctuation marks. Each token ID is looked up in an embedding table and turned into a vector that lives in a high-dimensional space shaped entirely by training, not by any human-designed dictionary of meaning. Those vectors enter a stack of transformer blocks, where self-attention lets every position pull in context from every other position through query, key, and value vectors, and a feed-forward layer processes each position further, with residual connections and layer normalization keeping the signal stable across many layers.

That stack does one thing at its output: it produces a probability distribution over the vocabulary for what token comes next. Everything that feels like reasoning, planning, or conversation is built from repeating that single operation, token after token, feeding each choice back in as part of the next input. This is the idea covered under next-token prediction, and it is worth holding onto as the anchor fact of the whole book, because every later chapter is really an elaboration of how that one operation gets trained, steered, and executed efficiently at scale.

Training that stack is where pretraining and cross-entropy loss enter. The model is shown enormous amounts of text and adjusted, one gradient step at a time inside a training loop, so that its predicted distribution assigns higher probability to the tokens that actually came next in the training data. Nothing in that process directly rewards truthfulness, helpfulness, or safety. Those qualities, when they show up, are either side effects of good enough prediction on good enough data or the product of later stages added on top. Scaling laws describe how loss tends to fall as model size, dataset size, and compute grow together, and the Chinchilla ratio refined that picture by showing that balance between model size and training tokens matters more than brute size alone. None of that is a guarantee that every capability improves smoothly, only an empirical trend observed under the conditions studied.

Pretrained next-token predictors are not naturally good conversational partners, and that is where supervised fine-tuning and reinforcement learning from human feedback come in. The InstructGPT pipeline, built from demonstration data, a reward model, and policy optimization against that reward model, bent a raw predictor toward following instructions and matching human preferences. This is alignment in the narrow, practical sense used throughout this book: a model reflects the preferences of whoever curated its training signal, not some neutral or universal standard. That single sentence has outsized consequences for how much to trust a model's tone, its refusals, and its confident-sounding answers, especially on topics where reasonable people disagree.

Even a well-trained, well-aligned model has to be run, and running it is its own layer of decisions, separate from training. Sampling settings such as temperature and top-p or nucleus sampling control how much randomness enters each token choice, so the same weights can produce a cautious, repetitive answer or a varied, occasionally erratic one depending purely on decoding configuration. The context window limits how much of a conversation, document, or codebase can be visible to the model at once, and anything outside that window simply does not exist from the model's point of view. The KV cache is a pure efficiency mechanism, storing key and value vectors from earlier tokens so that generating each new token does not require recomputing attention over the entire sequence from scratch, which is what makes long, interactive sessions practical rather than prohibitively slow.

Hallucination sits at the intersection of all of these layers rather than belonging to just one of them. Noisy or contradictory training data, genuine uncertainty about rare facts, an objective that rewards fluent continuation rather than verified truth, and decoding choices that favor plausible-sounding tokens over cautious ones, all contribute to the same visible symptom of a model stating something false with full confidence. No single fix closes all of those causes at once, which is why fine-tuning and alignment reduce but do not eliminate the problem. Engineering around it, through retrieval of verified sources, structured prompting, and output validation, has to substitute for a guarantee that training alone cannot provide.

Consider a concrete scenario that pulls every layer into one decision: building an internal tool that answers employee questions about a company's expense policy using an LLM API. The tokenizer step determines cost and context budget, since policy documents full of numbers and bullet lists may tokenize less efficiently than plain prose, quietly inflating the token count billed per request. The embedding and attention layers determine how well the model can connect a vague question like can I expense a conference hotel to the relevant clause buried in a long document, which is why retrieval quality matters as much as model quality. The fine-tuning and alignment layer determines whether the model answers directly or hedges with disclaimers. The decoding layer determines whether a low temperature setting is chosen to keep answers consistent across repeated questions. And the hallucination risk is exactly why a serious version of this tool would cite the specific policy clause it used rather than asserting a dollar limit from memory.

The checklist below turns that scenario into a general habit: a short set of questions to run through whenever evaluating or debugging an LLM-backed feature, mapped back to the mechanisms covered across the book.

Treat this checklist as a return path into the rest of the book rather than a replacement for it. A question about token counts points back to the chapter on tokenizers and byte-pair encoding. A question about why retrieval matters points back to the chapter on embeddings and vector geometry. A question about explaining unexpected outputs points back to the discussion of attention weights and their limits. A question about which training stage shaped a behavior points back to the chapters on pretraining, scaling, fine-tuning, and alignment in turn. A question about decoding settings or context limits points back to the chapter on inference. A question about hallucination points back to the final chapter's taxonomy and mitigations. The checklist is short because the underlying model, once learned, is short enough to carry around.

It helps to be explicit about what this mental model does not give. It does not predict exactly how a specific prompt will be answered, because that depends on training data, fine-tuning choices, and decoding settings that are rarely fully documented by model providers. It does not settle disputed questions about whether larger models will keep improving on every task, since scaling laws are observed trends under studied conditions, not physical laws. It does not resolve the deeper question of whose values a model should encode when it is aligned, a question that is social and political as much as technical. What it does give is the ability to ask sharper questions, to notice when a vendor's claim is vague where it could be specific, and to design systems that account for known failure modes instead of being surprised by them.

That shift, from surprise to anticipation, is the practical difference between using an LLM API as a black box and using it as an understood component. A black box forces trial and error: change the prompt, see what happens, repeat. An understood component supports hypotheses: if the answer is wrong because the relevant fact fell outside the context window, extend the retrieval step rather than rephrasing the prompt forever. If the answer is inconsistent across runs, lower the temperature rather than assuming the model is simply unreliable. If the model refuses a reasonable request, recognize that as a fine-tuning and alignment choice rather than an inherent limit of the underlying predictor. Each of these diagnostic moves depends directly on a mechanism covered earlier in this book.

The field will keep producing new names for things: new attention variants, new fine-tuning recipes, new context-window techniques, new decoding strategies. Most of these will turn out to be refinements of the same handful of ideas rather than replacements for them. A new attention mechanism will still be computing some version of queries, keys, and values. A new alignment method will still be shaping a pretrained predictor toward some set of human-specified preferences, with the same open question of whose preferences those are. A new context-handling trick will still be working around the basic fact that a transformer's attention computation grows more expensive as sequence length grows. Reading new work through this lens turns each announcement into a specific claim to evaluate, not a wave of jargon to accept on faith.

There is also a quieter benefit to holding this mental model, one less about keeping up with the field and more about calibrating expectations day to day. A system built from next-token prediction, however well trained and aligned, has no persistent memory beyond its context window, no verified connection between its confident tone and the truth of its claims, and no guarantee that scaling or fine-tuning closes every gap between what it says and what is actually correct. Knowing that is not cause for dismissal, since the same mechanism also produces genuinely useful translation, summarization, coding assistance, and drafting. It is cause for proportionate trust: treating outputs as strong first drafts produced by a capable but fallible predictor, verified where it matters, rather than as pronouncements from an oracle.

None of the mechanisms in this book are final. Tokenization schemes change, attention variants multiply, alignment techniques get revised, and context windows grow in ways that shift which tradeoffs matter most. What tends to stay stable is the layered structure itself: text becomes tokens, tokens become vectors, vectors get mixed by attention and refined by feed-forward layers, the whole stack is trained to predict the next token, and that raw predictor is then steered and deployed under decoding settings that shape what actually comes out. New techniques tend to slot into one of those layers rather than replacing the structure. Recognizing which layer a new idea belongs to is often enough to judge roughly what it can and cannot do, even before reading the fine print.

For an engineer building on top of these systems, that structural recognition pays off most clearly during debugging and design reviews, the moments when a vague sense that the model is being weird needs to turn into a specific, testable explanation. Being able to say this looks like a context-window truncation issue or this is probably a decoding temperature problem, not a training problem changes a conversation from guesswork to engineering. It also changes conversations with non-technical stakeholders, who often assume more reasoning and more certainty inside these systems than actually exists. Explaining that an answer is the product of a trained next-token predictor, not a lookup against verified facts, sets expectations that save everyone trouble later.

The table below offers one last way to hold the whole book in view: each major layer of the system, the concept that governs it, and the practical question an engineer should ask about it when something goes wrong or a decision needs to be made.

Reading that table left to right traces the same path a prompt takes through the system, and reading it as a troubleshooting guide turns an opaque failure into a short list of specific hypotheses. That is the practical transformation this book has aimed for throughout: trading a vague, almost mystical sense of what a large language model is doing for a layered, mechanical account that can be checked, tested, and reasoned about like any other piece of software infrastructure. The next model that gets released, whatever it is called and whatever numbers accompany its announcement, will still be a next-token predictor shaped by a tokenizer, an embedding space, attention, training, fine-tuning, and decoding choices. The specifics will change. The shape, for now, will not.