Large Language Models: Experts, Decoding and Prompting

5.1 — What Is a Large Language Model?

Definition

A Large Language Model (LLM) is a statistical model that assigns probabilities to sequences of tokens. Given a sequence of words (or subword tokens), it answers the question: “What token is most likely to come next?”

More precisely, an LLM learns a conditional probability distribution:

$$P(x_{t+1}\mid x_1,x_2,\dots ,x_t)$$

where $x_1,\dots ,x_t$ are the tokens seen so far, and $x_{t+1}$ is the next token to predict.

What Makes It “Large”?

Three things set LLMs apart from earlier, smaller language models:

Architectural Choice: Decoder-Only

Modern LLMs (GPT, LLaMA, Gemma, DeepSeek, Mistral, Qwen, etc.) use a decoder-only Transformer. Unlike the original Transformer that had both an encoder and decoder, decoder-only models have only the decoder stack.

This means:

• Every forward pass attends only to tokens to the left (causal masking), never to future tokens.
• The model generates text one token at a time, appending each new token to the context before generating the next one. This is called autoregressive generation.

Input:    "A teddy bear is"
                  ↓
      Decoder-only Transformer
                  ↓
Output:   distribution over next token → sample → "cute"

New input: "A teddy bear is cute"
                  ↓
      Decoder-only Transformer
                  ↓
Output:   distribution over next token → sample → "."

5.2 — The Problem with Dense Transformers

Every Transformer layer has two sub-components:

1. Multi-Head Attention — lets each token look at other tokens
2. Feed-Forward Network (FFN) — processes each token independently through a two-layer MLP

In a standard (“dense”) Transformer, every single parameter fires on every single token. If a model has 70 billion parameters, every token — whether it’s a mathematical symbol, a French word, or a code keyword — passes through all 70B parameters.

This is inefficient. A token like the word integral probably doesn’t need the model’s French grammar knowledge. Yet every parameter is activated regardless.

The insight behind MoE is simple:

5.3 — MoE Architecture: Experts and Routers

MoE replaces the single FFN in each Transformer layer with:

1. $N$ smaller FFNs — called experts, each with its own independent weight matrix
2. A router (gating network) — a lightweight network that looks at each token and decides which experts to send it to

For each token, only $k$ out of $N$ experts are activated. The rest remain dormant — they contribute nothing to that token’s computation and consume no FLOP.

Standard Transformer Layer          MoE Transformer Layer
──────────────────────────          ──────────────────────────────────
  Attention                           Attention
      ↓                                   ↓
  One large FFN                       Router (lightweight)
  (all parameters active)                 ↓
                               Expert₁  Expert₂  Expert₃ … Expertₙ
                                   ↑         ↑
                               (only top-k experts run)

Why replace the FFN specifically?
FFNs hold the vast majority of a Transformer’s parameters. Attention heads are relatively small. Replacing the FFN with a sparse expert layer gives the maximum possible compute savings.

What exactly is an expert?
An expert is just a smaller FFN. Nobody hand-labels them as “the math expert” or “the grammar expert” before training. Specialization emerges on its own during training through a feedback process:

• Code tokens preferentially route to certain experts.
• Those experts see more code → they get more gradient signal about code → they get better at code.
• The router learns to send code tokens to them because they perform better.
• The loop reinforces itself until experts develop genuine specializations.

Token-level routing — an important subtlety:
The router makes independent decisions for each token in each layer. The word “Python” might be sent to Expert 7 in Layer 2 and Expert 11 in Layer 5. Routing is not per-sentence or per-request — it’s per-token, per-layer.

Active parameters vs. total parameters:
This distinction is the key to understanding MoE’s value proposition:

$$P_{\text{active}}=k\cdot P_{\text{expert}}{P}_{\text{total}}=N\cdot P_{\text{expert}}$$

• $k$ = number of experts selected per token
• $P_{\text{expert}}$ = parameters per expert
• $N$ = total number of experts

When $k\ll N$, the model can be enormous in total parameter count while remaining cheap per token in compute.

Real-world example: DeepSeek-V3 has $P_{\text{total}}\approx 671\text{B}$ parameters but only $P_{\text{active}}\approx 37\text{B}$ are used per token — an 18× ratio. The model is 18× larger than a comparably-quality dense model would need to be, but inference costs nearly the same as a 37B model.

5.4 — Router Mathematics and Geometric Intuition

The router is a simple linear layer followed by a softmax. Here’s how it works step by step.

Step 1: Compute a score for each expert.

Given the token’s hidden-state vector $x\in {\mathbb{R}}^d$, the router computes a raw score for each expert $i$ using a learned gating vector $g_i\in {\mathbb{R}}^d$:

$$s_i=x^{\top }{g}_i$$

What this means:

• $x$ — the hidden representation of the current token (a vector of size $d$)
• $g_i$ — the learned “direction” associated with expert $i$ (also a vector of size $d$)
• $s_i$ — the dot product, measuring how aligned the token is with expert $i$

Geometric intuition: Think of the token embedding $x$ and each expert direction $g_i$ as arrows in a high-dimensional space. The dot product measures how much they point in the same direction. If a token is about code and Expert 4 has specialized in code, their directions will point similarly, and $s_i$ will be large. If Expert 4 were a history specialist, the directions would be nearly perpendicular, and $s_i$ would be small.

Step 2: Convert scores to probabilities.

Raw scores $s_i$ are converted to proper probabilities using softmax:

$$p_i=\frac{e^{s_i}}{{\sum }_{j=1}^N{e}^{s_j}}$$

What this means:

• $p_i$ — routing probability for expert $i$; describes how much the router “prefers” expert $i$ for this token
• The denominator normalizes across all $N$ experts so probabilities sum to 1
• Exponentiation amplifies differences: an expert with slightly higher score gets disproportionately higher probability

Step 3: Compute the output.

Dense MoE (all experts contribute, weighted by their probability):

$$\hat{y}=\sum _{i=1}^N{p}_i{E}_i(x)$$

• $E_i(x)$ — the output of expert $i$ applied to the token $x$
• $p_i$ — the routing probability (weight) for expert $i$
• $\hat{y}$ — the final output: a weighted average of all expert outputs

This is rarely used in practice because it provides no compute savings.

Sparse MoE (only the top-$k$ experts contribute):

$$\hat{y}=\sum _{i\in {\mathcal{T}}_k}{p}_i{E}_i(x)$$

• ${\mathcal{T}}_k$ — the set of indices of the $k$ experts with the highest routing probabilities
• Experts not in ${\mathcal{T}}_k$ are never evaluated; their contribution is exactly zero without any computation

This is the modern standard. Experts outside the top-$k$ are simply skipped.

Special case — Top-1 / Switch Transformer:

$$i^{\ast }=\arg \underset{i}{\max }{p}_i\hat{y}=p_{i^{\ast }}{E}_{i^{\ast }}(x)$$

• $i^{\ast }$ — the single expert with the highest score
• Only one expert runs; this halves the compute cost compared to Top-2
• The tradeoff: there’s no backup expert. If the router makes a bad decision for a token, there’s no correction mechanism.

5.5 — Dense MoE vs Sparse MoE

Concrete examples:

• Mixtral 8×7B: $N=8$ experts, Top-$k=2$ → 2 of 8 experts activate per token
• DeepSeek-V3: $N=256$ fine-grained experts, Top-$k=8$ → 8 of 256 experts activate per token

5.6 — Training Pathologies: What Goes Wrong

MoE sounds perfect in theory. In practice, training it naively produces a fundamental failure mode called routing collapse.

The Routing Collapse Problem

At the very start of training, all experts are identically random — they’re equally bad at everything. Token routing at this stage is essentially arbitrary.

By pure chance, one expert might receive slightly more tokens in the first few batches. What happens next is a self-reinforcing loop:

$$f_i\uparrow \Rightarrow \nabla W_i\uparrow \Rightarrow \text{Quality}(E_i)\uparrow \Rightarrow p_i\uparrow \Rightarrow f_i\uparrow$$

Reading it out: Expert $i$ gets more tokens → receives more gradient updates → becomes better → router assigns it higher probability → receives even more tokens → …

This feedback loop accelerates. Within a few thousand training steps, one expert dominates the routing, absorbing 70–90% of all tokens. The other experts receive almost no tokens, almost no gradient signal, and remain essentially random — they are called dead experts.

The result: you paid for $N$ experts but effectively trained one. The MoE is no better than a dense model, with extra routing overhead making it worse.

Why Dead Experts Are Permanent

A dead expert receives almost no tokens, so:

$$\nabla W_i\approx 0$$

With zero gradients, no parameter updates occur. The expert stays random indefinitely. Without an external intervention, it will never recover.

Expert Capacity and Token Overflow

A secondary problem: even with good routing, GPUs have fixed memory. Each expert can only hold and process a limited number of tokens per batch. Define the capacity $C$ as:

$$C=\frac{T}{N}\times \text{CF}$$

• $T$ — total tokens in the batch
• $N$ — number of experts
• $\text{CF}$ — capacity factor, a constant greater than 1 (e.g., 1.25) that provides headroom above the average load $T/N$

When an expert receives more than $C$ tokens, the excess tokens are overflow tokens. Classical systems simply drop them: instead of $y=E(x)$, they pass $y=x$ (just the residual, unchanged). Information is lost.

Modern systems (vLLM, Mixtral) prefer dropless approaches that process all tokens regardless, accepting uneven expert loads as a lesser evil.

5.7 — Remedies: How We Fix Routing Collapse

Remedy 1: Load Balancing Loss (Switch Transformer, Fedus et al. 2021)

The Switch Transformer introduced an auxiliary loss term that penalizes expert imbalance. First, two statistics are defined for each training batch:

• $f_i$ = fraction of tokens routed to expert $i$ (discrete, measured from actual routing decisions)
• $P_i$ = average routing probability assigned to expert $i$ by the router (continuous, differentiable)

The auxiliary loss is:

$$L_{\text{balance}}=\alpha N\sum _{i=1}^N{f}_i{P}_i$$

What each symbol means:

• $\alpha$ — a small scalar hyperparameter (typically around $10^{-2}$) that controls how strongly balancing is enforced vs. language modeling
• $N$ — number of experts (the normalization makes the target value of the loss independent of $N$)
• $f_i$ — how many tokens actually went to expert $i$ this batch
• $P_i$ — how strongly the router prefers expert $i$

Intuition: The product $f_i{P}_i$ is large when expert $i$ both receives many tokens and has high router probability — i.e., when it is dominating. Minimizing the sum of these products pressures the router to distribute tokens more evenly.

Why this works mechanically: Since $P_i$ is differentiable (it comes from the softmax), the gradient $\frac{\partial L_{\text{balance}}}{\partial P_i}$ flows into the router’s parameters. Overloaded experts get their $P_i$ reduced; underloaded ones get their $P_i$ increased. Over thousands of steps, this pushes routing toward uniform distribution.

Total training loss:

$$L=L_{\text{LM}}+L_{\text{balance}}$$

The fundamental tension: $L_{\text{LM}}$ says “send this token to the best possible expert.” $L_{\text{balance}}$ says “but also spread tokens evenly.” These goals conflict — the model is sometimes forced to route tokens to a suboptimal expert just for balance. DeepSeek later solved this tension (Section 5.8).

Remedy 2: Noisy Top-K Gating (Shazeer et al. 2017)

The original MoE paper adds Gaussian noise to router scores before the top-k selection:

$$s_i^{\prime }=s_i+{ϵ}_i,{ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$$

• $s_i^{\prime }$ — noisy score for expert $i$
• ${ϵ}_i$ — random noise sampled from a zero-mean Gaussian with variance ${\sigma }^2$
• The top-k selection then uses $s_i^{\prime }$ instead of $s_i$

Why this helps: Noise forces exploration. Even if Expert 3 has the highest raw score by a small margin, random noise might occasionally let Expert 7 win the top-k lottery. Expert 7 then receives gradient signal, learns something useful, and becomes competitive — breaking the early-stage collapse before it can entrench.

Think of it as randomized exploration, similar to the $ϵ$-greedy strategy in reinforcement learning.

Remedy 3: Router Z-Loss (ST-MoE)

A separate failure mode: router logits can grow very large in magnitude during training. When this happens, the softmax function saturates — one expert gets nearly all the probability mass and the rest get almost zero. For example, if logits are $[100,-20,-30]$, the softmax produces approximately $[1.0,0.0,0.0]$.

The z-loss penalizes large logit magnitudes:

$$L_z=\frac{1}{B}\sum _{x\in B}{\left(\log \sum _i{e}^{z_i(x)}\right)}^2$$

• $B$ — the number of tokens in the batch
• $z_i(x)$ — the raw logit (pre-softmax score) that expert $i$ assigns to token $x$
• $\log {\sum }_i{e}^{z_i(x)}$ — the log-sum-exp of all logits (the log of the denominator in softmax); this grows when logits are large
• The squared term heavily penalizes large values

Intuition: When logits are small, $\log {\sum }_i{e}^{z_i(x)}$ is bounded. When one logit blows up to 100, the log-sum-exp is dominated by that term and becomes large, incurring heavy penalty. This gently constrains the logits to remain in a reasonable range without directly interfering with routing decisions.

5.8 — DeepSeekMoE: The Modern Evolution

DeepSeek identified three fundamental problems with classical MoE and proposed architectural solutions for each.

Background: Why Classical MoE Wastes Expert Capacity

Imagine training an MoE on a large English text corpus. Every single token — regardless of topic — needs basic English grammar, punctuation logic, and common word relationships. This means all experts must learn these common patterns just to function.

After training, you might find:

• Expert 1 knows grammar + some math
• Expert 2 knows grammar + some code
• Expert 3 knows grammar + some history

The grammar knowledge is redundantly duplicated across all experts. Those parameters are wasted — they could have encoded more specialized knowledge instead.

Innovation 1: Shared Experts

DeepSeek separates expert knowledge into two tiers:

Shared experts — always active for every token, no routing needed. These handle universal knowledge.

Routed experts — only a subset activates per token, selected by the router. These handle specialized knowledge.

The output formula becomes:

$$\hat{y}=E_{\text{shared}}(x)+\sum _{i\in {\mathcal{T}}_k}{p}_i{E}_i(x)$$

• $E_{\text{shared}}(x)$ — output from the shared expert (always computed)
• ${\sum }_{i\in {\mathcal{T}}_k}{p}_i{E}_i(x)$ — weighted sum of selected routed experts (only $k$ of $N$ computed)

Analogy: Think of a university degree. Every student takes the core curriculum (shared expert) — writing, math, critical thinking. Then each student takes electives (routed experts) — neuroscience, software engineering, law. The core curriculum doesn’t need to be duplicated in every elective. Shared experts work the same way.

Innovation 2: Fine-Grained Expert Segmentation

Classical MoE uses a small number of large experts (e.g., 8 experts, Top-2). The granularity of routing combinations is limited:

$$\left(\genfrac{}{}{0ex}{}{8}{2}\right)=28\text{ possible expert combinations}$$

DeepSeek instead uses many small experts (e.g., 256 experts, Top-8):

$$\left(\genfrac{}{}{0ex}{}{256}{8}\right)\approx 10^{14}\text{ possible expert combinations}$$

The parameter budget is the same — each small expert is $m$ times smaller if there are $m$ times as many of them. But the routing combinations are astronomically larger, allowing the model to form exquisitely specific “expert ensembles” tailored to each token’s unique requirements.

Example: A token like eigenvalue might need a blend of linear algebra, numerical analysis, and physics notation. With 256 micro-experts, the router can select precisely the three sub-experts covering each of those domains. With 8 large experts, the best you can do is pick the two coarsest relevant categories.

Math: If we segment $N$ experts into $mN$ smaller experts and increase selection from $k$ to $mk$, the total compute per token stays the same (more experts, each smaller, same number selected). But specialization improves dramatically.

Innovation 3: Auxiliary-Loss-Free Balancing (DeepSeek-V3)

This is arguably DeepSeek’s most significant contribution to MoE training.

The core problem: $L_{\text{balance}}$ contaminates the language model gradient. Router parameters receive two competing gradient signals simultaneously — one from $L_{\text{LM}}$ (optimize language modeling) and one from $L_{\text{balance}}$ (spread tokens evenly). This causes the router to sometimes choose a suboptimal expert just to satisfy the balancing constraint, directly harming output quality.

DeepSeek’s solution: Separate the balancing mechanism from the gradient computation entirely, by introducing an expert bias $b_i$ for each expert:

$$s_i^{\prime }=s_i+b_i\text{(used only for routing selection)}$$

• $s_i$ — the original routing score from the router
• $b_i$ — a learned bias term for expert $i$; adjusted after each batch based on expert load
• $s_i^{\prime }$ — the adjusted score used to decide which $k$ experts to select

Bias update rule (no gradients involved):

$$b_i\leftarrow b_i-\gamma \text{if expert }i\text{ is overloaded}$$ $$b_i\leftarrow b_i+\gamma \text{if expert }i\text{ is underloaded}$$

where $\gamma$ is a small step size.

The critical detail: Once the top-$k$ experts are selected (using $s_i^{\prime }$), the output weighting uses the original scores $s_i$ (not $s_i^{\prime }$):

$$\hat{y}=\sum _{i\in {\mathcal{T}}_k(s^{\prime })}{p}_i(s)E_i(x)$$

• ${\mathcal{T}}_k(s^{\prime })$ — the top-$k$ experts selected using the biased scores $s^{\prime }$
• $p_i(s)$ — the weights computed from the original unbiased scores $s$

This means: the bias term nudges which experts get selected (promoting underused ones), but once selected, experts are weighted by their true router confidence. The bias never enters the gradient computation. $L_{\text{LM}}$ remains pure.

Why this is elegant: Load balancing is now a simple feedback control loop (like a thermostat), completely separate from deep learning optimization. The language model and the routing policy are finally decoupled.

MoE Evolution Timeline

Shazeer et al. 2017   →  Original Sparse MoE: sparsely-gated top-k routing
GShard (Google) 2020  →  Top-2 routing scaled to trillion parameters
Switch Transformer    →  Top-1 routing + load balancing auxiliary loss (2021)
Mixtral 8×7B  2024    →  First widely-used open-source MoE (N=8, k=2)
DeepSeekMoE   2024    →  Shared experts + fine-grained segmentation
DeepSeek-V3  2024-25  →  Aux-loss-free bias balancing; 671B total / 37B active

5.9 — The Full Decoding Pipeline

Once an LLM has processed a prompt, it produces a probability distribution over the next token. The decoding strategy decides which token to actually pick. This section covers every major decoding strategy, from simplest to most sophisticated.

Here is the complete journey from a prompt to a generated token:

Prompt (tokens x₁, x₂, ..., xₜ)
              ↓
   Transformer (deterministic computation)
              ↓
   Logits  z ∈ ℝ^|V|    (one raw score per vocabulary token)
              ↓
   Temperature scaling  (optional: z → z/T)
              ↓
   Softmax
              ↓
   Probabilities  p ∈ [0,1]^|V|    (sum to 1)
              ↓
   Filtering (Top-k or Top-p)  (optional)
              ↓
   Decoding Strategy  (argmax, beam search, or sample)
              ↓
   Chosen token  xₜ₊₁
              ↓
   Append to context → repeat

The most important insight in this section:

When GPT or Claude gives you a different response to the same question, it’s not because the model itself changed — it’s because the sampling step draws from a probability distribution, introducing stochasticity.

5.10 — Greedy Decoding

What it is: The simplest possible decoding strategy. At every step, pick the single token with the highest probability:

$$x_{t+1}=\arg \underset{v}{\max }p(v\mid x_1,\dots ,x_t)$$

• $v$ — a candidate vocabulary token
• $p(v\mid x_1,\dots ,x_t)$ — the probability of token $v$ given all previous tokens

Example:

"The cat sat on the ___"

mat    45%  ← argmax picks this
floor  25%
chair  15%
roof    8%

Greedy always picks “mat” here. Every time. Forever.

Why it exists: It is fast, reproducible, and requires zero randomness. Useful when you need deterministic outputs.

Why it fails: Locally optimal choices do not guarantee globally optimal sequences. The best next word does not always lead to the best sentence. Greedy decoding leads to repetitive, safe, generic outputs — it keeps returning to the same token patterns because it never explores alternatives.

5.11 — Beam Search

What it is: Instead of committing to one token at each step (greedy), beam search maintains $k$ candidate sequences — called beams — simultaneously. At each step, every beam is expanded into its $k$ most probable continuations, and the best $k$ overall sequences (by cumulative score) are kept.

Scoring: Each sequence is scored by the sum of log-probabilities of its tokens:

$$\text{score}(x_1,\dots ,x_t)=\sum _{\tau =1}^t\log p(x_{\tau }\mid x_{<\tau })$$

• $\log p(x_{\tau }\mid x_{<\tau })$ — log probability of token $x_{\tau }$ given all prior tokens
• Summing log-probabilities is equivalent to multiplying probabilities (log turns multiplication into addition, which prevents numerical underflow)

Why log-probabilities? Since all probabilities are in $(0,1)$, multiplying many of them together gives astronomically small numbers. Log transforms them to negative numbers in a manageable range. Higher (less negative) log-probability = better sequence.

How it works (example with $k=2$):

Step 0: Start from [BOS]

Step 1: Expand → {a (p=0.2), the (p=0.6)}
  Keep top 2 beams: "a" and "the"

Step 2: Expand each beam
  "a"   → {cute (p=0.7), big (p=0.2)}
  "the" → {red (p=0.5), small (p=0.4)}
  Keep top 2 by cumulative score

Step 3: Continue until [EOS] token
  Return the highest-scoring complete sequence

Why it’s better than greedy: Greedy might pick “a” as the first word (p=0.2) even though “the” (p=0.6) leads to a much better overall sentence. Beam search explores “the” as well, and can discover that “the cute bear” has higher total probability than “a big bear.”

Where it’s used: Machine translation, speech recognition, document summarization — tasks where you want the single most probable output.

Why chatbots don’t use it: Beam search optimizes for the most statistically probable response, which tends to be safe, bland, and repetitive. It has no diversity mechanism. For open-ended chat, the most probable response is usually the most boring one. Modern assistants use sampling-based approaches instead.

5.12 — Pure Sampling, Top-k, and Top-p

Pure Sampling

Instead of picking the most probable token, draw $x_{t+1}$ randomly according to the full probability distribution $p$. Higher-probability tokens are chosen more often, but lower-probability tokens still get occasional chances.

This creates variety — the model can express different ideas and phrasings rather than always defaulting to the safest choice. But it also risks selecting nearly-impossible tokens (probability 0.00001%) that produce incoherent text.

Top-k Sampling

What it is: Keep only the $k$ most probable tokens. Discard the rest. Renormalize the remaining probabilities to sum to 1, then sample from those $k$ tokens:

$$p_{\text{top-}k}(v)=\frac{p(v)\cdot 1[v\in {\mathcal{T}}_k]}{{\sum }_{u\in {\mathcal{T}}_k}p(u)}$$

• $1[v\in {\mathcal{T}}_k]$ — indicator function: 1 if token $v$ is in the top-k, 0 otherwise
• The denominator renormalizes the surviving tokens

Example with $k=3$:

Before:  mat=45%, floor=25%, chair=15%, roof=8%, banana=0.1%
After:   mat=52%, floor=29%, chair=17%  (roof and banana discarded)

Weakness: $k$ is fixed regardless of context. When the model is very confident (e.g., one token has 95% probability), forcing Top-50 means sampling from 49 nearly-impossible tokens. When the model is very uncertain, Top-10 might cut off many reasonable continuations.

Top-p (Nucleus) Sampling

What it is: Instead of a fixed count, include the smallest set of tokens whose cumulative probability reaches a threshold $p$:

$$\mathcal{N}=\arg \underset{S}{\min }\left\{S:\sum _{v\in S}p(v)\ge p\right\}$$

• $\mathcal{N}$ — the “nucleus”: the smallest set of top tokens covering $p$ fraction of probability mass
• Typical $p\in [0.9,0.95]$

Example with $p=0.90$:

Token probabilities (sorted):
  mat    60% → cumulative: 60%
  floor  20% → cumulative: 80%
  chair  10% → cumulative: 90%  ← stop here, 90% reached

Nucleus = {mat, floor, chair}
Sample from these three.

Why Top-p is adaptive:

• When the model is confident (one token has 80% probability), the nucleus is small — just 1 or 2 tokens. We don’t waste samples on implausible words.
• When the model is uncertain (probabilities are spread evenly), the nucleus is large — 20, 30, or more tokens. The model has freedom to be creative.

Top-k vs Top-p comparison:

5.13 — Temperature Scaling

What it is: A scalar parameter $T>0$ applied to the raw logits before the softmax. It reshapes how “peaked” or “flat” the resulting probability distribution is:

$$p_T(v)=\frac{\exp (z_v/T)}{{\sum }_u\exp (z_u/T)}$$

• $z_v$ — the raw logit (pre-softmax score) for token $v$
• $T$ — temperature
• Dividing logits by $T$ before softmax changes the sharpness of the distribution

What temperature actually does:

Important clarification: Temperature does not change what the model knows. It doesn’t add intelligence or creativity. It only reshapes the confidence of the distribution. A model at $T=1.5$ doesn’t “know” more — it just hedges its bets more broadly.

The math intuition: Dividing all logits by a large $T$ makes them all small, and when small values go through softmax, the differences between them shrink — the distribution flattens. Dividing by a tiny $T$ makes differences between logits huge — the highest logit dominates everything.

Practical guide by task:

Full production pipeline (in order):

Raw logits z
    ↓  divide by T
Scaled logits z/T
    ↓  softmax
Probabilities p_T
    ↓  Top-k or Top-p filter
Filtered + renormalized probabilities
    ↓  sample
Next token

5.14 — Guided Decoding

The problem it solves: Standard sampling can produce structurally invalid outputs even when the prompt explicitly requests a format. Ask for JSON and the model might produce:

{name: "Saravan", age: 22,}

This is invalid JSON — missing quotes around keys, trailing comma. Even though the prompt said “return JSON,” the model just predicts likely token sequences, with no awareness of grammatical structure.

What guided decoding is: At every generation step, a grammar (JSON schema, regex, context-free grammar, finite state machine) determines exactly which tokens are valid continuations of the partially-generated output. All invalid tokens are masked to zero probability before sampling:

$$p_{\text{guided}}(v)=\frac{p(v)\cdot 1[v\in {\mathcal{V}}_{\text{valid}}]}{{\sum }_{u\in {\mathcal{V}}_{\text{valid}}}p(u)}$$

• ${\mathcal{V}}_{\text{valid}}$ — the set of tokens that are valid at the current position according to the grammar
• $1[v\in {\mathcal{V}}_{\text{valid}}]$ — masks invalid tokens to zero
• The denominator renormalizes valid tokens to sum to 1

Example for JSON generation:

Generated so far: {
Next position allows: only "}" or a string (JSON key)

Generated so far: {"name"
Next position allows: only ":"

Generated so far: {"name":
Next position allows: only a JSON value (string, number, bool, null, array, object)

At each step, invalid tokens are silently zeroed out. The model still generates — it just cannot produce structurally broken output.

Where this is used today:

• OpenAI Structured Outputs
• OpenAI JSON mode
• Function calling / tool calling
• Grammar-constrained generation (Outlines library)
• LLM API schema validation

5.15 — Context Length and Context Rot

Prompting strategies are techniques for getting better outputs from a pre-trained, frozen LLM without modifying its weights. The model stays the same; you change how you talk to it.

What Is Context Length?

Every time an LLM processes a request, it can only “see” a limited amount of text at once. This window of text is called the context window (also: context length, context size, window size).

Everything the model uses to generate its next token must fit inside this window:

┌────────────────────────────────────────────┐
│               Context Window               │
│                                            │
│  System prompt                             │
│  Chat history                              │
│  User message                              │
│  Retrieved documents                       │
│  Model's own partial output                │
│                                            │
└────────────────────────────────────────────┘

If total tokens exceed the context limit, older content gets truncated — the model literally cannot see it.

Why context length matters:

• A model with a 4K token limit cannot summarize a 50-page document in one pass.
• A model with a 200K token limit can read an entire codebase before answering a question.
• More context = more information available = better, more informed answers (in theory).

Modern models support 8K to 1M+ token contexts. But bigger context windows come with a cost: standard self-attention scales quadratically — doubling context length quadruples attention compute.

Context Rot

The surprising finding (Hong et al., 2025): Larger context does not always lead to better performance. Performance often degrades as context grows — even when the relevant information is present inside the window. This is called context rot.

The “Needle in a Haystack” benchmark demonstrates this precisely: a single key fact (the “needle”) is hidden inside $N$ pages of filler text (the “haystack”). The question requires finding and using that fact. As $N$ grows, model accuracy drops — even when the needle is within the stated context limit.

Why this happens: Attention is supposed to find and weight relevant information. But when thousands of irrelevant tokens compete with a handful of relevant ones, the signal gets diluted. The relevant tokens’ attention scores are suppressed by the sheer volume of surrounding noise.

Practical consequence: Don’t assume “give it everything” is always the best strategy. Curating the prompt to include only relevant context often outperforms stuffing in everything available. This is one core motivation for Retrieval-Augmented Generation (RAG): instead of injecting all documents, retrieve only the 3–5 most relevant paragraphs.

5.16 — Anatomy of a Good Prompt

A well-structured prompt typically contains four components:

Why this framework matters: When a prompt produces a bad output, the problem is almost always that one of these four components is missing, vague, or contradictory. Debugging a prompt = diagnosing which component failed.

Ordering tip: Context usually comes first (sets the frame), instructions next (sets the task), input after (provides the material), constraints last (sets the rules).

5.17 — In-Context Learning: Zero-Shot and Few-Shot

What is In-Context Learning (ICL)?

ICL is the remarkable ability of a pre-trained LLM to adapt to a completely new task using only examples embedded in the prompt — without any weight updates, gradient computation, or fine-tuning whatsoever.

Zero-Shot Learning

You ask the model to perform a task without providing any examples. The model relies entirely on patterns learned during pre-training.

Prompt: "Translate to Spanish: Hello, how are you?"
Output: "Hola, ¿cómo estás?"

Zero-shot works because large pre-training datasets exposed the model to many tasks implicitly. The model generalizes.

Few-Shot Learning

You provide $m$ input-output examples before the actual question. The model infers the pattern and extends it:

Prompt:
Dog → Perro
Cat → Gato
Horse → Caballo
Bird → ?

Output: Pájaro

Few-shot typically outperforms zero-shot because the examples remove ambiguity about the desired task, format, and style.

What is architecturally happening?
The model is not “learning” in the traditional ML sense (no parameters change). Instead, the attention mechanism uses the example pairs as temporary working memory. When generating the answer, every attention head can attend back to the examples in the context. The model essentially pattern-matches on the examples at inference time. Pre-trained weights supply the general language capability; the in-context examples supply the task specification.

Cost tradeoffs:

Counterintuitive finding: Modern reasoning-specialized models (OpenAI o3, DeepSeek-R1) sometimes perform worse with few-shot examples than with zero-shot. Why? The examples can accidentally constrain the model’s reasoning — it may copy the solution method demonstrated in the examples instead of deriving the optimal approach from scratch. For such models, a clear task description plus “think step by step” often outperforms explicit examples.

5.18 — Chain of Thought (CoT)

Paper: Wei et al., 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

The Problem with Direct Answering

A standard LLM answers by computing:

$$P(A\mid Q)$$

the probability of an answer $A$ given a question $Q$. The model jumps directly from question to answer in one forward pass. For simple questions, this is fine. For complex questions requiring multiple steps of reasoning (multi-step math, logic puzzles, planning), it often fails — the model tries to compress an entire reasoning chain into a single next-token prediction.

Human analogy: if asked $847\times 932$, you wouldn’t just magically know the answer. You’d compute intermediate steps: partial products, carries, additions. The intermediate steps are essential.

What CoT Does

By adding “Think step by step” to the prompt (or providing a few examples with explicit reasoning), the model generates intermediate reasoning steps before the final answer:

Standard prompting: $$P(A\mid Q)$$

Chain of Thought: $$P(A,R\mid Q)=P(R\mid Q)\cdot P(A\mid R,Q)$$

• $Q$ — the question
• $R$ — the reasoning chain (sequence of intermediate steps)
• $A$ — the final answer

The factored form reveals why CoT helps: $A$ is now conditioned on both $Q$ and $R$. The model’s answer is informed by its own explicitly generated reasoning. Each reasoning step becomes new context that the model attends to when generating subsequent steps.

Without CoT:
Question → [giant jump] → Answer
             (often wrong)

With CoT:
Question → Step 1 → Step 2 → Step 3 → Answer
              ↑         ↑         ↑
        (each step enriches context for the next)

Mechanistic view: In standard decoding, the hidden state $h_t$ must encode the entire solution implicitly. With CoT, intermediate reasoning tokens are written to the context window. Each step’s computation is offloaded from the internal representation to the external token stream. Small, explicit steps are individually easier to get right than one large implicit computation.

Why CoT scales with model size:
CoT improves dramatically for models above ~100B parameters (Wei et al., 2022). Smaller models cannot generate coherent multi-step reasoning because they lack the prerequisite knowledge (arithmetic, logical deduction, planning) needed for the intermediate steps to be meaningful. CoT doesn’t install new capabilities — it unlocks existing ones.

Interpretability bonus: When a model gives a wrong answer without CoT, you have no idea where it went astray. With CoT, the reasoning chain is inspectable — you can see exactly which step introduced the error. This makes debugging and improving prompts vastly easier.

Cost: Every reasoning token is generated autoregressively. A 10-step reasoning chain before the answer means 10 extra forward passes before the model can output the final answer. Longer reasoning = higher latency and cost, proportional to the number of reasoning tokens.

5.19 — Self-Consistency

Paper: Wang et al., 2022. Self-Consistency Improves Chain of Thought Reasoning in Language Models.

The Remaining Problem with CoT

Even with CoT, a single reasoning path can go wrong. The model might follow a plausible-looking but incorrect line of reasoning and arrive at a wrong answer confidently.

The Self-Consistency Solution

Instead of generating one reasoning path and trusting its answer, generate $n$ independent reasoning paths (using temperature > 0 so they differ), extract the final answer from each, and take a majority vote:

$$P(A)=\sum _{i}P(A\mid R_i)P(R_i)$$

• $R_i$ — the $i$-th sampled reasoning path
• $P(A\mid R_i)$ — probability that reasoning path $R_i$ leads to answer $A$
• $P(A)$ — marginalized probability of answer $A$ across all reasoning paths
• In practice: just count how many paths gave each answer; pick the most common one

Example:

Run 1: "The train travels 60 km/h for 3 hours = 180 km."  → Answer: 180
Run 2: "Speed × time = 60 × 3 = 180."                    → Answer: 180
Run 3: "3 hours at 60 km/h: 3 × 60 = 80. Wait, that's wrong. 180." → Answer: 180
Run 4: "Distance = 6×10 × 3 = 170?"                       → Answer: 170
Run 5: "60 km × 3 = 180 km."                              → Answer: 180

Majority vote: 180 (4 out of 5 paths agree)
Final answer: 180 ✓

Why it works: When multiple independent reasoning strategies — using different methods, notation, and logic — all converge on the same answer, the probability that the answer is correct is high. Disagreement across paths is a signal of uncertainty; agreement is a signal of confidence.

Reported improvement: Wang et al. show 10–20 percentage point improvements over standard CoT on arithmetic, commonsense, and symbolic reasoning benchmarks.

Cost/quality tradeoff: Running $n$ reasoning paths costs $n$ times the compute and latency. Typical values: $n\in [5,40]$. The improvement tapers off at larger $n$, so there’s a sweet spot.

Connection to modern reasoning models:
OpenAI o-series, DeepSeek-R1, and Gemini Thinking internalize this idea. Instead of generating $n$ separate external paths and voting, these models are trained to internally generate, evaluate, prune, and revise reasoning paths before committing to an answer — effectively performing search over reasoning chains as part of the forward computation. Self-consistency is the conceptual ancestor of these reasoning models.