Inside Modern Attention: Positions, Normalization and Efficiency

3.1 — The Attention Mechanism

Why Self-Attention at All

Before transformers, language models used RNNs, which read a sentence one word at a time, left to right. If word 1 mattered for understanding word 100, that information had to survive a 99-step relay race through every word in between. Over long distances, it faded — this is the classic “long-range dependency” problem.

Self-Attention: The Core Idea

Take a sentence of $n$ tokens. Each token $i$ starts life as an embedding vector $x_i\in {\mathbb{R}}^d$ — just a list of $d$ numbers representing that word’s meaning, with no notion of position yet.

We learn three weight matrices that transform each embedding into three different-purpose vectors:

$$W_Q\in {\mathbb{R}}^{d\times d_k},W_K\in {\mathbb{R}}^{d\times d_k},W_V\in {\mathbb{R}}^{d\times d_v}$$

where:

• $W_Q$ — the Query projection matrix. Learned during training.
• $W_K$ — the Key projection matrix. Learned during training.
• $W_V$ — the Value projection matrix. Learned during training.
• $d$ — the size of the input embedding for each token.
• $d_k$ — the size of the Query and Key vectors (they must match, since we’ll take their dot product).
• $d_v$ — the size of the Value vectors (can differ from $d_k$).

For every token, we multiply its embedding by each matrix to get three vectors:

$$q_i=x_i{W}_Q\text{(Query — "what am I looking for?")}$$

$$k_i=x_i{W}_K\text{(Key — "what do I offer?")}$$

$$v_i=x_i{W}_V\text{(Value — "what information do I actually carry?")}$$

where: $x_i$ is token $i$’s embedding; $q_i,k_i,v_i$ are the Query, Key, and Value vectors produced for token $i$.

A useful analogy: imagine a library. Every book (token) has a Key — a short label describing what it’s about. Every reader (also a token, since every token both asks and answers) has a Query — what they’re currently searching for. When a Query matches a Key well, that book’s Value — its actual content — gets pulled in and read.

In matrix form, stack every token’s embedding into one big matrix $X\in {\mathbb{R}}^{n\times d}$ (one row per token) and do all the projections at once:

$$Q=X{W}_Q,K=X{W}_K,V=X{W}_V$$

where: $Q\in {\mathbb{R}}^{n\times d_k}$, $K\in {\mathbb{R}}^{n\times d_k}$, $V\in {\mathbb{R}}^{n\times d_v}$ — each row corresponds to one token’s Query, Key, or Value vector.

The full attention computation is:

$$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

where:

• $Q{K}^T$ — every Query dotted with every Key, producing a raw “relevance score” for every pair of tokens.
• $\sqrt{d_k}$ — a scaling factor explained in the next section.
• $\text{softmax}(\cdot )$ — turns each row of scores into a probability distribution (values between 0 and 1, summing to 1 per row).
• The final multiplication by $V$ — blends the Value vectors according to those probabilities.

Why Divide by the Square Root of d_k

The dot product $q\cdot k$ is a sum of $d_k$ individual multiplications. As $d_k$ grows, this sum tends to grow larger in magnitude too (more terms being added up). Very large scores push softmax into a “saturated” regime — one score dominates completely, the rest collapse toward zero, and the gradients used for learning become vanishingly small everywhere.

Dividing by $\sqrt{d_k}$ rescales the scores so their variance stays close to 1 regardless of how large $d_k$ is. It’s purely a numerical-stability fix — it keeps softmax’s inputs in a healthy range so training gradients keep flowing.

where: $d_k$ is the dimensionality of the Query/Key vectors (defined in 1.2).

What the QK Transpose Matrix Looks Like

$$Q{K}^T\in {\mathbb{R}}^{n\times n}$$

where: $n$ is the number of tokens in the sequence. The result is a square matrix — one row and one column per token.

Entry $(i,j)$ of this matrix is $q_i\cdot k_j$: how strongly token $i$’s Query matches token $j$’s Key — in other words, how much token $i$ should attend to token $j$. After the softmax, every row sums to exactly 1, so row $i$ becomes a clean probability distribution describing “how token $i$ splits its attention across the whole sequence.”

Multiplying this weight matrix by $V$ then gives each token a new representation: a weighted average of every token’s Value vector, weighted exactly by that attention distribution. This new representation is what carries contextual information forward into the rest of the network.

The Permutation Invariance Problem

Here’s the catch. Everything in “Self-Attention: The Core Idea” (Section 3.1) — $Q{K}^T$, softmax, the multiplication by $V$ — is built entirely out of dot products between vectors. Nowhere does the actual order of tokens enter the computation.

If you shuffle the input tokens:

$$[x_1,x_2,x_3]⟶[x_3,x_1,x_2]$$

self-attention produces the exact same pairwise relevance scores, just relabeled. It has no way to know that $x_1$ originally came first.

This property is called permutation invariance — the function’s output doesn’t change based on token order, only on which tokens are present.

Multi-Head Attention (MHA)

A single attention computation learns one type of relationship between tokens — say, subject-verb agreement. But language has many simultaneous relationship types: syntax, coreference (“it” refers to what?), semantic similarity, and more. Multi-Head Attention runs $h$ independent attention computations (“heads”) in parallel, each with its own learned projections, so each head can specialize in a different kind of relationship.

For head $i$:

$$W_Q^{(i)}\in {\mathbb{R}}^{d\times d_k},W_K^{(i)}\in {\mathbb{R}}^{d\times d_k},W_V^{(i)}\in {\mathbb{R}}^{d\times d_v}$$

$${\text{head}}_i=\text{Attention}\left(Q{W}_Q^{(i)},K{W}_K^{(i)},V{W}_V^{(i)}\right)$$

where: $h$ is the total number of heads; $W_Q^{(i)},W_K^{(i)},W_V^{(i)}$ are head $i$’s own private projection matrices, distinct from every other head’s.

Every head’s output gets glued together side-by-side (concatenated) and passed through one more learned matrix to mix the heads back into a single representation:

$$\text{MHA}(Q,K,V)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)\cdot W_O$$

where: $W_O\in {\mathbb{R}}^{h{d}_v\times d}$ is the output projection matrix, which combines all $h$ heads’ outputs into one vector of the original model dimension $d$.

Typical setup: $d=512$ with $h=8$ heads gives $d_k=d_v=64$ per head — the total parameter count is kept similar to a single wide head, but the model gets $8$ independent “perspectives” instead of $1$.

The KV Cache Problem

During inference — generating text one token at a time — producing the next token requires attending to every previous token’s Key and Value vectors. Recomputing all of them from scratch at every single step, for every new token, would be enormously wasteful, since past tokens’ Keys and Values never change once computed.

The fix: compute each token’s $K$ and $V$ once, and store them in memory for reuse at every future step. This stored memory is the KV Cache.

The problem is size. The cache must hold Key and Value vectors for every past token, across every head, across every layer:

$$\text{KV Cache size}=2\times n\times h\times d_v\times \text{bytes per element}$$

where:

• $2$ — because we store both $K$ and $V$.
• $n$ — number of tokens processed so far (grows as generation continues).
• $h$ — number of attention heads.
• $d_v$ — size of each head’s Value vector.
• bytes per element — how many bytes each number takes (e.g., 2 bytes for FP16).

For a 1-million-token context with 128 heads, this reaches hundreds of gigabytes. This cache — not raw compute — is the main bottleneck for serving long-context language models efficiently.

Multi-Query Attention (MQA)

Idea: Keep every head’s own Query, but force all heads to share one single Key and one single Value.

$$G=1\Rightarrow \text{Multi-Query Attention (MQA)}$$

where: $G$ is the number of distinct Key/Value groups. Setting $G=1$ means there’s only one shared group for all heads.

$${\text{head}}_i=\text{Attention}(Q_i,K,V)$$

where: $Q_i$ is head $i$’s own private Query, but $K$ and $V$ are the same shared vectors used by every head.

KV cache reduction: Instead of storing $h$ separate Key vectors and $h$ separate Value vectors, you store just $1$ of each — an $h\times$ reduction in cache size.

Tradeoff: Quality dips slightly, because every head is now forced to see the same Key/Value “view” of each token — the diversity that made multiple heads useful in the first place is partly lost. The memory savings, though, are large.

Grouped Query Attention (GQA)

Idea: A middle ground between full MHA (every head has its own K/V) and MQA (all heads share one K/V). Split the $h$ query heads into $G$ groups; heads inside the same group share one Key and Value.

$$1<G<h\Rightarrow \text{Grouped Query Attention (GQA)}$$

For group $g$, containing query heads $Q_{g,1},\dots ,Q_{g,h/G}$:

$${\text{head}}_{g,i}=\text{Attention}(Q_{g,i},K_g,V_g)$$

where: $K_g,V_g$ are the single shared Key and Value vectors for every head inside group $g$; $h/G$ is the number of query heads per group.

KV cache size ends up $G\times$ smaller than full MHA, and $h/G\times$ larger than MQA — a tunable sweet spot. GQA is used in LLaMA 2, Mistral, and most modern open models, because it recovers almost all of MHA’s quality while cutting cache size substantially.

Multi-Head Latent Attention (MLA)

MQA and GQA save memory by making heads share Keys and Values. MLA (DeepSeek’s approach) takes a completely different route: it compresses the K/V information itself into a much smaller vector, before it’s ever split into heads.

The core compression trick

Instead of storing a full-size Key and Value for every head, MLA stores one small latent vector per token:

$$c_{KV}=x\cdot W_{\downarrow }^{KV},W_{\downarrow }^{KV}\in {\mathbb{R}}^{d\times d_c}$$

where:

• $x$ — the token’s input embedding.
• $W_{\downarrow }^{KV}$ — the down-projection matrix (the $\downarrow$ marks “compressing down”).
• $c_{KV}\in {\mathbb{R}}^{d_c}$ — the compressed latent vector actually stored in the cache.
• $d_c$ — the latent dimension, chosen to be much smaller than the total size of all heads’ K/V combined ($d_c\ll d_k\cdot h$).

When attention actually needs to run, each head reconstructs its own Key and Value by up-projecting the shared latent:

$$K^{(i)}=c_{KV}\cdot W_{\uparrow }^{K,(i)},V^{(i)}=c_{KV}\cdot W_{\uparrow }^{V,(i)}$$

where: $W_{\uparrow }^{K,(i)},W_{\uparrow }^{V,(i)}$ are head $i$’s private up-projection matrices (the $\uparrow$ marks “decompressing back up” to full size), applied on the fly rather than stored.

What actually sits in the cache: just $c_{KV}$, a vector of size $d_c$ — not $h$ separate Keys and $h$ separate Values.

$$\text{KV Cache (MLA)}=d_c\text{vs.}\text{KV Cache (full MHA)}=2h{d}_k$$

If $d_c\ll 2h{d}_k$, this is a massive memory win — and unlike MQA/GQA, no head is forced to share its view; each head still gets a full, unique Key and Value reconstructed from the shared latent.

The matrix absorption trick (an inference speed-up)

At inference time, head $i$’s attention score involves:

$$q_i\cdot k_i=\left(x{W}_{\downarrow }^Q{W}_{\uparrow }^{Q,(i)}\right)\cdot \left(c_{KV}{W}_{\uparrow }^{K,(i)}\right)$$

where: $W_{\downarrow }^Q$ and $W_{\uparrow }^{Q,(i)}$ are the equivalent down/up-projection matrices on the Query side; $x$ is the current token’s embedding.

Because matrix multiplication is associative, $W_{\uparrow }^{Q,(i)}$ and $W_{\uparrow }^{K,(i)}$ can be pre-multiplied together once, offline, into a single fixed matrix:

$$A^{(i)}=W_{\uparrow }^{Q,(i)}{\left(W_{\uparrow }^{K,(i)}\right)}^T$$

where: $A^{(i)}$ is this pre-merged matrix for head $i$ — computed once, ahead of time, and reused at every inference step.

The dot product then simplifies to:

$$q_{\downarrow }^{(i)}\cdot A^{(i)}\cdot c_{KV}$$

The consequence: the model never has to explicitly reconstruct the full Key vector $K^{(i)}$ at inference time — it applies $A^{(i)}$ directly to the cached latent $c_{KV}$. The same trick merges the Value up-projection with the output matrix $W_O$.

The payoff: MLA gets the memory footprint and inference speed roughly on par with MQA, but the attention quality of full multi-head attention — because it never actually threw away per-head information, it just compressed and later reconstructed it.

3.2 — Why We Need Positional Encoding

The Properties We Need

Since self-attention is order-blind (“The Permutation Invariance Problem” (Section 3.1)), we need to explicitly inject position information via a positional encoding vector $p_m\in {\mathbb{R}}^d$ for each position $m$. A good one should satisfy:

1. Uniqueness: $p_m\ne p_n$ whenever $m\ne n$ — every position gets a distinct signature.
2. Smoothness: nearby positions get similar-looking encodings (position 5 should resemble position 6 more than it resembles position 5000).
3. Relative recoverability: from $p_m$ and $p_n$ alone, the model should be able to work out the distance $m-n$.
4. Extrapolation: it should keep working sensibly on sequences longer than any seen during training.

The standard way to inject it is simple addition to the word embedding before anything else happens:

$$z_i=x_i+p_i$$

where: $x_i$ is token $i$’s word embedding (pure meaning, no position); $p_i$ is its positional encoding; $z_i$ is the combined vector that actually enters the attention and feed-forward layers.

Two Families of Positional Encoding

Learned (absolute): a trainable lookup table $P\in {\mathbb{R}}^{L_{max}\times d}$, where position $m$ simply gets whatever vector $P[m]$ the model learned during training. where: $L_{max}$ is the maximum sequence length seen during training. Problem: since it’s just a lookup table, it has no idea what to output for positions beyond $L_{max}$ — it cannot extrapolate.

Rule-based (fixed): a mathematical formula, with no learned parameters, that can be evaluated at any position — including ones never seen in training. This is the family Parts 3–5 explore.

3.3 — Sinusoidal Positional Encoding

The Formula

The original “Attention Is All You Need” paper defines position $m$’s encoding using sine and cosine pairs across the embedding dimensions:

$$\text{PE}(m,2i)=\sin \left(\frac{m}{10000^{2i/d}}\right),\text{PE}(m,2i+1)=\cos \left(\frac{m}{10000^{2i/d}}\right)$$

where:

• $m$ — the token’s position (0, 1, 2, …).
• $d$ — the total embedding dimension.
• $i$ — a dimension-pair index, running from $0$ to $d/2-1$; each $i$ picks out one sin/cos pair of dimensions.

It’s cleaner to define a frequency per pair:

$${\omega }_i=\frac{1}{10000^{2i/d}}=10000^{-2i/d}$$

so that the formula becomes just:

$$\text{PE}(m,2i)=\sin ({\omega }_{i}m),\text{PE}(m,2i+1)=\cos ({\omega }_{i}m)$$

The Intuition: Multiple Clocks

Different dimension pairs oscillate at different speeds:

• $i=0$: ${\omega }_0=1$ — fast, changes noticeably at every single position.
• $i=1$: ${\omega }_1=10000^{-2/d}$ — slower.
• $i=d/2-1$: ${\omega }_{d/2-1}=10000^{-1}=0.0001$ — extremely slow, barely moves across thousands of positions.

Numerical Walkthrough (d = 4)

With $d=4$, there are only two frequency pairs: $i=0$ and $i=1$.

For $i=0$: ${\omega }_0=10000^0=1$, so $\text{PE}(m,0)=\sin (m)$ and $\text{PE}(m,1)=\cos (m)$.

For $i=1$: ${\omega }_1=10000^{-2/4}=10000^{-0.5}=1/100$, so $\text{PE}(m,2)=\sin (m/100)$ and $\text{PE}(m,3)=\cos (m/100)$.

Dims 0–1 swing rapidly across just four positions. Dims 2–3 barely budge. Together, the four numbers form a unique fingerprint per position — exactly like the clock analogy above.

The Relative Position Property (Key Math)

This is the property that makes sinusoidal encoding genuinely elegant, not just clever bookkeeping.

Claim: for any fixed offset $k$, there’s a rotation matrix $M_k$ — depending only on $k$, never on $m$ — that maps position $m$’s encoding to position $m+k$’s encoding:

$$\left(\begin{array}{c}\sin ({\omega }_i(m+k))\\ \cos ({\omega }_i(m+k))\end{array}\right)=\left(\begin{array}{cc}\cos ({\omega }_{i}k)& \sin ({\omega }_{i}k)\\ -\sin ({\omega }_{i}k)& \cos ({\omega }_{i}k)\end{array}\right)\left(\begin{array}{c}\sin ({\omega }_{i}m)\\ \cos ({\omega }_{i}m)\end{array}\right)$$

where: $k$ is the offset between two positions; ${\omega }_i$ is the frequency for pair $i$ (as in 3.1); $m$ is the starting position.

Proof sketch, via the standard angle-addition identities:

$$\sin ({\omega }_{i}m+{\omega }_{i}k)=\sin ({\omega }_{i}m)\cos ({\omega }_{i}k)+\cos ({\omega }_{i}m)\sin ({\omega }_{i}k)$$ $$\cos ({\omega }_{i}m+{\omega }_{i}k)=\cos ({\omega }_{i}m)\cos ({\omega }_{i}k)-\sin ({\omega }_{i}m)\sin ({\omega }_{i}k)$$

These two identities are literally the matrix equation above, written out. Crucially, $M_k$’s entries depend only on $k$.

Why this matters in practice: the dot product between two positional encodings works out to a function of the distance between them, not their absolute positions:

$$⟨{\text{PE}}_m,{\text{PE}}_n⟩=\sum _{i=0}^{d/2-1}\left[\sin ({\omega }_{i}m)\sin ({\omega }_{i}n)+\cos ({\omega }_{i}m)\cos ({\omega }_{i}n)\right]=\sum _{i=0}^{d/2-1}\cos ({\omega }_i(m-n))$$

This is a function of $(m-n)$ only. So even though the encoding is added in absolute form, the attention mechanism — which works via dot products — can implicitly extract relative position information from it.

Why Not Simpler Encodings

Raw position numbers ($p_m=[m]$): the model would see huge numeric jumps as sequences get longer, with no periodic structure and no way to recover relative distance cleanly.

One-hot vectors ($p_1=[1,0,0,\dots ]$, $p_2=[0,1,0,\dots ]$): positions 2 and 3 look exactly as “different” from each other as positions 2 and 5000 — there’s no notion of smoothness or nearness at all.

Sinusoidal: multi-scale (many frequencies), periodic, unique per position, smooth between neighbors, and relative-distance-recoverable — all with zero learned parameters, and it generalizes to any sequence length.

3.4 — From Absolute to Relative Positional Encoding

The Problem with Absolute Encoding

Standard transformers add $\text{PE}(m)$ straight into the token embedding — this is absolute positional encoding, since each token’s stamp depends only on its own index $m$.

But look at what attention actually needs: when deciding how much token $m$ should attend to token $n$, what matters is the gap $m-n$ between them, not their individual absolute positions. Two tokens 3 apart at the start of a document should relate the same way as two tokens 3 apart near the end.

T5 Relative Bias

Idea: add a learned scalar bias directly onto the raw attention logit:

$$\text{score}(m,n)=\frac{q_m\cdot k_n}{\sqrt{d_k}}+b(m,n)$$

where: $b(m,n)$ is a learned bias term depending on the positions $m$ and $n$.

T5 makes $b(m,n)$ depend only on the distance $m-n$, and buckets that distance — many different distances share the same learned value:

$$b(m,n)={\beta }_{\text{bucket}(m-n)}$$

where: $\beta$ is a learned parameter, one per attention head, indexed by which bucket the distance $m-n$ falls into.

Why bucket? To generalize past training length. If training only ever saw distances up to 128, distances from 64–128 get grouped into one shared bucket; at inference, distances beyond training length still fall gracefully into that last bucket instead of hitting an undefined case.

Limitation: $\beta$ is a real learned parameter per head — it still has to be trained, and generalization to lengths far beyond training remains imperfect.

ALiBi: Attention with Linear Biases

Idea (Press et al., 2021): skip learning entirely — use a fixed, deterministic linear penalty based purely on distance:

$$\text{score}(m,n)=\frac{q_m\cdot k_n}{\sqrt{d_k}}+\mu \cdot (n-m)$$

where: $\mu <0$ is a fixed, head-specific slope, chosen ahead of time (not learned); since $n\le m$ for a past token $n$, the term $(n-m)$ is negative, so the bias is always negative — the farther back a token is, the harsher the penalty.

Each head is assigned a different slope (typically a geometric sequence like $\mu \in \{-1,-1/2,-1/4,\dots \}$) — some heads penalize distance sharply (stay very local), others gently (can still reach far).

Key properties:

• Zero learned parameters for position.
• Naturally biases attention toward nearby tokens by default.
• Extrapolates cleanly to sequences longer than training — the linear penalty just keeps extending, with no undefined behavior.

$$\text{bias}(m,n)=\mu \cdot (n-m),\mu <0,n\le m$$

Visually, the attention matrix ends up with a triangular pattern of steadily increasing penalty the further back you look.

3.5 — Rotary Positional Embedding (RoPE)

The Core Idea

RoPE (Su et al., 2021) takes yet another angle: instead of adding a position vector to the input, or adding a bias to the score, it rotates the Query and Key vectors themselves — by an angle proportional to their position — right inside the attention computation.

$$\text{Rotated}(q_m)\cdot \text{Rotated}(k_n)=f(m-n)$$

2D Rotation: The Building Block

In just 2 dimensions, rotating a vector by angle $\alpha$ uses the standard rotation matrix:

$$R(\alpha )=\left(\begin{array}{cc}\cos \alpha & -\sin \alpha \\ \sin \alpha & \cos \alpha \end{array}\right)$$

Rotate the Query by angle $m\theta$ and the Key by angle $n\theta$:

$${\tilde{q}}_m=R(m\theta )q_m,{\tilde{k}}_n=R(n\theta )k_n$$

where: $\theta$ is a fixed base angle; $m,n$ are the two token positions; ${\tilde{q}}_m,{\tilde{k}}_n$ are the rotated Query and Key.

Now take their dot product:

$${\tilde{q}}_m^T{\tilde{k}}_n=q_m^{T}R(m\theta {)}^{T}R(n\theta )k_n=q_m^{T}R((n-m)\theta )k_n$$

This uses two rotation-matrix facts: $R(\alpha {)}^T=R(-\alpha )$ (transposing a rotation reverses it), and $R(-m\theta )R(n\theta )=R((n-m)\theta )$ (composing two rotations adds their angles).

The result depends only on $(n-m)$ — the individual values of $m$ and $n$ disappear entirely, leaving pure relative distance baked directly into the dot product.

Extension to Many Dimensions

Real Query/Key vectors have $d_k$ dimensions, not 2. RoPE handles this by splitting the vector into $d_k/2$ independent 2D pairs, each rotating at its own frequency:

$${\theta }_i=10000^{-2i/d_k},i=0,1,\dots ,d_k/2-1$$

where: $i$ indexes which pair of dimensions we’re looking at; ${\theta }_i$ is that pair’s rotation frequency — exactly the same multi-frequency idea as sinusoidal PE in Part 3.

The full rotation is block-diagonal — each pair rotates independently of every other pair:

$$R_{\theta ,m}=\left(\begin{array}{cccc}{\text{block}}_1& 0& \cdots & 0\\ 0& {\text{block}}_2& \cdots & 0\\ ⋮& & \ddots & ⋮\\ 0& \cdots & 0& {\text{block}}_{d_k/2}\end{array}\right),{\text{block}}_i=\left(\begin{array}{cc}\cos (m{\theta }_i)& -\sin (m{\theta }_i)\\ \sin (m{\theta }_i)& \cos (m{\theta }_i)\end{array}\right)$$

The relative-distance property survives intact across all pairs:

$$q_m{k}_n^T=x_m{W}_q{R}_{\theta ,n-m}{W}_k^T{x}_n^T$$

where: $x_m,x_n$ are the raw token embeddings; $W_q,W_k$ are the Query/Key projection matrices; $R_{\theta ,n-m}$ is the block-diagonal rotation for the relative distance $n-m$. The final attention score is a function of relative distance only — nothing about absolute position leaks through.

Practical Implementation of RoPE

In real code, the block-diagonal matrix is never explicitly built (that would be wasteful). Instead, for a vector split into consecutive pairs $q=[q_1,q_2,q_3,q_4,\dots ]$, each pair is rotated with simple element-wise arithmetic:

$$\text{RoPE}(q,m)=\left(\begin{array}{c}{q}_1\cos (m{\theta }_0)-q_2\sin (m{\theta }_0)\\ q_1\sin (m{\theta }_0)+q_2\cos (m{\theta }_0)\\ q_3\cos (m{\theta }_1)-q_4\sin (m{\theta }_1)\\ q_3\sin (m{\theta }_1)+q_4\cos (m{\theta }_1)\\ ⋮\end{array}\right)$$

where: $q_1,q_2,\dots$ are the original vector’s individual entries, grouped into consecutive pairs, each pair rotated by its own frequency ${\theta }_i$ from “Extension to Many Dimensions” (Section 3.5).

This is just multiplications and additions — no matrix construction needed — which is exactly the kind of operation GPUs handle extremely efficiently.

The Long-Term Decay Property

RoPE has a bonus emergent behavior: the maximum possible attention score between two tokens naturally shrinks as their distance $|m-n|$ grows.

Why: each frequency pair contributes a term $\cos ({\omega }_i(m-n))$, and different pairs oscillate at different phases. For large distances, these many out-of-sync cosine terms partially cancel each other out (destructive interference), pulling the overall achievable score down.

Consequence: RoPE builds in a mild recency bias for free — nearby tokens tend to be easier to attend to strongly than very distant ones, which loosely matches how language actually works (nearby context is usually more relevant).

Why RoPE Is the Default Choice Today

RoPE checks every box: no extra parameters to learn, exact relative-distance encoding, natural long-range decay, and a clean, GPU-friendly implementation. It’s the default in LLaMA 1/2/3, Mistral, DeepSeek, Qwen, and nearly every modern open LLM.

3.6 — DeepSeek’s Decoupled RoPE and Partial RoPE

The Conflict Between MLA and RoPE

Here’s a subtle engineering snag: MLA caches only the compressed latent $c_{KV}$ (“Multi-Head Latent Attention (MLA)” (Section 3.1)) — it deliberately never materializes the full Key vectors. But standard RoPE needs to rotate the actual Key vector by an angle depending on its position, before the dot product happens:

$${\tilde{K}}_n^{(i)}=c_{KV,n}{W}_{\uparrow }^{K,(i)}{R}_{\theta ,n}$$

where: $c_{KV,n}$ is the cached latent for token $n$; $W_{\uparrow }^{K,(i)}$ is head $i$’s up-projection (from “Multi-Head Latent Attention (MLA)” (Section 3.1)); $R_{\theta ,n}$ is RoPE’s position-dependent rotation for position $n$.

The problem: $R_{\theta ,n}$ depends on $n$, but $c_{KV,n}$ is cached without that rotation baked in — because rotation must be applied fresh depending on how far away the current query token is. This wrecks the matrix absorption trick from “Multi-Head Latent Attention (MLA)” (Section 3.1): you can no longer pre-merge $W_{\uparrow }^{Q,(i)}$ and $W_{\uparrow }^{K,(i)}$ into one fixed matrix $A^{(i)}$ ahead of time, because a position-dependent rotation now sits in between them, and that rotation is different for every single cached token.

Decoupled RoPE: DeepSeek’s Solution

The fix: split each token’s representation into two separate streams that serve two separate jobs.

• Content stream: the compressed latent $c_{KV}$ from MLA, used for content-based matching — RoPE is never applied here, so the matrix-absorption speed-up (“Multi-Head Latent Attention (MLA)” (Section 3.1)) stays fully intact.
• Position stream: a small, separate, uncompressed pair of vectors — $k_n^R=x_n{W}^{KR}$ for Keys and $q_m^R=x_m{W}^{QR}$ for Queries — to which RoPE is applied.

where: $W^{KR},W^{QR}$ are dedicated projection matrices just for this small positional stream; $k_n^R,q_m^R$ are the resulting position-carrying Key and Query vectors.

The final attention score for head $i$ simply adds the two streams together:

$${\text{score}}_i(m,n)=\underset{\text{content — from MLA, no RoPE}}{\underbrace{{\tilde{q}}_m^{(i)}\cdot {\tilde{k}}_n^{(i)}}}+\underset{\text{position — from the RoPE stream}}{\underbrace{{\tilde{q}}_m^R\cdot {\tilde{k}}_n^R}}$$

The content path keeps all of MLA’s speed benefits untouched; the position path carries all the positional signal, cached separately (a small extra cost, but unavoidable and cheap compared to the savings from MLA itself).

Partial RoPE

Even within the small position stream, rotating every dimension is more compute than necessary. Partial RoPE rotates only the first $d_r$ dimensions, leaving the rest untouched:

$$q=\underset{\text{rotated (carries position)}}{\underbrace{[q_1,\dots ,q_{d_r}]}}\oplus \underset{\text{untouched (pure content)}}{\underbrace{[q_{d_r+1},\dots ,q_{d_k}]}}$$

where: $d_r$ is the number of leading dimensions that get RoPE applied; $d_k$ is the total dimension of the vector; $\oplus$ means “stitched back together into one vector.”

For example, with $d_k=128$, RoPE might rotate only dimensions 1–32 and leave 33–128 exactly as they were.

Why this is enough: position doesn’t need to permeate the entire vector to be recoverable — a compact “positional footprint” in a subset of dimensions carries all the signal the attention score needs.

Benefits: fewer sine/cosine computations per token, less memory traffic moving rotated values around, and the untouched dimensions stay purely content-based with zero positional interference.

3.7 — Sparse Attention for Long Sequences

The Quadratic Cost Problem

Standard attention computes a full $n\times n$ score matrix (“What the QK Transpose Matrix Looks Like” (Section 3.1)), which costs $O(n^2)$ in sequence length $n$. At $n=1,000,000$ tokens, that matrix has $10^{12}$ entries — computationally infeasible to form in full.

Sliding Window Attention (Longformer / SWA)

Idea: restrict each token to only attend within a fixed local window of size $w$:

$$\text{token }m\text{ attends to tokens }m-w,\dots ,m+w$$

where: $w$ is the window radius — how many tokens on each side a token is allowed to see.

This produces a banded attention matrix — only a diagonal strip is ever filled in, cutting cost to:

$$\text{Complexity: }O(n\cdot w)\text{ instead of }O(n^2)$$

The receptive field trick (used in Mistral): although any single layer only sees $w$ neighbors, stacking $L$ layers lets information hop from neighbor to neighbor to neighbor, so the effective receptive field after $L$ layers grows to roughly $L\times w$ — like gossip spreading through a social network, one hop at a time.

Limitation: a genuinely important token sitting 100,000 positions away will simply never fall inside any window, no matter how many layers you stack.

Longformer: Local and Global Tokens

Longformer’s hybrid fix: keep sliding-window attention for most tokens, but designate a handful of global tokens (e.g., a [CLS] token) that attend to — and are attended to by — everyone, with full unrestricted attention.

$$\text{Global token}\leftrightarrow \text{all tokens}\text{(full attention)}$$ $$\text{Local token}\leftrightarrow \text{window neighbors only}\text{(sliding window)}$$

This gives cheap local context for most tokens, plus a small number of “broadcast hubs” that let global information flow anywhere — without ever paying the full $O(n^2)$ cost.

DeepSeek Sparse Attention (DSA)

The gap Longformer can’t close: what if the one truly relevant token sits at position 700,000, while the current token is at position 1,000,000, and there’s no global-token mechanism watching that specific spot? A fixed window and a handful of hand-picked global tokens simply miss it.

DSA’s answer: don’t fix the window in advance — dynamically pick out the most relevant tokens for each query, wherever they happen to be.

The Lightning Indexer

A cheap, lightweight, MQA-style attention pass whose only job is to estimate relevance before the real, expensive attention runs:

$$\text{index score}(m,n)=\text{ReLU}\left({\tilde{q}}_m\cdot {\tilde{k}}_n^{\text{index}}\right)$$

where:

• ${\tilde{q}}_m$ — a lightweight query projection for the current token, separate from the main attention’s Query.
• ${\tilde{k}}_n^{\text{index}}$ — a single shared Key (MQA-style — one Key for all heads, so this pass is cheap) representing token $n$ for indexing purposes only.
• $\text{ReLU}(\cdot )$ — zeroes out any negative score, so clearly-irrelevant tokens get exactly zero and are dropped from consideration rather than merely down-weighted.

Quantization + Hadamard Rotation

To make the indexer even cheaper, its vectors are stored in INT8 (8-bit integers) instead of FP16 — roughly a $4\times$ compute reduction.

The catch: naive quantization struggles with vectors that have a huge dynamic range, e.g. $[1000,0.01,0.001,500]$ — the tiny values get crushed to zero once everything is rescaled to fit INT8’s limited range.

The fix: apply a Hadamard transform before quantizing. A Hadamard matrix contains only $+1$s and $-1$s:

$$H_2=\left(\begin{array}{cc}1& 1\\ 1& -1\end{array}\right),H_4=\frac{1}{2}\left(\begin{array}{cccc}1& 1& 1& 1\\ 1& -1& 1& -1\\ 1& 1& -1& -1\\ 1& -1& -1& 1\end{array}\right)$$

where: $H_2,H_4$ are Hadamard matrices of size 2 and 4 respectively — used to “mix” a vector’s entries before quantizing it.

Key property: $H^{T}H=I$ — a Hadamard matrix is orthogonal, meaning it acts purely as a rotation (it doesn’t stretch or shrink vectors, just reorients them).

Effect on the vector: a spiky vector like $[8,0,0,0]$ becomes a flat $[4,4,4,4]$ after the Hadamard transform — the same total “energy,” but now spread evenly across all dimensions, so no single value dominates and quantization preserves everything accurately.

Why rotating first doesn’t change the final answer:

$$q^{T}k=(Hq{)}^T(Hk)=q^T{H}^{T}Hk=q^{T}Ik=q^{T}k$$

where: $I$ is the identity matrix. Because $H$ is orthogonal, rotating both the Query and Key by the same $H$ leaves their dot product completely unchanged — so this is a pure numerical-stability trick with zero effect on the actual attention math.

The Full DSA Pipeline

$$\underset{\text{all tokens}}{\underbrace{q_m,\{k_n\}}}\stackrel{\text{Hadamard}}{\to }\underset{\text{rotated}}{\underbrace{H{q}_m,\{H{k}_n\}}}\stackrel{\text{INT8 quantize}}{\to }\text{approx. scores}\stackrel{\text{top-K select}}{\to }\text{chosen tokens}\stackrel{\text{full MLA attention}}{\to }\text{output}$$

Net result: roughly 2–3× faster processing on long sequences, 30–40% memory reduction, and negligible loss in output quality — because the cheap Lightning Indexer only decides which tokens matter; the actual attention computation on the selected tokens still runs at full precision and quality.

3.8 — Layer Normalization

What It Is and Where It Lives

Layer Normalization sits at the “Add & Norm” blocks of the transformer — once right after multi-head attention, and once right after the feed-forward network, in every layer.

The Formula

For a single token’s activation vector $x\in {\mathbb{R}}^d$:

$$\hat{x}=\frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}},\text{LN}(x)=\gamma \hat{x}+\beta$$

where:

$$\mu =\frac{1}{d}\sum _{i=1}^d{x}_i,{\sigma }^2=\frac{1}{d}\sum _{i=1}^d(x_i-\mu {)}^2$$

• $\mu$ — the mean of $x$’s entries (computed across the feature dimension, for this one token).
• ${\sigma }^2$ — the variance of $x$’s entries.
• $\hat{x}$ — $x$ after being re-centered (subtract $\mu$) and re-scaled (divide by standard deviation) — now has mean 0 and unit variance.
• $\gamma ,\beta \in {\mathbb{R}}^d$ — learned per-dimension scale and shift parameters, letting the network re-introduce whatever scale/offset it actually needs after normalizing.
• $ϵ$ — a tiny constant (around $10^{-6}$) added purely to avoid dividing by zero.

Contrast with Batch Normalization: BatchNorm computes $\mu ,{\sigma }^2$ across the batch dimension (across different examples); LayerNorm computes them across the feature dimension, independently for each token. This makes LayerNorm indifferent to batch size and sequence length — a good fit for NLP, where sequences are variable-length and batches are sometimes small.

Post-LN vs Pre-LN

Post-LN (the original Transformer design): normalize after adding the residual:

$$x_{l+1}=\text{LN}(x_l+\text{SubLayer}(x_l))$$

Pre-LN (today’s default): normalize before the sublayer even runs:

$$x_{l+1}=x_l+\text{SubLayer}(\text{LN}(x_l))$$

where: $x_l$ is the residual stream entering layer $l$; $\text{SubLayer}(\cdot )$ is either the attention block or the feed-forward block.

Pre-LN trains more stably, especially at large scale, because the raw residual stream $x_l$ flows straight through the network unnormalized, keeping gradients well-behaved. Post-LN, by contrast, can suffer from training instability in very deep networks.

RMSNorm: The Modern Variant

RMSNorm (Zhang et al., 2019) simplifies LayerNorm by dropping the mean-centering step entirely — it normalizes only by the root-mean-square magnitude:

$$\text{RMS}(x)=\sqrt{\frac{1}{d}\sum _{i=1}^d{x}_i^2},\text{RMSNorm}(x)=\gamma \cdot \frac{x}{\text{RMS}(x)}$$

where: $\text{RMS}(x)$ measures the overall magnitude of $x$ without first subtracting a mean; $\gamma$ is still a learned per-dimension scale, but there’s no learned shift $\beta$ this time.

No mean subtraction, no bias term — fewer operations, fewer parameters, faster in practice. Used in LLaMA, Mistral, and most modern LLMs, almost always paired with Pre-LN.

3.9 — Synthesis: How It All Fits Together

The Modern Transformer Layer

A single transformer layer in a modern LLM (LLaMA-style) chains these steps:

1. RMSNorm (Pre-LN)
2. Multi-Head Attention with RoPE applied to Q and K
3. Residual add
4. RMSNorm (Pre-LN)
5. Feed-Forward Network
6. Residual add

Step 2’s attention block typically uses GQA (or MLA, in DeepSeek’s models) for KV cache efficiency — the two efficiency ideas (positional encoding choice, and KV-cache-saving attention variant) are independent and simply stacked together.

The Position Information Flow

Token embedding x_m
         |
         + (pure word meaning — no position yet)
         |
    [Attention layer]
         |
    Apply RoPE to Q and K:
         q_m' = R(m*theta) q_m
         k_n' = R(n*theta) k_n
         |
    Dot product:
         q_m' . k_n' = f(m - n)   <- only relative distance!
         |
    Attention weights + Value aggregation
         |
    Output (now position-aware)

Position is injected inside the attention layer, not at the input. The word embedding itself stays purely about meaning; geometry (position) is handled entirely by the attention mechanism.

The KV Efficiency Hierarchy

where: $n$ is sequence length; $h$ is head count; $d_k$ is per-head K/V dimension; $G$ is the number of GQA groups; $d_c$ is MLA’s latent dimension.

The Positional Encoding Hierarchy

3.10 — Quick Reference: Key Equations

Attention $$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

Sinusoidal PE $$\text{PE}(m,2i)=\sin \left(\frac{m}{10000^{2i/d}}\right),\text{PE}(m,2i+1)=\cos \left(\frac{m}{10000^{2i/d}}\right)$$

ALiBi $$\text{score}(m,n)=\frac{q_m\cdot k_n}{\sqrt{d_k}}+\mu (n-m),\mu <0$$

RoPE (2D block for pair i) $$R(m{\theta }_i)=\left(\begin{array}{cc}\cos (m{\theta }_i)& -\sin (m{\theta }_i)\\ \sin (m{\theta }_i)& \cos (m{\theta }_i)\end{array}\right),{\theta }_i=10000^{-2i/d_k}$$ $${\tilde{q}}_m^T{\tilde{k}}_n=q_m^T{R}_{\theta ,(n-m)}{k}_n\text{(relative distance only)}$$

MLA Compression $$c_{KV}=x\cdot W_{\downarrow }^{KV}\text{(cache this)},K^{(i)}=c_{KV}\cdot W_{\uparrow }^{K,(i)},V^{(i)}=c_{KV}\cdot W_{\uparrow }^{V,(i)}\text{(reconstruct on demand)}$$

Layer Normalization $$\text{LN}(x)=\gamma \cdot \frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}+\beta ,\text{RMSNorm}(x)=\gamma \cdot \frac{x}{\text{RMS}(x)}$$