ENGINEERING BLOG / CHAPTER 02 OF 10

From RNNs to the Transformer

Saravan Kumar M KUpdated 2026-10-26~14 min read7 sections

Chapter 1 ended with a limitation: Word2vec embeddings are static. This chapter revisits the recurrent models (RNNs, LSTMs) that dominated before 2017, shows why their sequential nature became a bottleneck, introduces the attention mechanism, and then assembles the full encoder–decoder Transformer, finishing with an end-to-end translation example. By the end you should understand why the Transformer was such a leap and have a mental model of every block inside it.

2.1 — Recurrent Neural Networks

To model a sequence, the pre-Transformer answer was recurrence: process tokens one at a time, carrying a hidden state that summarizes everything seen so far. RNNs were first introduced in the 80s; they are a class of neural networks where connections form a temporal sequence.

Definition. A Recurrent Neural Network (RNN) processes a sequence token by token, updating a hidden state $h_{t}$ from the current input $x_{t}$ and the previous state $h_{t - 1}$ , so that $h_{t}$ encodes the history of the sequence.

Intuition. Think of reading a sentence while keeping a running mental summary. At each word you update that summary. The same weights are reused at every step (parameter sharing across time), which is what lets an RNN handle variable-length input.

$h_{t} = tanh (W_{hh} h_{t - 1} + W_{x h} x_{t} + b_{h})$

The recurrent update: the new state mixes the previous state and the current input.

An RNN unrolled over time; the same cell (shared weights) is applied at each step.

On our running example the RNN reads A, then cute, then teddy bear, then is, then reading, each time passing its hidden state to the next step.

An unrolled recurrent network. Image: Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.

What do we do with the hidden states?

The same recurrent network can be wired for the three NLP shapes of Chapter 1:

RNNs can be wired for classification (last state), “multi”-classification (every state) and generation (encoder then decoder).

Classification (e.g. sentiment): the last hidden state is fed to a classifier that outputs the opinion.
“Multi”-classification (e.g. tagging): every hidden state is fed to a classifier that outputs one tag per token.
Generation (e.g. translation): a source text is read by one RNN and an output text is produced by another.

Key insight. RNNs are inherently sequential: step $t$ cannot be computed until step $t - 1$ is done. This kills parallelism on modern GPUs and makes long sequences slow to train — a key motivation for the Transformer.

2.2 — The vanishing-gradient problem and LSTMs

Definition. A Long Short-Term Memory (LSTM) network is an RNN variant that adds a separate cell state and three gates (input, forget, output) to control what information is kept, discarded, or exposed at each step.

LSTMs were introduced in the 1997 paper “Long short-term memory”; they use a more structured approach in the cell’s hidden state.

An LSTM cell. Image: Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.

Intuition. Vanilla RNNs struggle to remember information across many steps because gradients shrink (vanish) as they are back-propagated through time. The LSTM’s gated cell state acts like a conveyor belt with controllable valves: information can flow far down the sequence almost untouched, letting the network learn long-range dependencies.

The vanishing-gradient problem: the learning signal shrinks each time it passes back through a step.

Example. In the sentence ‘The keys to the cabinet … are on the table’, the verb ‘are’ must agree with ‘keys’, not ‘cabinet’. An LSTM can carry the plural information across the intervening words far better than a plain RNN.

Key insight. LSTMs mitigated but never eliminated the sequential bottleneck and the difficulty of very long-range dependencies. The field needed a fundamentally parallel way to mix information across a sequence.

Summary of main methods so far (non-exhaustive list)

Method	Pros	Cons
Word2vec (e.g. CBOW, Skip-gram)	Very simple, yet powerful; intuitive embeddings	Word order does not count; embeddings not context aware
Recurrent Neural Networks (e.g. traditional RNN, LSTM)	Word order matters; state-of-the-art results	Vanishing gradient problem; slow computations

2.3 — History of attention

Attention was introduced in 2014 (“Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al.).
Translation tasks had a real issue with long-term dependencies.
Seq2seq was unable to “remember” what the input sentence was saying.

A seq2seq model squeezes the whole source sentence into one vector; the decoder must write the translation from that single summary.

Picture an encoder that has read “A cute teddy bear is reading” and a decoder that has produced “Un ours en peluche mignon” and now has to say the next word — “lit”. All it has is the single summary vector passed from the encoder, which has lost much of what the input sentence was saying. Attention was introduced as an add-on to RNN translation models, letting the decoder ‘look back’ at all encoder states instead of relying on a single fixed summary vector.

With attention, each decoder step looks back at all encoder states and takes a weighted blend of them.

Seq2seq with attention. Image: Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.

The Transformer took this idea and made it the only mechanism.

2.4 — The attention mechanism

Definition. Attention computes an output for each position as a weighted average of value vectors, where the weights (how much to attend) come from the compatibility between a query vector and the corresponding key vectors.

Intuition. Think of a soft dictionary lookup. The query asks ‘what am I looking for?’; each key advertises ‘what do I contain?’; the dot product query·key scores the match; a softmax turns scores into weights; the output is the weighted blend of values. Unlike a hard lookup, you retrieve a mixture of all entries, weighted by relevance — and crucially every position can be computed in parallel.

The concept of Query, Key, Value: compare the query with every key, turn the scores into weights, blend the values.

Given a query $q$ , we want to know which key $k$ the query should pay “attention” to with respect to the associated value $v$ . For the query of “teddy bear”, the scores $q \cdot k_{a}, q \cdot k_{c u t e}, q \cdot k_{t e dd y b e a r}, q \cdot k_{i s}, q \cdot k_{re a d in g}, q \cdot k_{.}$ are turned into weights that multiply $v_{a}, v_{c u t e}, v_{t e dd y b e a r}, \dots$ and are summed.

Attention can be computed efficiently with matrices $Q$ , $K$ , $V$ that contain the queries $q$ , keys $k$ and values $v$ respectively, along with the dimension $d_{k}$ of the keys. Scaled dot-product attention, the heart of the Transformer:

$A tt e n t i o n (Q, K, V) = so f t ma x (\frac{Q K ^{T}}{d _{k}}) V$

Queries Q, keys K, values V; the $1/ d_{k}$ scaling keeps the softmax in a stable regime.

Query–Key scoring, scaling, softmax, then a value-weighted sum.

Example. Resolving ‘it’ in ‘The animal didn’t cross the street because it was too tired’: the query from ‘it’ matches the key of ‘animal’ more strongly than ‘street’, so attention pulls ‘animal’‘s value into the representation of ’it’. Self-attention is how the model performs coreference for free.

An attention-weight matrix: the row for “it” puts most of its weight on “animal”.

Key insight. The $d_{k}$ scaling matters: without it, large dot products push the softmax into saturated regions where gradients vanish. Dividing by $d_{k}$ keeps variance ≈ 1 and training stable.

Without scaling, large dot products make softmax a one-hot spike where gradients vanish; with scaling the weights stay smooth.

Multi-head attention

Definition. Multi-head attention runs several attention operations (‘heads’) in parallel on different learned projections of Q, K, V, then concatenates and linearly recombines their outputs.

Intuition. One attention pattern can only capture one type of relationship at a time. Multiple heads let the model attend to different things simultaneously — one head may track syntax, another long-range coreference, another local phrasing — much like using several convolutional filters in a CNN (multiple filters of a convolutional layer in computer vision).

$M u lt i He a d (Q, K, V) = C o n c a t (h e a d_{1}, \dots, h e a d_{h}) W^{O}$

Heads are computed independently then merged by the output projection $W^{O}$ . The layer is composed of $h$ attention heads, each with its own matrices $W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}$ that project the input to obtain queries, keys and values.

Multi-head attention: h heads attend in parallel, their outputs are concatenated and projected with an output matrix.

Multi-head attention. Image: Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.

Key insight. Each head works in a lower-dimensional subspace ( $d_{m o d e l} / h$ ), so multi-head attention adds expressive power at roughly the same compute as a single full-width head.

2.5 — The Transformer architecture

Introduced in ‘Attention Is All You Need’ (Vaswani et al., 2017), the Transformer stacks attention and feed-forward layers with residual connections and layer normalization, dispensing with recurrence entirely. It relies on the self-attention mechanism and gave state-of-the-art results on machine translation tasks. The original design is an encoder–decoder built for translation.

Definition. The Transformer is a sequence-to-sequence architecture made of an encoder stack (which builds contextual representations of the input) and a decoder stack (which generates the output autoregressively), connected by cross-attention, using only attention and feed-forward layers — no recurrence or convolution.

Intuition. Read the diagram bottom-to-top. Inputs are embedded and given positional information; the encoder refines them with self-attention + feed-forward blocks; the decoder generates one token at a time, attending both to what it has produced so far (masked self-attention) and to the encoder’s output (cross-attention); a final linear + softmax produces a probability over the vocabulary.

The original encoder–decoder Transformer. Each block is repeated N times. Image: Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.

The Transformer at a glance: input with positional encoding, encoder stack, output shifted right, decoder stack, linear + softmax.

The main building blocks are:

Attention layer (MHA): self-attention (encoder–encoder, decoder–decoder) and the encoder–decoder attention layer
Feed Forward Neural Network (FFNN)
Positional Encoding (PE)

Key insight. Three ingredients make every block trainable at depth: (1) residual connections (Add) give gradients a shortcut, (2) layer normalization stabilizes activations, and (3) the position-wise feed-forward network adds non-linear capacity. Attention mixes information across positions; the FFN processes each position independently.

Input

Text is “tokenized”.
Learned embeddings for tokens.

Parameters: $V$ is the vocabulary size and $d_{m o d e l}$ is the number of embedding dimensions.

Positional encoding — the ‘trick’

Definition. Positional encoding injects information about token order into the otherwise order-agnostic attention mechanism, typically by adding sinusoidal vectors to the input embeddings.

Intuition. Attention is permutation-invariant: shuffle the input tokens and the set of outputs is just shuffled too — the model has no inherent sense of order. Since word order carries meaning (‘dog bites man’ ≠ ‘man bites dog’), we must add position information explicitly. The original paper uses sine and cosine waves of different frequencies so that each position gets a unique, smooth fingerprint.

$P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d}})$

Even dimensions use sine; odd dimensions use cosine. Different $i$ give different wavelengths.

The idea is to add position information to the inputs. It can be either learned or hardcoded, and the goal is to let the model understand the relative position of the inputs: the dot product of two position vectors is at its maximum for equal positions and decreases as positions move apart.

Sinusoidal positional encodings: each row is a position, each column a dimension.

Key insight. Because attention has no recurrence or convolution, position must be supplied as data. Chapter 3 revisits this with relative and rotary (RoPE) schemes that have since become standard.

Encoder

Encoder–encoder attention / self-attention
Feed Forward Neural Network
Normalization layer

Parameters: $N$ (layers stacked), $h$ (number of attention heads), $d_{FF}$ , $d_{k ey}$ , $d_{v a l u e}$ (sub-layer dimensions) and $d_{m o d e l}$ (embedding dimensions).

Output “shifted right”

Learned embeddings for output tokens.
In practice, the output starts with [BOS] during translation.

Parameters: $V$ (vocabulary size) and $d_{m o d e l}$ (embedding dimensions).

Masked self-attention and ‘shifted right’ output

Definition. Masked (causal) self-attention prevents each decoder position from attending to future positions, so that predicting token $t$ can only use tokens $1 \dots t - 1$ .

Intuition. During training we feed the whole target sentence at once for efficiency, but the model must not ‘cheat’ by peeking at the answer. A causal mask sets the attention scores to future tokens to $- \infty$ before the softmax, so their weights become zero. The decoder input is the target sequence ‘shifted right’ by one (prefixed with a start token) so that at each step it predicts the next token.

A causal mask: each position can only attend to itself and earlier positions.

Key insight. This causal masking is exactly what later turns the decoder into a standalone language model — the backbone of GPT-style LLMs in Chapter 5.

Decoder

Decoder–decoder attention / self-attention
Encoder–decoder attention
Feed Forward Neural Network
Normalization layer

Parameters: $N$ , $h$ , $d_{FF}$ , $d_{k ey}$ , $d_{v a l u e}$ and $d_{m o d e l}$ , as in the encoder.

Output

Linear projection.
A classification problem that outputs the probability of belonging to a class, where class = word.

Parameters: $V$ (vocabulary size) and $d_{m o d e l}$ (embedding dimensions).

Computational tricks

Multi-head attention. Idea: run multiple self-attention layers in parallel. Benefit: it enables the model to capture different attention features in parallel. Comparison: multiple filters of a convolutional layer in computer vision.

Label smoothing. Idea: a 2015 vision paper showed that overconfidence is bad, so introduce noise in the true labels (instead of a probability of exactly 1 for the right word and 0 for all others, soften the target a little). Benefits: it is a general technique that prevents overfitting, and it improves accuracy and the BLEU score.

2.6 — End-to-end example: machine translation

Intuition. Putting it together for English→French: the source sentence is tokenized, embedded, and given positional encodings; the encoder produces a contextual representation of every source token. The decoder then generates the French sentence token by token — at each step it attends to its own previous outputs (masked self-attention) and to the encoder representations (cross-attention), then emits a probability distribution over the vocabulary and picks the next token. This repeats until an end-of-sequence token is produced.

Let us stitch all the pieces together with our example, “A cute teddy bear is reading.”

The complete journey of one sentence through the Transformer.

Step 2 — Embed. Each token gets an embedding.

Step 3 — Add position. A position embedding is added to each embedding, giving a position-aware embedding per token. Stacked together, these form the position-aware embeddings matrix.

Step 4 — Encode. The matrix goes into the encoder. Inside the self-attention layer, three learned matrices $W_{q}$ , $W_{k}$ , $W_{v}$ turn the matrix into $Q$ , $K$ and $V$ . Each token’s query is compared with every token’s key, the resulting weights are applied to $V$ , and the heads are recombined with $W_{o}$ . The result goes through the feed-forward network. After the encoder stack, each token has a context-aware encoded embedding.

Step 5 — Start decoding. The decoder receives the encoded embeddings and the first output token [BOS]. It passes through the self-attention layer (masked), the encoder–decoder attention layer and the feed-forward network. A softmax layer then produces a probability for every word in the vocabulary — a vector such as [0.001, 0.0003, …, 0.4, …, 0.002]. The most probable word is Un.

Step 6 — Repeat. Un is fed back in: the decoder now reads [BOS] Un and produces ours; then en, peluche, mignon, lit and finally [EOS]. The output is “Un ours en peluche mignon lit.”

Example. Translating ‘The cat sat’ → ‘Le chat s’est assis’: when generating ‘chat’, cross-attention focuses on ‘cat’; when generating the verb, it blends ‘sat’ with the already-produced French subject to get agreement right. Every output token is one full forward pass through the decoder.

Key insight. Training uses ‘teacher forcing’ (feed the true previous tokens) so all decoder positions are computed in parallel; inference is autoregressive and therefore sequential. This train/inference asymmetry recurs throughout this blog (see KV caching in Chapter 6).

2.7 — Chapter summary

NLP tasks reduce to three shapes: sequence→label (sentiment), sequence→per-token-label (NER), sequence→sequence (translation).
Tokenization converts text to integer IDs; subword schemes (BPE, WordPiece) balance vocabulary size against out-of-vocabulary handling.
Word embeddings (Word2vec) place semantically similar words near each other but are static / context-independent.
RNNs process sequences with a recurrent hidden state but are inherently sequential and suffer vanishing gradients.
LSTMs add gated cell states to capture longer-range dependencies, yet remain sequential.
Attention computes a query–key–value weighted average, enabling parallel, content-based mixing across all positions.
Scaled dot-product attention divides scores by $d_{k}$ for stable gradients; multi-head attention captures several relations at once.
The Transformer is an encoder–decoder built only from attention + feed-forward blocks, with residuals and layer norm.
Positional encoding injects word order into the permutation-invariant attention.
Masked self-attention + ‘shifted-right’ inputs let the decoder generate autoregressively without peeking at the future.