The idea: every token takes a weighted mix of the others
Self-attention lets each token in a sentence look at the other tokens and pull in what it
needs. For every pair it computes a score, turns each row of scores into weights that add up to 1, and
replaces the token by the weighted sum of the others' values. The whole layer is one formula:
softmax(Q·Kᵀ / √d_k + mask) · V.
The canvas does this for the sentence my cat ate its fish with vectors of only 4 numbers, so
every multiplication fits on screen. How a Large Language Model Works shows where this
layer sits inside a whole GPT; this page opens up the layer itself.
(Demo 1: one query step by step)
Queries, keys and values
Each token starts as an embedding, a row of the matrix X. To make the numbers
readable, the 4 columns here have names: animal, food, action and ref (a word such as "its" that refers back).
cat is [2 0 0 0], ate is [0 0 2 0].
Three weight matrices turn every embedding into three new vectors:
- Query
Q = X·W_Q: what this token is looking for.itsgets[2 0 1 0], mostly "where is the animal?". - Key
K = X·W_K: what this token offers to others.catoffers[2 0 0 0], "I am an animal". - Value
V = X·W_V: what this token hands over when someone attends to it.
In a trained model all three matrices are learned. Here they are small hand-picked integers, and
W_V is the identity, so a value row is just the token's features.
Scores: Q·Kᵀ
The score of query i and key j is their dot product
q_i·k_j: multiply the matching numbers and add them. It is large when the key has what the query
asks for. All scores at once are the matrix product S = Q·Kᵀ, an n × n matrix: row = the token
that looks, column = the token being looked at. For its the row is [0 4 2 2 2]:
the highest score is for cat. (Demo 2: full attention matrix)
Why divide by √d_k
A dot product adds up d_k products, so with longer vectors the scores get larger: for random
vectors their spread grows like √d_k. Softmax of large numbers is almost one-hot. One key gets
nearly all the weight, the others get almost none, and the gradients through softmax become tiny, so the
model learns slowly. Dividing by √d_k keeps the scores in the same range for any width. That is
the "scaled" in scaled dot-product attention.
Here d_k = 4, so the division is only by 2 and the effect is mild: the weight on
cat goes from 0.53 to 0.78. Real heads have d_k = 64 or 128.
(Demo 4: scaling vs no scaling)
The causal mask
A model that writes text one token at a time (GPT) must not use words that come after the current one,
because when it generates they do not exist yet. The causal mask adds −∞ to every score
above the diagonal (key after query). After softmax e^−∞ = 0, so those tokens get weight 0 and
the weight matrix is lower-triangular. my, the first token, can only attend to itself.
Without the mask every token sees every token, as in an encoder such as BERT. Then cat puts
0.65 of its weight on the later verb ate. With the mask it can only split its weight between
my and itself. (Demo 3: causal mask off, then on)
Softmax and the weighted sum of V
Softmax turns each row of scores into weights: w_j = e^z_j / Σ e^z. Every
weight is between 0 and 1 and a row adds up to 1. The output of token i is
out_i = Σ_j w_ij · v_j: each value row is scaled by its weight and the rows are added. For
its: 0.07·v_my + 0.53·v_cat + 0.20·v_ate + 0.20·v_its = [1.27 0 0.39 0.27]. The
high "animal" number shows that the meaning of cat has flowed into its.
In a full transformer block this output is added back to the token's vector (the residual connection) and goes on to the MLP and the next layer.
Multi-head attention
One softmax row can only express one kind of looking. Multi-head attention runs several
smaller attentions side by side. With d_model = 4 and 2 heads, head 1 uses columns 1–2 of Q, K and
V and head 2 uses columns 3–4, each with d_k = 2. The head outputs are
concatenated back to 4 numbers and multiplied by an output matrix W_O.
On the canvas, head 1 finds the animal (its → cat 0.73) and head 2 finds the
action (its → ate 0.58). The single 4-wide head mixes both questions into one score,
so it cannot say "the animal and also the verb" as cleanly. (Demo 5: two heads, then concatenate)
This page vs a real model
| Parameter | This page | GPT-2 small |
|---|---|---|
| tokens (n) | 5 | up to 1,024 |
d_model | 4 | 768 |
heads × d_k | 1 × 4 or 2 × 2 | 12 × 64 |
| attention layers | 1 | 12 |
| weights | hand-picked integers | learned |
The score matrix is n × n, so attention costs grow with the square of the context length. That is why long contexts are expensive and why generation keeps a KV cache (see LLM). See also Neural Network Forward Pass for softmax and weighted sums in an ordinary network, RNN and LSTM for the older way of reading a sequence, and BPE Tokenizer for where the tokens come from.
See also the dot product: attention scores are dot products of query and key vectors.
See also LoRA and Quantization: LoRA adapters are most often added to W_Q and W_V.
See also CNN vs Transformer: the same attention on image patches (a vision transformer), side by side with a convolutional network.
See The KV Cache for how the cached K and V rows grow and are stored.
See also Mixture of Experts: the router's softmax over experts works like one row of attention weights, but only the top-k experts run.
See also Word2Vec: how static word embeddings are learned from neighbouring words before attention makes them depend on context.
What the page leaves out
Position information (learned position embeddings or RoPE): without it attention does not know word order.
The output matrix W_O is described but not applied. Also left out: training the weights by
backpropagation, cross-attention between an encoder and a decoder, padding masks for batches, the KV cache,
and fast kernels such as FlashAttention that compute the same result without storing the n × n matrix.