The Transformer replaced recurrence with attention. Instead of processing tokens one at a time, it allows every position to directly examine other positions and decide which ones matter for its current representation.
The Core Operation
Each token representation is projected into a query, key and value. The query asks what the token is looking for, the keys describe what other tokens offer, and the values contain the information that can be combined.
Why Divide by the Square Root?
As the dimensionality of the keys grows, dot products can become large. Large scores push softmax toward extremely sharp distributions, which can make optimization unstable. Scaling the scores helps keep them in a more manageable range.
Multi-Head Attention
Instead of performing one attention operation, the model performs several in parallel. Different heads may learn different relationships: local syntax, long-range dependencies, entity references or other useful patterns.
Positional Encoding
Because attention alone does not preserve sequence order, the model needs positional information. In the original paper, sinusoidal encodings were added to the token embeddings. Modern systems often use learned positions or rotary position embeddings instead.
| Component | Purpose |
|---|---|
| Token embedding | Represents token identity |
| Position signal | Represents order |
| Attention | Combines contextual information |
| Feed-forward block | Transforms each position independently |
Residual Connections and Normalization
Residual paths allow information and gradients to travel through deep networks more easily. Normalization stabilizes activations. The exact placement of normalization has changed in many later architectures, but the basic role remains critical.
My Implementation Checklist
- Token and position embeddings
- Causal attention mask
- Multi-head attention module
- Feed-forward network
- Residual connections
- Layer normalization
- Final language-model head