The Transformer Architecture

The transformer (Vaswani et al., 2017) replaced recurrence and convolution in sequence modeling with two ingredients: self-attention and position-wise feed-forward networks.

A transformer block has two sub-layers, each wrapped in a residual connection and layer normalization:

1. Multi-head self-attention — mixes information across token positions.
2. Position-wise MLP — applies the same two-layer feed-forward network to each position independently.

These blocks are stacked N times. The full network has three pieces:

• Embeddings turn input token IDs into vectors and add positional information (sinusoidal or learned).
• Encoder / decoder stack of N blocks each. Decoder-only LLMs drop the encoder and use masked self-attention so positions cannot see the future during training.
• Unembedding projects the final vector at each position back to a distribution over the vocabulary.

Why it works:

• Parallelism: unlike RNNs, every position is processed in parallel during training, which is why transformers scale to trillion-parameter models.
• Long context: attention lets any token directly read any other token up to the context length, with O(1) hops.
• Compositionality: stacking many blocks lets the network build hierarchical features — layers learn combinations of features; there is no fixed syntax-to-reasoning partition.