Transformer architecture is the reusable pattern behind modern foundation models: embed a sequence of tokens, patches, or modality features; run stacked attention and feed-forward blocks; then adapt the attention mask, objective, and output interface. The same substrate supports pretraining and transfer learning, scaling, multimodal transfer, and open model families.
Self-attention and encoder-decoder design
The original Transformer made attention, rather than recurrence or convolution, the sequence modeling primitive. Its encoder-decoder stack uses multi-head scaled dot-product self-attention plus position-wise feed-forward layers, residual connections, layer norm, and positional encodings; the decoder adds masked self-attention and encoder-decoder cross-attention so generation can condition on prior target tokens and encoded source tokens. 1 The key architectural tradeoff is parallelism and short dependency paths at the price of quadratic self-attention over sequence length, which later work either scaled directly or optimized in efficient attention and training systems. 1
Decoder-only GPT lineage
GPT-1 selected the Transformer decoder as a reusable language-model backbone: causal self-attention, learned positions, BPE tokenization, next-token pretraining, then supervised fine-tuning by serializing each downstream task into a text sequence with minimal task-specific heads. 2 GPT-2 kept that causal decoder interface but scaled it across WebText and model sizes, making natural-language context act as task specification; its zero-shot behavior improved with scale, while the paper still framed left-to-right conditioning as less directly suited to some bidirectional understanding tasks than BERT-style encoders. 3 GPT-4 remains described at this level as a Transformer-style, next-token-pretrained model, with post-training alignment and multimodal input support layered over a scaled GPT lineage; the report withholds architecture size and training details but emphasizes predictable scaling and long-context/vision work as system-level extensions. 4 This branch feeds instruction tuning and alignment, reasoning with large language models, and agentic and coding models.
Encoder-only bidirectional pretraining
BERT took the opposite mask choice: a multi-layer Transformer encoder with fully visible self-attention, [CLS]/[SEP] task formatting, segment embeddings, and position embeddings. 5 Its masked language modeling objective lets each token representation condition on both left and right context, while next sentence prediction and fine-tuning heads turned the same encoder into a strong NLU substrate. 5 Architecturally, BERT demonstrates that Transformer does not imply generation; the same attention block becomes a representation learner when the decoder and causal mask are removed. This encoder-only path is one anchor of pretraining and transfer learning and later parameter-efficient fine-tuning.
Text-to-text unification
T5 returned to encoder-decoder Transformers but made the interface uniform: every task is cast as text input plus text output, so training, fine-tuning, and decoding share one procedure. 6 Its encoder uses fully visible attention, the decoder uses causal self-attention plus cross-attention, and the paper compares this encoder-decoder choice against decoder-only language models and prefix-LM variants inside the same text-to-text setup. 6 The main lesson is that architecture and objective co-design matter: span-corruption denoising over C4 made encoder-decoder pretraining especially effective, even though the underlying blocks are close to the original Transformer. 6
Vision and multimodal transfer
ViT showed that a Transformer encoder can operate on images by changing tokenization rather than the core block: split an image into fixed patches, linearly project each patch, add a class token and position embeddings, then run a standard encoder. 7 This strips away most CNN-specific inductive bias, so performance depends heavily on large-scale pretraining; with enough data, the patch-token Transformer becomes competitive with or stronger than convolutional baselines. 7
CLIP then made Transformer architecture part of a multimodal transfer system: an image encoder, often a ResNet or ViT, is paired with a Transformer text encoder and trained contrastively on 400M image-text pairs. 8 At inference, prompts and class names become text embeddings that act like zero-shot classifier weights, shifting the architectural interface from generating tokens or classifying a [CLS] vector to aligning modalities in a shared embedding space. 8 GPT-4 extends the GPT branch toward multimodal input by accepting interleaved text and images while still producing text outputs, tying vision-language capability back to a Transformer-style next-token model. 4 These models sit closest to multimodal foundation models.
Transformer as the common substrate
Across these branches, the stable substrate is stacked attention and feed-forward blocks with residual/norm scaffolding; the changing design choices are the attention mask, input tokenization, objective, and output interface. 1 Encoder-only BERT, decoder-only GPT, encoder-decoder T5, patch-token ViT, contrastive CLIP, and multimodal GPT-4 are therefore less separate inventions than different operating modes of the same architectural family. 5 3 6 7 8 4
This explains why the Transformer became the common substrate for open foundation model families and why its limitations motivate post-Transformer sequence models. The central tension is that self-attention gives flexible global interaction and easy scaling, but its cost and sequence-length behavior push systems work, efficient attention variants, and alternative sequence models to search for cheaper ways to preserve that same flexible conditioning. 1
Footnotes
-
Improving Language Understanding by Generative Pre-Training ↩
-
BERT - Pre-training of Deep Bidirectional Transformers for Language Understanding ↩ ↩2 ↩3
-
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer ↩ ↩2 ↩3 ↩4
-
An Image is Worth 16x16 Words - Transformers for Image Recognition at Scale ↩ ↩2 ↩3
-
Learning Transferable Visual Models From Natural Language Supervision ↩ ↩2 ↩3