Transformer#
The Transformer is an architecture proposed by Google in 2017 in “Attention Is All You Need”. It is built entirely on self-attention, dropping RNNs and CNNs, and has become the foundation of modern NLP and large models.
1. What It Is#
The Transformer is a sequence-to-sequence (Seq2Seq) model architecture. Its core is self-attention, which processes the relationships between all positions in a sequence in parallel.
Key characteristics:
- Parallel computation: unlike RNNs, which process tokens step by step, it handles the whole sequence at once
- Long-range dependencies: attention is computed directly between any two positions, with no information decay
- Scalability: stacking more layers and adding parameters improves capability (GPT, BERT, etc.)
Overall architecture:
graph TB
subgraph Encoder
A[Input Embedding] --> B["#43; Positional Encoding"]
B --> C[Multi-Head Attention]
C --> D[Add & Norm]
D --> E[Feed Forward]
E --> F[Add & Norm]
end
subgraph Decoder
G[Output Embedding] --> H["#43; Positional Encoding"]
H --> I[Masked Multi-Head Attention]
I --> J[Add & Norm]
J --> K[Cross Attention]
K --> L[Add & Norm]
L --> M[Feed Forward]
M --> N[Add & Norm]
end
F --> K
N --> O[Linear + Softmax]
Core components:
| Component | Role |
|---|---|
| Self-Attention | Computes association weights between each position and every other position in the sequence |
| Multi-Head | Runs several attention heads in parallel to capture features in different subspaces |
| Positional Encoding | Injects position information (sinusoidal or learned) |
| Feed Forward | Two fully connected layers + activation, applied position-wise |
| Add & Norm | Residual connection + LayerNorm, stabilizes training |
Three variants:
| Type | Representative models | Structure | Typical tasks |
|---|---|---|---|
| Encoder-only | BERT | Encoder only | Classification, NER, sentence similarity |
| Decoder-only | GPT, LLaMA | Decoder only | Text generation, dialogue |
| Encoder-Decoder | T5, BART | Full structure | Translation, summarization |
2. How to Use It#
Using Hugging Face Transformers#
from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
# ========== Encoder model (e.g. BERT) ==========
tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
model = AutoModel.from_pretrained("bert-base-chinese")
text = "The Transformer is the foundation of modern NLP"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# Token embeddings (batch_size, seq_len, hidden_size)
last_hidden = outputs.last_hidden_state
# Sentence embedding (the [CLS] position)
sentence_embedding = last_hidden[:, 0, :]
# ========== Decoder model (e.g. GPT) ==========
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))
Common Task Examples#
from transformers import pipeline
# Text classification
classifier = pipeline("sentiment-analysis")
classifier("I love this movie!") # [{'label': 'POSITIVE', 'score': 0.99}]
# Text generation
generator = pipeline("text-generation", model="gpt2")
generator("The future of AI is", max_length=50)
# Question answering
qa = pipeline("question-answering")
qa(question="What is Transformer?", context="Transformer is a neural network architecture.")
# Translation
translator = pipeline("translation_en_to_zh", model="Helsinki-NLP/opus-mt-en-zh")
translator("Hello, world!")
# Fill in the blank
fill = pipeline("fill-mask", model="bert-base-chinese")
fill("The weather is really [MASK] today")
Common Pretrained Models#
| Model | Type | Parameters | Highlights |
|---|---|---|---|
| BERT | Encoder | 110M/340M | Bidirectional encoding, suited to understanding tasks |
| GPT-2 | Decoder | 117M-1.5B | Autoregressive generation |
| T5 | Enc-Dec | 60M-11B | Unified text-to-text format |
| LLaMA | Decoder | 7B-70B | Open source and efficient, today’s mainstream base model |
| Qwen | Decoder | 0.5B-72B | Strong in both Chinese and English |
| ChatGLM | Decoder | 6B-130B | Optimized for Chinese dialogue |
Further Reading#
- Attention formula:
Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V - Positional encoding: the original uses sinusoidal encoding; modern models mostly use RoPE (Rotary Position Embedding)
- Optimized variants: FlashAttention (memory optimization), GQA (Grouped-Query Attention), sliding window