Transformer#

The Transformer is an architecture proposed by Google in 2017 in “Attention Is All You Need”. It is built entirely on self-attention, dropping RNNs and CNNs, and has become the foundation of modern NLP and large models.


1. What It Is#

The Transformer is a sequence-to-sequence (Seq2Seq) model architecture. Its core is self-attention, which processes the relationships between all positions in a sequence in parallel.

Key characteristics:

  • Parallel computation: unlike RNNs, which process tokens step by step, it handles the whole sequence at once
  • Long-range dependencies: attention is computed directly between any two positions, with no information decay
  • Scalability: stacking more layers and adding parameters improves capability (GPT, BERT, etc.)

Overall architecture:

graph TB
    subgraph Encoder
        A[Input Embedding] --> B["#43; Positional Encoding"]
        B --> C[Multi-Head Attention]
        C --> D[Add & Norm]
        D --> E[Feed Forward]
        E --> F[Add & Norm]
    end
    subgraph Decoder
        G[Output Embedding] --> H["#43; Positional Encoding"]
        H --> I[Masked Multi-Head Attention]
        I --> J[Add & Norm]
        J --> K[Cross Attention]
        K --> L[Add & Norm]
        L --> M[Feed Forward]
        M --> N[Add & Norm]
    end
    F --> K
    N --> O[Linear + Softmax]

Core components:

ComponentRole
Self-AttentionComputes association weights between each position and every other position in the sequence
Multi-HeadRuns several attention heads in parallel to capture features in different subspaces
Positional EncodingInjects position information (sinusoidal or learned)
Feed ForwardTwo fully connected layers + activation, applied position-wise
Add & NormResidual connection + LayerNorm, stabilizes training

Three variants:

TypeRepresentative modelsStructureTypical tasks
Encoder-onlyBERTEncoder onlyClassification, NER, sentence similarity
Decoder-onlyGPT, LLaMADecoder onlyText generation, dialogue
Encoder-DecoderT5, BARTFull structureTranslation, summarization

2. How to Use It#

Using Hugging Face Transformers#

from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM

# ========== Encoder model (e.g. BERT) ==========
tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese")
model = AutoModel.from_pretrained("bert-base-chinese")

text = "The Transformer is the foundation of modern NLP"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Token embeddings (batch_size, seq_len, hidden_size)
last_hidden = outputs.last_hidden_state

# Sentence embedding (the [CLS] position)
sentence_embedding = last_hidden[:, 0, :]


# ========== Decoder model (e.g. GPT) ==========
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0]))

Common Task Examples#

from transformers import pipeline

# Text classification
classifier = pipeline("sentiment-analysis")
classifier("I love this movie!")  # [{'label': 'POSITIVE', 'score': 0.99}]

# Text generation
generator = pipeline("text-generation", model="gpt2")
generator("The future of AI is", max_length=50)

# Question answering
qa = pipeline("question-answering")
qa(question="What is Transformer?", context="Transformer is a neural network architecture.")

# Translation
translator = pipeline("translation_en_to_zh", model="Helsinki-NLP/opus-mt-en-zh")
translator("Hello, world!")

# Fill in the blank
fill = pipeline("fill-mask", model="bert-base-chinese")
fill("The weather is really [MASK] today")

Common Pretrained Models#

ModelTypeParametersHighlights
BERTEncoder110M/340MBidirectional encoding, suited to understanding tasks
GPT-2Decoder117M-1.5BAutoregressive generation
T5Enc-Dec60M-11BUnified text-to-text format
LLaMADecoder7B-70BOpen source and efficient, today’s mainstream base model
QwenDecoder0.5B-72BStrong in both Chinese and English
ChatGLMDecoder6B-130BOptimized for Chinese dialogue

Further Reading#

  • Attention formula: Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V
  • Positional encoding: the original uses sinusoidal encoding; modern models mostly use RoPE (Rotary Position Embedding)
  • Optimized variants: FlashAttention (memory optimization), GQA (Grouped-Query Attention), sliding window