Word2Vec#
Word2Vec is a word-vector training method introduced by Google in 2013. It maps words to dense, low-dimensional vectors so that semantically similar words end up close together in vector space.
1. What It Is#
Word2Vec is a word embedding method. Its core idea is that “a word’s meaning is determined by the words around it” (the distributional hypothesis).
Key characteristics:
- Dense vectors: each word is represented by a 100-300 dimensional float vector instead of a high-dimensional sparse one-hot vector
- Semantic similarity: semantically similar words have high cosine similarity (e.g. “king” and “queen”)
- Vector arithmetic: supports semantic analogies, such as
king - man + woman ≈ queen
Two architectures:
| Architecture | Input | Output | Characteristics |
|---|---|---|---|
| CBOW | Context words | Center word | Fast to train, suited to large corpora |
| Skip-gram | Center word | Context words | Better quality, suited to small corpora |
graph LR
subgraph CBOW
A1[the] --> B1[?]
A2[cat] --> B1
A3[on] --> B1
A4[mat] --> B1
B1 --> C1[sat]
end
subgraph Skip-gram
D1[sat] --> E1[the]
D1 --> E2[cat]
D1 --> E3[on]
D1 --> E4[mat]
end
2. How to Use It#
Using a Pretrained Model#
from gensim.models import KeyedVectors
# Load pretrained word vectors (e.g. Google News, 300 dimensions)
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
# Get a word vector
vector = model['king'] # 300-dimensional vector
# Compute similarity
similarity = model.similarity('king', 'queen') # 0.65
# Find the most similar words
similar_words = model.most_similar('python', topn=5)
# Semantic analogy: king - man + woman = ?
result = model.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
# [('queen', 0.71)]
Common Pretrained Models#
| Model | Dimensions | Vocabulary | Source |
|---|---|---|---|
| Google News | 300 | 3M | News corpus |
| GloVe | 50-300 | 400K | Wikipedia + Gigaword |
| fastText | 300 | 2M | Common Crawl |
| Tencent Chinese | 200 | 8M | Chinese corpus |
3. How to Train It#
Training with Gensim#
from gensim.models import Word2Vec
# Prepare the corpus: a list of tokenized sentences
sentences = [
['i', 'love', 'machine', 'learning'],
['deep', 'learning', 'is', 'fun'],
['natural', 'language', 'processing', 'is', 'NLP'],
# ... more sentences
]
# Train the model
model = Word2Vec(
sentences,
vector_size=100, # vector dimensions
window=5, # context window size
min_count=1, # minimum word frequency
sg=1, # 0=CBOW, 1=Skip-gram
workers=4, # number of parallel threads
epochs=10, # number of training epochs
)
# Save and load
model.save('word2vec.model')
model = Word2Vec.load('word2vec.model')
# Use
vector = model.wv['machine']
similar = model.wv.most_similar('learning')
Key Parameters#
| Parameter | Description | Recommended |
|---|---|---|
vector_size | Vector dimensions | 100-300 |
window | Context window | 5-10 |
min_count | Minimum word frequency filter | 5-10 |
sg | 0=CBOW, 1=Skip-gram | 1 for small corpora |
negative | Number of negative samples | 5-20 |
epochs | Number of training epochs | 5-20 |
Training Tips#
- Corpus size: at least millions of sentences; the larger the corpus, the better the results
- Tokenization quality: Chinese text must first be segmented with a tool like jieba
- Window size: small windows capture syntactic relations, large windows capture semantic relations
- Negative sampling: faster and more widely used than Hierarchical Softmax
Further Reading#
- GloVe: combines global word co-occurrence statistics with local context
- fastText: supports subwords, so it can handle out-of-vocabulary words
- ELMo / BERT: contextual, dynamic word vectors that resolve polysemy