Word2Vec#

Word2Vec is a word-vector training method introduced by Google in 2013. It maps words to dense, low-dimensional vectors so that semantically similar words end up close together in vector space.


1. What It Is#

Word2Vec is a word embedding method. Its core idea is that “a word’s meaning is determined by the words around it” (the distributional hypothesis).

Key characteristics:

  • Dense vectors: each word is represented by a 100-300 dimensional float vector instead of a high-dimensional sparse one-hot vector
  • Semantic similarity: semantically similar words have high cosine similarity (e.g. “king” and “queen”)
  • Vector arithmetic: supports semantic analogies, such as king - man + woman ≈ queen

Two architectures:

ArchitectureInputOutputCharacteristics
CBOWContext wordsCenter wordFast to train, suited to large corpora
Skip-gramCenter wordContext wordsBetter quality, suited to small corpora
graph LR
    subgraph CBOW
        A1[the] --> B1[?]
        A2[cat] --> B1
        A3[on] --> B1
        A4[mat] --> B1
        B1 --> C1[sat]
    end
    subgraph Skip-gram
        D1[sat] --> E1[the]
        D1 --> E2[cat]
        D1 --> E3[on]
        D1 --> E4[mat]
    end

2. How to Use It#

Using a Pretrained Model#

from gensim.models import KeyedVectors

# Load pretrained word vectors (e.g. Google News, 300 dimensions)
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)

# Get a word vector
vector = model['king']  # 300-dimensional vector

# Compute similarity
similarity = model.similarity('king', 'queen')  # 0.65

# Find the most similar words
similar_words = model.most_similar('python', topn=5)

# Semantic analogy: king - man + woman = ?
result = model.most_similar(positive=['king', 'woman'], negative=['man'], topn=1)
# [('queen', 0.71)]

Common Pretrained Models#

ModelDimensionsVocabularySource
Google News3003MNews corpus
GloVe50-300400KWikipedia + Gigaword
fastText3002MCommon Crawl
Tencent Chinese2008MChinese corpus

3. How to Train It#

Training with Gensim#

from gensim.models import Word2Vec

# Prepare the corpus: a list of tokenized sentences
sentences = [
    ['i', 'love', 'machine', 'learning'],
    ['deep', 'learning', 'is', 'fun'],
    ['natural', 'language', 'processing', 'is', 'NLP'],
    # ... more sentences
]

# Train the model
model = Word2Vec(
    sentences,
    vector_size=100,   # vector dimensions
    window=5,          # context window size
    min_count=1,       # minimum word frequency
    sg=1,              # 0=CBOW, 1=Skip-gram
    workers=4,         # number of parallel threads
    epochs=10,         # number of training epochs
)

# Save and load
model.save('word2vec.model')
model = Word2Vec.load('word2vec.model')

# Use
vector = model.wv['machine']
similar = model.wv.most_similar('learning')

Key Parameters#

ParameterDescriptionRecommended
vector_sizeVector dimensions100-300
windowContext window5-10
min_countMinimum word frequency filter5-10
sg0=CBOW, 1=Skip-gram1 for small corpora
negativeNumber of negative samples5-20
epochsNumber of training epochs5-20

Training Tips#

  • Corpus size: at least millions of sentences; the larger the corpus, the better the results
  • Tokenization quality: Chinese text must first be segmented with a tool like jieba
  • Window size: small windows capture syntactic relations, large windows capture semantic relations
  • Negative sampling: faster and more widely used than Hierarchical Softmax

Further Reading#

  • GloVe: combines global word co-occurrence statistics with local context
  • fastText: supports subwords, so it can handle out-of-vocabulary words
  • ELMo / BERT: contextual, dynamic word vectors that resolve polysemy