Efficient Estimation of Word Representations in Vector Space
ReadTrains dense word vectors from a simple prediction task and shows that arithmetic on them recovers analogies. The cleanest introduction to learned embeddings.
My notes
Everything I do in retrieval sits on top of this idea: meaning as position in a vector space. The paper is short and oddly modest about how big the idea is.
Two Architectures
vec("king") − vec("man") + vec("woman") ≈ vec("queen")What I Took Away
- The trick was removing complexity — no hidden non-linear layer — so it could train on billions of tokens. Scale beat cleverness, a theme that keeps coming back on this list.
- Skip-gram is better for rare words, CBOW is faster. Negative sampling (the follow-up paper) is what made it practical.
- Embeddings are learned as a by-product of a prediction task nobody actually cares about. Same pattern as BERT and GPT later.
Connects directly to my hybrid retrieval work: dense vectors capture "similar meaning", but BM25 still wins on exact identifiers. word2vec is where that trade-off starts.