BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
ReadPre-trains a transformer encoder by masking tokens and predicting them from both sides, then fine-tunes for each task. The template for encoder models and most embedding models.
My notes
Encoder-only is still what I reach for in rerankers and classifiers. Good to finally read where the [CLS] token convention came from.
Notes
- Masking is what makes bidirectionality possible — a left-to-right LM cannot see the right side, BERT can because the target is hidden.
- Of the chosen 15%: 80% become [MASK], 10% random token, 10% unchanged, so the model does not overfit to the [MASK] token.
- Next sentence prediction turned out to be mostly unnecessary (RoBERTa dropped it).
- Cross-encoders for reranking are basically BERT with a [CLS] relevance head — the same setup I use in hybrid retrieval.