Sequence to Sequence Learning with Neural Networks
ReadAn encoder reads a sentence into one vector and a decoder writes the translation out. The encoder-decoder shape, and the bottleneck it creates, set up attention.
My notes
The most useful thing about reading this now is seeing the bottleneck clearly — the whole sentence squeezed into one fixed vector. Attention is the obvious fix once you see it.
Surprising Details
- Reversing the source sentence helped a lot — it shortens the distance between the first source words and the first target words.
- 4-layer LSTMs, beam search at decode time. Beam size 2 already got most of the gain.
- Long sentences did better than expected, but the fixed-size bottleneck is clearly the ceiling.
Read this right before "Attention Is All You Need" if you can. The transformer makes much more sense as "seq2seq minus the bottleneck, minus the recurrence".