Yash DabhadeAI Engineer

← Papers

Attention Is All You Need

Vaswani et al. · 2017 · read 17 Sept 2026

Read

Replaces recurrence with attention so every token can look at every other token in one step. Every later language model paper assumes you know this one.

Read the paper ↗

My notes

Third time reading it, first time it felt easy — because I read seq2seq, ResNet and LayerNorm first. Order matters more than I thought.

One Transformer Block

Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

Pieces Worth Really Understanding

  • Q·Kᵀ scores every token pair at once — that parallelism, not accuracy, is why it replaced RNNs.
  • Divide by √d_k or the softmax saturates for large dimensions and gradients vanish.
  • Multiple heads = multiple subspaces to attend in. In practice some heads do syntax, some do position, some do copying.
  • Cost is O(n²) in sequence length. Every long-context trick since (sparse, sliding window, flash attention) is fighting this.

How It Connects to My Work

When a RAG answer ignores a retrieved chunk, it is ultimately an attention problem — the chunk is in context but not attended to. Ordering and chunk size matter because of this mechanism.