Deep Residual Learning for Image Recognition
ReadSkip connections let a layer learn a residual instead of a full mapping, so networks with a hundred layers train. The residual stream inside a transformer comes directly from here.
My notes
The key observation is humbling: deeper plain networks had higher training error, not just test error. So it was an optimization problem, not overfitting.
The Block
y = F(x, {W_i}) + xWhy It Matters Beyond Vision
- Gradients get a direct path back through the identity, so very deep stacks stay trainable.
- In transformers every attention and MLP block writes into a residual stream. Interpretability work literally treats that stream as the model's working memory.
- 152 layers won ImageNet 2015. Before this, ~20 was considered deep.