Papers

The first learning rule for a linear classifier: nudge the weights after every mistake. Every network since is stacks of this unit with a smoother training signal.

Frank Rosenblatt · 1958 · Foundations Read

A deep convolutional network trained on GPUs cut ImageNet error nearly in half. It shows why depth, ReLU, and data augmentation became the default recipe.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012 · Vision Read

Derives a tractable lower bound for training a latent-variable generative model by gradient descent. The probabilistic groundwork under diffusion models.

Kingma and Welling · 2013 · Generative models Read

Trains dense word vectors from a simple prediction task and shows that arithmetic on them recovers analogies. The cleanest introduction to learned embeddings.

Mikolov et al. · 2013 · Embeddings, NLP Read

Per-parameter step sizes from running estimates of the gradient's mean and variance. The optimizer most later papers assume without discussion.

Kingma and Ba · 2014 · Foundations, Optimization Read

Randomly dropping units during training makes a network behave like an averaged ensemble. A regularizer you meet in nearly every architecture.

Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov · 2014 · Foundations, Regularization Read

Trains a generator against a discriminator as a two-player game instead of maximizing likelihood. Read it to understand adversarial training and why it is unstable.

Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio · 2014 · Generative models Read

An encoder reads a sentence into one vector and a decoder writes the translation out. The encoder-decoder shape, and the bottleneck it creates, set up attention.

Sutskever, Vinyals, and Le · 2014 · Sequence models, NLP Read

Normalizing activations per mini-batch lets much deeper networks train with larger learning rates. LayerNorm and RMSNorm are variations on this move.

Ioffe and Szegedy · 2015 · Foundations Read

Skip connections let a layer learn a residual instead of a full mapping, so networks with a hundred layers train. The residual stream inside a transformer comes directly from here.

He et al. · 2015 · Vision, Foundations Read

Trains a small model to match a large model's softened output distribution rather than hard labels. The basis for most model compression and teacher-student setups.

Hinton, Vinyals, and Dean · 2015 · Compression Read

Replaces recurrence with attention so every token can look at every other token in one step. Every later language model paper assumes you know this one.

Vaswani et al. · 2017 · Transformers, NLP Read

Pre-trains a transformer encoder by masking tokens and predicting them from both sides, then fine-tunes for each task. The template for encoder models and most embedding models.

Devlin, Chang, Lee, and Toutanova · 2018 · Transformers, NLP Read

Trains a network to reverse a gradual noising process, with a simple objective that turns out to be a reweighted variational bound. The foundation of modern image generation.

Ho, Jain, and Abbeel · 2020 · Generative models Read

Shows that a large enough decoder can do new tasks from examples in its prompt, with no gradient updates. In-context learning and prompting start here.

Brown et al. · 2020 · LLMs, Prompting Read

Pairs a dense retriever with a generator so answers can draw on documents outside the weights. The reference design behind retrieval-augmented systems.

Lewis et al. · 2020 · RAG Read

Loss falls as a smooth power law in parameters, data, and compute across many orders of magnitude. Turned model sizing into something you could plan.

Kaplan et al. · 2020 · Scaling, LLMs Read

Fine-tunes a frozen model by learning low-rank updates to its weight matrices. It is how most people adapt a large model on one GPU.

Hu et al. · 2021 · Fine-tuning Read

Routes each token to a single expert feed-forward block, so parameter count grows without growing compute per token. The simplified mixture-of-experts behind many current large models.

Fedus, Zoph, and Shazeer · 2021 · Scaling, Transformers Read

Asking a large model to write intermediate steps before its answer raises accuracy on reasoning tasks. A short paper that reframes prompting as controlling computation.

Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, and Zhou · 2022 · Prompting, LLMs Read

Interleaves reasoning traces with tool actions so a model can plan, act, and revise. Most agent loops are a variation on this pattern.

Yao, Zhao, Yu, Du, Shafran, Narasimhan, and Cao · 2022 · Agents Read

Revisits the scaling fits and finds most large models were undertrained on data for their size. Explains why later models use far more tokens per parameter.

Hoffmann et al. · 2022 · Scaling, LLMs Read

Applies supervised fine-tuning, a preference reward model, and PPO to make GPT-3 follow instructions. The alignment pipeline every chat model descends from.

Ouyang et al. · 2022 · Alignment, Fine-tuning Read

Rewrites the RLHF objective so the policy trains directly on preference pairs, with no reward model or sampling loop. Read it to see what the RL step was actually computing.

Rafailov et al. · 2023 · Alignment, Fine-tuning Read

Dettmers et al. · 2023 · Fine-tuning Read Illustrative

A promptable segmentation model trained on a billion masks that transfers without fine-tuning. Shows what a foundation model looks like outside language.

Kirillov et al. · 2023 · Vision Read

A model labels its own training data with API calls that improve its predictions, then learns to make them. Shows how tool use can be trained rather than prompted.

Schick et al. · 2023 · Agents, Fine-tuning Read