Papers and annotated summaries.
Introduces the Transformer, a sequence model with no recurrence or convolution. Every token attends to every other token through stacked multi-head self-attention, so training runs in parallel across the whole sequence…
Pre-trains a Transformer encoder on unlabeled text by hiding random words and recovering them from context on both sides. One pre-trained model can then be adapted to many tasks with a small output layer and brief…
Very deep networks used to train worse as layers were added. ResNet has each block learn only a correction that is added to its input through a shortcut, so gradients flow easily and networks with 100+ layers train…
Generates images by learning to reverse a process that gradually turns data into noise. A network predicts the noise added at each step; sampling starts from pure noise and removes it step by step. Why it matters It…
Trains hundreds of models to find the best split of a fixed compute budget between model size and training data. The answer: grow both at about the same rate. The 70B Chinchilla model, trained on far more tokens, beat…
Freezes the pre-trained weights and trains a pair of small low-rank matrices beside selected layers. This cuts trainable parameters and optimizer memory by orders of magnitude while staying close to full fine-tuning…
The Vision Transformer (ViT) splits an image into patches, treats each patch as a token and feeds the sequence to a standard Transformer encoder, with no convolutions. Why it matters With enough pre-training data it…
Shows that the RLHF objective can be optimised directly on pairs of preferred and rejected answers with a simple classification-style loss, with no separate reward model and no RL loop. Why it matters It matched or beat…
Uses large-scale reinforcement learning with automatically checkable rewards, such as correct maths answers or passing code tests, to teach a model long step-by-step reasoning without human-written reasoning examples…