Week 10
Recurrent networks and attention: sequences as text
Proteins as text: a protein sequence is a string over 20 letters, so models from natural language processing apply. After a CNN recap, lectures build recurrent networks, show why their gradients vanish or explode, fix that with LSTM and GRU gating, and apply RNNs to protein structure prediction. The week ends with attention, introduced through image captioning and machine translation, which sets up the transformers and protein language models of week 11.
Objectives
- Write the hidden-state recurrence of a vanilla RNN, unroll it over a sequence, and identify which weights are shared across time steps.
- Choose an RNN setup (many-to-one, aligned many-to-many, bidirectional, encoder–decoder) for a given task, such as per-residue secondary structure versus whole-protein fold class.
- Show with back-propagation through time that gradients are products of Jacobians over time steps, explain when they vanish or explode, and apply gradient clipping.
- Write the LSTM (input, forget, output gates and cell state) and GRU (reset and update gates) equations, and explain how gating preserves long-range information.
- Set up a bidirectional RNN that maps a protein sequence (one-hot or profile features) to per-residue structural outputs such as secondary structure or backbone torsion angles.
- Explain soft attention in image captioning: the decoder's state weights CNN feature-map locations to form a context vector for each word.
- Explain how additive (Bahdanau) attention in encoder–decoder translation replaces the single fixed-length context vector, and compute the attention weights for a toy example.
- Compute scaled dot-product attention from queries, keys, and values, including the softmax normalization and the 1/√d scaling.
Reading
| Goodfellow et al. 10 or Zhang et al. 8, 10 | Goodfellow 10 covers unfolded computational graphs, RNN variants, bidirectional and encoder–decoder RNNs, long-term dependencies, LSTM and GRU, and gradient clipping. Zhang's chapters (syllabus numbering) are RNNs with back-propagation through time, then attention mechanisms (scoring functions, Bahdanau attention, multi-head and self-attention, positional encoding); work the attention code. |
Lectures
| 20.0 | CNN recap transcript | 8:15 |
| 20.1 | Recurrent neural networks transcript | 25:58 |
| 20.2 | The exploding or vanishing gradient problem transcript | 23:58 |
| 20.3 | Modern RNNs: LSTM and GRU transcript | 24:48 |
| 20.4 | RNNs for protein structure prediction transcript | 11:35 |
| 21.1 | Introducing attention: image captioning transcript | 17:09 |
| 21.2 | Introducing attention: language translation transcript | 21:24 |
| 2:13:07 |