Week 9

Convolutional networks: contact maps as images

Convolutional networks, motivated by treating a protein's L × L map of residue-pair features as an image. Lectures cover why convolutions fit images, how convolution and pooling layers work, and the classic architectures, then the general machinery of training deep nets: model class, loss, regularization, and optimization. The AlphaFold 1 papers are the case study: a deep dilated ResNet predicts inter-residue distance distributions, which become a potential minimized by gradient descent, linking back to the structure-as-optimization view of weeks 3–5.

Objectives

  • Compute the output size and parameter count of a convolutional layer from input size, kernel size, padding, stride, and channel counts, and apply max and average pooling.
  • Explain how local connectivity, weight sharing, and translation equivariance make convolutions suited to images, and compare a conv layer's parameter count with a dense layer on the same input.
  • Trace the design changes from LeNet to AlexNet, VGG, GoogLeNet, and ResNet, and explain what residual connections and batch normalization do for training very deep networks.
  • Set up contact or distance prediction as image-to-image learning: build an L × L × C input from sequence and MSA covariation features and predict a per-pair output with a fully convolutional 2D network.
  • Explain how stacking dilated 3 × 3 convolutions grows the receptive field, and why AlphaFold 1 uses 220 dilated residual blocks on 64 × 64 crops of the pair map.
  • Apply and compare regularizers for deep nets: L2 weight decay, dropout, early stopping, data augmentation, and batch normalization.
  • Write the update rules for minibatch SGD, momentum, AdaGrad, RMSProp, and Adam, and explain the roles of learning-rate schedules and weight initialization.
  • Describe how AlphaFold 1 turns predicted distance distributions (64 bins over 2–22 Å) into a protein-specific potential, corrects it with a reference distribution, adds torsion and steric terms, and minimizes it over backbone torsion angles by gradient descent.

Reading

Goodfellow et al. 7–9 or Zhang et al. 6, 7, 13Goodfellow 7 (regularization: norm penalties, augmentation, early stopping, dropout), 8 (optimization: SGD, momentum, adaptive rates, initialization, batch norm), and 9 (convolution, pooling, priors behind them). Zhang's chapters (in the numbering the syllabus follows) are CNNs, modern CNNs (AlexNet to ResNet and DenseNet), and computer vision (augmentation, fine-tuning); run the code for convolution and ResNet.

Lectures

18.1CNNs: why? transcript17:09
18.2CNNs: how? transcript34:31
18.3CNNs: what? + case study transcript32:58
19.1Neural network computation: model class, loss, and regularization transcript33:35
19.2Neural network computation: optimization transcript31:01
2:29:14

Papers

AW Senior et al. “Improved protein structure prediction using potentials from deep learning”. Nature 577, 2020. PDF

Can a neural network predict enough about a protein's geometry that structure prediction becomes simple optimization? A deep dilated residual CNN takes sequence and MSA features and predicts full distance distributions between Cβ atoms plus torsion-angle distributions; these form a potential minimized by gradient descent (L-BFGS) from several starts. At CASP13 AlphaFold reached TM-score ≥ 0.7 on 24 of 43 free-modelling domains versus 14 for the next best method. It is this week's case study for CNNs on contact maps and closes the loop with the energy-minimization lectures.

AW Senior et al. “Protein structure prediction using multiple deep neural networks in the 13th Critical Assessment of Protein Structure Prediction (CASP13)”. Proteins 87(12), 2019. PDF

The CASP13 methods paper for the same AlphaFold entry (group A7D). It compares three systems built from three networks: the distance predictor, a GDT-net that scores candidate structures, and a generative network that proposes backbone fragments for memory-augmented simulated annealing. Simple gradient descent on the distance potential did as well as the costly fragment-assembly search, and the entry ranked first in free modelling (summed z-score 68.3 vs 48.2). Read it alongside the Nature paper to see which components mattered.