Week 8

AI and neural networks

The course turns from shallow machine learning to deep learning. Two lectures place AI, machine learning, and deep learning in context and survey where AI is used in healthcare and drug discovery. The third builds the first neural network: a feedforward net trained by gradient descent with back-propagation. Everything later in the course (CNNs for contact maps, RNNs and transformers for sequences, AlphaFold) builds on this.

Objectives

  • Place a given method (rule-based expert system, heuristic search, SVM, deep network) within AI, machine learning, or deep learning, and say what each relies on: hand-coded knowledge, hand-designed features, or learned features.
  • Name where AI enters healthcare and the drug discovery pipeline (target identification, virtual screening, molecule generation, property and toxicity prediction, medical imaging, clinical records), and the kind of data each use needs.
  • Compute the forward pass of a small multilayer perceptron by hand: affine map, activation, and output unit for each layer.
  • Show that no linear model can fit XOR, and give a one-hidden-layer ReLU network that does.
  • Derive the loss for each output unit from maximum likelihood: linear output with squared error, sigmoid with binary cross-entropy, softmax with categorical cross-entropy.
  • Compare ReLU, leaky ReLU, sigmoid, and tanh hidden units, and explain how saturation slows gradient-based training.
  • Compute the gradients of the loss for every weight in a two-layer network by back-propagation on a computational graph, applying the chain rule node by node.
  • State the universal approximation theorem and explain why it does not guarantee learnability, and why deeper networks can represent some functions with far fewer units than shallow ones.

Reading

Goodfellow et al. 6Goodfellow's deep feedforward networks chapter: the XOR example, gradient-based learning with maximum-likelihood costs, output and hidden units, architecture design (universal approximation, depth), and back-propagation on computational graphs. Focus on 6.1–6.2 and 6.5; work the XOR network and the back-propagation algorithm by hand.

Lectures

16.1Introduction to artificial intelligence transcript46:54
16.2AI in healthcare and drug discovery transcript25:30
17A first take on neural networks transcript54:44
2:07:08