Week 11
Protein language models
Protein language models: transformers trained with self-supervision on millions of unlabeled sequences. The idea is representation learning plus transfer: pre-train once on raw sequences, then reuse the embeddings for structure, interactions, and mutational effects with little labeled data. Rives et al. show structure emerging in ESM-1b's representations, Bepler and Berger survey the field and add structural supervision, and Marquet et al. predict conservation and variant effects from embeddings without MSAs. This extends week 10's attention to the full transformer.
Objectives
- Assemble a transformer encoder block (multi-head self-attention, positional encoding, residual connections, layer normalization, position-wise feed-forward) and state how its cost grows with sequence length.
- Write the masked language modeling loss used by BERT and ESM and the autoregressive (next-token) loss, and say which one gives bidirectional context and which allows direct sampling.
- Distinguish supervised, unsupervised, semi-supervised, and self-supervised learning, and place pre-training and fine-tuning among them.
- Choose between a linear probe, a small head on frozen embeddings, and full fine-tuning for a downstream protein task, given its labeled-data size.
- Extract per-residue and per-protein (mean-pooled) embeddings from a pre-trained pLM and train a predictor on them for a residue-level or protein-level property.
- Explain how Rives et al. show that secondary structure and residue contacts are linearly decodable from ESM-1b representations, and why held-out sequence clusters are needed for that claim.
- Describe Bepler and Berger's multi-task model: a bidirectional LSTM language model also trained on contact prediction and structural similarity, and what that supervision adds to the embeddings.
- Score a missense variant from a pLM (log-probability of mutant versus wild-type residue at a masked position), and describe how VESPA combines predicted conservation, BLOSUM62, and pLM substitution probabilities, evaluated by Spearman correlation on DMS data.
Reading
| Goodfellow et al. 15 or Zhang et al. 14 | Goodfellow 15 covers representation learning: greedy unsupervised pre-training, transfer learning and domain adaptation, semi-supervised learning, and distributed representations. Zhang's NLP pre-training chapter (syllabus numbering) builds word2vec embeddings and BERT with masked language modeling, the recipe protein language models copy. |
Papers
A Rives et al. “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences”. PNAS 118(15), 2021. PDF
Do the language-model methods of NLP learn biology when trained on evolutionary sequence data? The authors train transformers with masked language modeling on up to 250 million UniParc sequences, ending with the 650M-parameter, 33-layer ESM-1b. Its representations organize by amino acid properties and remote homology, and linear projections recover secondary structure and long-range contacts that generalize across folds; features improve contact, secondary structure, and mutational effect prediction. It is the anchor case study for pLMs.
T Bepler and B Berger. “Learning the protein language: Evolution, structure, and function”. Cell Systems 12(6), 2021. PDF
A review of protein language models for property prediction: statistical and physics-based modeling, then RNN and transformer language models, embeddings, and transfer learning. The authors argue that sequence alone is not enough and add priors through multi-task learning, training a bidirectional LSTM with contact prediction and structural similarity tasks. It gives the conceptual map (glossary, model types, open problems) for the week's topics.
C Marquet et al. “Embeddings from protein language models predict conservation and variant effects”. Human Genetics 141, 2022. PDF
Can pLM embeddings replace multiple sequence alignments for variant effect prediction? A CNN on ProtT5 embeddings predicts residue conservation from single sequences nearly as well as ConSeq with MSAs (MCC 0.596 vs 0.608). VESPA feeds that prediction, BLOSUM62, and ProtT5 substitution probabilities to logistic regression; on 39 DMS sets it is competitive with ESM-1v, DeepSequence, and GEMME, and scores the human proteome in about 40 minutes on one GPU. It shows embeddings applied to mutational effects.