# Lecture 20.4: RNN for Protein Structure Prediction

[Recording](https://www.youtube.com/watch?v=VbTEWHJ-9XY)

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Okay, in the end I will talk about a case study. Again I'm going to talk about protein structure prediction.

00:00:08 Previously we talked about treating protein structures as 2D images.

00:00:15 There are different kind of formulations. For instance, I regard, I represent the protein structure, the 3D protein structure as a 2D matrix or graph, right?

00:00:30 Well each row is a residue or amino acid and then I have, well each column is an amino acid as well.

00:00:36 Then I have a 2D grid, right? Well each pixel is corresponding to the something between a pair of residues, right?

00:00:49 That something could be one channel, like binary.

00:00:52 Yes or not or somewhere in between 0 and 1 for the probability that two residues are close to each other, right?

00:01:00 This could be two channels, right?

00:01:04 That, you know, both are about the distance, the actual real value, the distance instead of, you know, a binary or between 0 and 1, the actual distance.

00:01:14 But one channel is for the mean and another channel is the standard deviation and something, right?

00:01:19 So if that is the case, then these so-called contact prediction, the residue-residue contact prediction problem, right?

00:01:30 Which was an easier problem for protein structure prediction in the 3D, but that was a major breakthrough three or four years ago, right?

00:01:40 Because people then can utilize the predicted context and do the 3D prediction, right?

00:01:45 So that was not end-to-end, but the field was not ready for end-to-end.

00:01:49 And then let's think about this contact prediction problem is essentially, you know, a pixel level image segmentation problem

00:01:58 because you are trying to predict for each pixel, you know, the values in one or multiple channels, right?

00:02:07 And the way this, you know, image task is solved is also, you know, built upon treating protein features as images.

00:02:18 If you remember, you know, you have a sequence, right?

00:02:22 But at each position, you have a bunch of features like 20 dimensions of one heart, 20 dimensions of position-specific scoring function or matrices,

00:02:31 three dimensions of, you know, secondary structures, right?

00:02:36 Two dimensions of solving accessibility and so on, right?

00:02:41 So actually, you know, and then you have a matrix, right?

00:02:45 So that's the input.

00:02:46 I was talking about the output first.

00:02:48 So that's the image task, but the input people also treated as image, even though those features don't have really a, you know, order, right?

00:03:00 It's not like in natural images, you do have order, right?

00:03:04 So, but still it didn't prevent people from utilizing convolutional neural networks and residual convolutional neural networks

00:03:13 and treat them as a, you know, image task, right?

00:03:17 So it's an image task by learning from images, right?

00:03:22 But this neither image is natural, right?

00:03:26 The output image at least makes some sense because there's order across the different residues, right?

00:03:33 The input image doesn't make even much sense if you think about it, okay?

00:03:38 But still, right, there was much success that was also, you know, leading to the first alpha-fold one, right?

00:03:46 So now let's talk about a different formulation today, okay?

00:03:51 So if we are really, you know, interested in the end-to-end prediction, well, the output end is not going to be just the residue contact map, but exact the 3D coordinates or, you know, Cartesian or internal coordinates.

00:04:08 So this is sort of the end-to-end differentiable learning that has been a major thing in computer vision and NLP, but, you know, first it was introduced by, you know, Al-Kawashi in 2019.

00:04:23 So back then the performance wasn't fantastic and, you know, there wasn't a lot of attention paid to it, but I think this was apparently picked up by alpha-fold two, right?

00:04:38 Of course, you can say alpha-fold two is doing end-to-end differential learning, but it took people a long while to figure out how to do that among other things, okay?

00:04:49 But this was, you know, his solution back then, right?

00:04:52 So what he proposed was a recurrent geometric networks because, again, what you have is a protein sequence that has an order, right?

00:05:02 Right?

00:05:03 So, you know, so in this case, right, you know, what they used is essentially a recurrent neural networks.

00:05:10 Well, we have these, you know, internal states and then they're going to try to use these internal states and predict or output the geometry.

00:05:22 So in other words, it's like, you know, machine translation is accepted that the one language is in the amino acid field, another language is in the geometry field, right?

00:05:32 Because in our case, there is a much simpler structure compared to the natural languages.

00:05:40 That is, our tokenization is already defined.

00:05:45 In other words, we have character level sequences, right?

00:05:51 We don't have to do the word and tokenization, right?

00:05:54 So our tokens are characters.

00:05:56 And the input and output have perfect correspondence, right?

00:06:02 As we will see later on, if you are translating from French to English, right, you don't have the perfect structure that the first word in French sentence is going to be the first word in the English sentence.

00:06:20 Is that right?

00:06:21 So in our case, it's actually a perfect correspondence because for the first residue we are trying to predict is phi, psi, and omega.

00:06:29 Remember, we learned in the early weeks the way you can do the dimensionality reduction for the 3D coordinate space is to consider the phi, psi, omega, the backbone torsional angles, right?

00:06:43 We disregard the boundary length, we disregard the boundary length, we disregard the boundary angles, and you only pay attention to the torsions, right?

00:06:52 The Ramachandran plot is even saying that let's don't even pay attention to omega, which is very restricted, let's only pay attention to phi and psi.

00:07:00 So once you have these phi and psi, right, which are like the French of your protein sequence, which is English, right, although this is much easier, and then you can remember, you know, this is not your end goal, right?

00:07:15 Your end goal is the resulting structure, right?

00:07:20 The resulting 3D structure, right?

00:07:22 So to put this, you know, into an end-to-end fashion, your loss function has to be exactly what you care about.

00:07:29 That is, you know, starting from this, you know, this geometry predicted, you can think of them as intermediate outputs,

00:07:37 they're going to eventually use to assemble into a 3D structure and calculating the distance RMSD root mean standard, root mean squared difference, okay?

00:07:51 or root mean squared deviation, as we said, that can be used to measure the difference between two structures, right?

00:08:01 And this is calculated based on all the distances so that you don't have to do alignment, right?

00:08:08 Then you have a loss, right?

00:08:09 So then this whole thing is trained end-to-end, and then, you know, make sure that this can back-crop gate all the way to calculate these parameters, okay?

00:08:20 And let's take a look at these individual blocks, right?

00:08:25 So you have the, I think this is very similar to the GRUs essentially, because there's no cell state, there's only hidden state, right?

00:08:35 You have the hidden state from the prior or the last time point T minus one, right?

00:08:40 But notice that they use the three kinds of hidden state, I think it's respectively, right?

00:08:48 Respectively, you know, for corresponding to, maybe corresponding to the fact that they are looking at different kind of geometries to predict, right?

00:08:58 But, I think this is still a single, single HT minus one, but it's going to be fed into different parameters of different gates, that's my understanding, right?

00:09:13 So, different gating mechanisms, which you're putting here, with different parameters, right?

00:09:18 So, he's seemingly still using three gates, right? If I understand this correctly.

00:09:24 And you determine how much to forget, how much to rewrite, and then at the end, you know, you output, right?

00:09:34 So that's the protein structure prediction by recurrent geometric networks, okay?

00:09:41 So, any questions so far?

00:09:46 So, this RGN in 2019 was recently updated last year.

00:09:53 You want to guess one of the major changes in RGN version 2 versus RGN version 1?

00:10:00 Huh?

00:10:02 Secondary structure.

00:10:05 Secondary structure.

00:10:08 You're talking about the features, is that right?

00:10:12 Yeah.

00:10:13 So, unlike alpha fold, they don't emphasize too much features.

00:10:20 They don't even use multiple sequence alignment here, because, probably partially because, you know, if you go the way of alpha fold, it's a very steep competition.

00:10:30 So, they still focus on a single sequence, and they don't put too much about the secondary structure.

00:10:38 But, you're right that you could consider the secondary structure sort of,

00:10:42 intermediately and somehow affecting the loss function.

00:10:47 That is possible, right?

00:10:48 So, that is a good point, right?

00:10:50 Because when you just try to force those distances predicted to match the ground truth distances,

00:10:59 you may have some weird structures that are non-natural, right?

00:11:02 You may break secondary structures, you may break the, you know, the geometry, you may have something really weird, right?

00:11:08 So, any other guess?

00:11:13 Any other guess?

00:11:18 I put a hint here already.

00:11:20 That's right.

00:11:25 So, he replaced the Arnons with transformers, and he used the masked language modeling.

00:11:32 And his masks are slightly different from the traditional ones.

00:11:35 **[Uncertain recognition]** Now, where would be ?
