Lecture 21.1: Introducing Attention & Image Captioning

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Okay, so for the rest of this week, we'll talk about these new waves.

00:00:08 Attention, self-attention, transformers, and protein language models.

00:00:13 So I can do like 10 minutes before I give it to the journal club.

00:00:18 So we'll talk about attention and self-attention.

00:00:22 So for this lecture, I don't think there's any need to make any slides by myself

00:00:29 because these perfect slides I think already made by Fei-Fei Li at Stanford.

00:00:36 So here I just shamelessly borrowed their slides in the following.

00:00:40 I think they are very good, and I'll try to explain them to you carefully.

00:00:45 So as we learned before, we have seen these different recurrent neural networks.

00:00:50 We know what these blocks mean, and there's this recurrent formula on road in these representations.

00:00:58 right, in these illustrations, right?

00:01:01 And any reminder, just want to make sure that people don't forget,

00:01:07 can you give one of the applications for these, for instance, many to one?

00:01:13 One of these applications for this many to one kind of recurrent neural networks?

00:01:19 One of these applications for this many to one kind of recurrent neural networks?

00:01:22 The input is having some sequence information,

00:01:24 and ultimately the single label, remember?

00:01:29 Pardon me?

00:01:34 You can classify the language, right?

00:01:39 The sequence information can be important, yeah.

00:01:42 Another application is to, for instance, you read the Twitter message,

00:01:51 and you try to figure out what's the current state of the Twitter, right?

00:01:56 You read the tweet and then try to figure out the current state of the Twitter.

00:02:00 Is he happy or depressed or something like that, right?

00:02:03 And we talked about, you know, it's very similar to the convolution neural networks, right?

00:02:14 That, you know, you can use, the input can be variable lengths,

00:02:18 but we use the shared weights, right, for this FW, right?

00:02:23 So this is shared.

00:02:25 This is very similar to the kernels used in convolutions, right?

00:02:29 Okay, so now let's start talking about attentions,

00:02:33 and then I will generalize to general attentions,

00:02:36 and then go to the special case of the self attentions, okay?

00:02:40 So I think I can start with just attentions.

00:02:43 So let's talk about image captioning.

00:02:46 And remember, the image captioning is you start with image,

00:02:50 and then you will learn through CNN, you have a feature map, okay?

00:02:53 Let's say the feature map is, again, the 2D grid, right?

00:02:57 And you know that far away here, the output is we're going to generate a sentence, okay?

00:03:02 A sequence, okay?

00:03:03 Let's keep that in mind.

00:03:04 Now we actually have seen this architecture before, right?

00:03:07 So I'm going to extend this one by one.

00:03:10 So what you could have is something I didn't talk about is, you know,

00:03:15 this encoder-decoder structure that I haven't talked about.

00:03:19 I gave it in one of the slides.

00:03:21 But you can think of this as, you know, an encoder here and a decoder here, right?

00:03:26 Which is, you know, after the encoder.

00:03:28 The encoder is sort of, you know, mapping the input of the image here into a hidden state.

00:03:34 And then output is, you know, sorry, the decoder is taking the hidden state or the output of the encoder,

00:03:41 and then eventually give the output.

00:03:43 In this case, the desired output is the sequence, okay?

00:03:47 Again, the encoder is taking these feature maps, z, and then get a hidden state.

00:03:54 Initially, there's a h0 at time zero, initial, okay?

00:03:58 Initial time.

00:03:59 Because why is there a time?

00:04:01 Because my output is a sequence, okay?

00:04:03 Okay, I haven't given you the output or the decoder structure yet, but let's do it one by one.

00:04:09 So the decoder is trying to get this character or word one by one, right?

00:04:16 So for instance, in this case, the output should be person wearing a hat, right?

00:04:23 So y1 should be person, okay?

00:04:27 And the way you get the y1 is, you know, using the hidden state calculated at h1, right?

00:04:34 This is the decoder hidden state, right?

00:04:37 And the way you calculate the decoder hidden state is using context C and the current state, I'm sorry, and the current and initial, in this case, the last time point, zero.

00:04:51 So y0.

00:04:52 So in our case, it's just a starting token, okay?

00:04:56 And we can keep doing that, right?

00:04:58 So the next thing is, right?

00:05:00 So you are, you know, this y1 is just a person, right?

00:05:04 And then, you know, the desired output, right?

00:05:08 Using the context vector, and the y1 is y2 that we want to say wearing.

00:05:15 And in the end, you have person wearing hat, right?

00:05:19 And then the corresponding, you know, in this case, the state is y0, y1, y2, right?

00:05:25 Start code on, sorry, starting, not code on, token, and then person, and then wearing, right?

00:05:32 Is this clear how image captioning is doing?

00:05:35 You have to be very clear about why the decoder looks like this, okay?

00:05:40 Is this clear?

00:05:41 Again, at the beginning, you only have a starting token, and then you are going to rely on the context, right?

00:05:49 But also keep in mind that these two will be combined with the current hidden state of the decoder, which is a function of the initial hidden state at time zero, okay?

00:06:04 And the initial hidden state at time zero can be the end of the encoder, okay?

00:06:13 So what could be a problem with this, right?

00:06:16 The problem is, you know, and this is the end, right?

00:06:22 And then what would be the problem?

00:06:24 The problem is here, okay?

00:06:28 This C, you can regard it as a context, and this context can be regarded as like a summary of the feature map through this hidden state, right?

00:06:38 For instance, H at time T, capital T, in the encoder, right?

00:06:43 So this C is kind of fixed for a different time, right?

00:06:49 You see that when we're generating this sequence output, at different time I'm using the same C.

00:06:55 This never changes.

00:06:56 This is one problem.

00:06:58 Another problem is that when you have a very long or large image in this case, or very long, you know, description in this case, right?

00:07:08 And then your C is going to be, you know, for instance, a very big summary is going to be a summary, a vector, right?

00:07:17 Of a very large image that's going to apply to all the long sentences, right?

00:07:22 All the words of this long sentence.

00:07:24 So that is a problem, right?

00:07:26 So the bottleneck is C, right?

00:07:28 So you want to make this context, right?

00:07:32 A little bit, what?

00:07:34 What should I say?

00:07:36 Knowing where to look at in the feature map and knowing how to adapt that information for different time T.

00:07:47 So that's my goal.

00:07:49 Is that clear?

00:07:50 Okay, so that's what I want to do next, using the so-called attentions.

00:07:55 So this is actually very similar to how our vision systems work because if you think about how we are looking at things, right?

00:08:05 So, you know, you have a beam of light, right?

00:08:08 Going into your eyes and then on the back side of the eye there's a retina, right?

00:08:14 But if you look at the retina, there's a range, right?

00:08:18 But only in one place it's very sensitive.

00:08:22 One small region.

00:08:23 It's very sensitive to the, you know, the photons or the light signals, right?

00:08:30 So that's why, you know, our eyes actually are most sensitive to things ahead of us, right ahead of us, right?

00:08:37 If it's something like more than zero degrees, maybe 10 or 20 degrees, then we have less vision about those objects.

00:08:46 Is that right?

00:08:47 Right?

00:08:48 But how do we make sure that we can, you know, figure out what's going on in front of us?

00:08:52 Because the world is not like lying on a line like zero degrees perpendicular to our eyes, right?

00:08:59 So what do we do?

00:09:02 We actually roll our eyes and turn our necks a lot, right?

00:09:08 Right?

00:09:09 Even in a very short time, right?

00:09:10 Even, even, you know, this is called a saccade.

00:09:13 It's like high frequency motions that our eyes keep rolling even by looking at some objects you think is fixed and static in front of us.

00:09:22 We actually roll our eyes very quickly and then essentially we are basically using that motion to attend, right?

00:09:30 Attend to different parts of the feature map in front of us.

00:09:35 Is that right?

00:09:36 So our eyes are doing that as well.

00:09:38 So can we do something similar?

00:09:41 Can we make a context vector depending on time t?

00:09:45 And that thing should attend, right?

00:09:48 Time dependent context vector should attend to different portions of the input images or the feature maps.

00:09:57 Is that right?

00:09:58 Right?

00:09:59 Naturally, when you are generating the first sentence, person, you are paying attention to the person, right?

00:10:06 Wearing a hat is where you pay attention to here.

00:10:09 So, right?

00:10:10 So naturally, we can introduce attention mechanism.

00:10:13 This is basically saying what?

00:10:15 So, when we look at the first hidden state or initial hidden state, right, to the decoder, H0, right?

00:10:24 We can look at how much this aligns to the feature map, Z, right?

00:10:31 So imagine that there's an alignment score, right?

00:10:35 Showing the alignment between the feature map and H0, right?

00:10:40 So this shows how much this H0 should attend to different portions of the feature maps.

00:10:47 Is that right?

00:10:48 Okay.

00:10:49 In reality, this function alignment is, you know, represented as FATT, which is parametrized by MLP, for instance.

00:11:00 Okay.

00:11:01 So, it's not the ultimate or the only choice, but currently think of it as a learnable MLP.

00:11:08 But these scores, right, can take any values, right?

00:11:11 So I would like to normalize them, right, using softmax, right?

00:11:16 So that I have all these values between 0 and 1 and sum up to 1 for any given time t, right?

00:11:24 So now this is like attention I paid to for each part of the feature map.

00:11:30 Is that right?

00:11:31 Right?

00:11:32 So this is what the attention mechanism is.

00:11:34 It's just actually the weights, okay?

00:11:36 And in that case, now when I calculate the context, right, for instance, at time 1, I can actually use the weighted average of the feature map, right?

00:11:47 So by doing this, you know, element-wise product and sum, right?

00:11:54 So it's weighted average, and I get the first context, C1, right?

00:11:59 Okay.

00:12:00 So I can, right, that's what I wrote about.

00:12:02 And then you can do something similar, right?

00:12:05 And then before we go to the C2, right?

00:12:08 So your C1, as I said, and Y0, in this case, the start token, will be fed to the decoder's first hidden unit.

00:12:19 And then the H1, right, will be, you know, outputting this Y1.

00:12:27 Eventually we won't map this to person as much as possible, okay?

00:12:31 Again, here we have C1, which is determined by H0 and the feature, right?

00:12:38 Right?

00:12:39 In between, what we did was alignment and softmax, remember?

00:12:42 So that's what these arrows mean.

00:12:47 Then let me finish this attention mechanism, right?

00:12:50 What about C2, right?

00:12:51 For C2, we do something similar, but you are not using H0 anymore, right?

00:12:56 Right?

00:12:57 You're gonna try to see how H1 is paying attention to these feature maps.

00:13:04 Is that right?

00:13:05 So now we, you know, look at, to get C2, right?

00:13:09 To get C2, again, we are looking at how this H1 attends to different parts of the feature map, right?

00:13:19 And similarly, the new C2, together with H1, right, as well as Y1, will give us the second unit of the decoder, whose output should eventually map to where in, right?

00:13:39 And then we keep going forward.

00:13:42 So this is the attention mechanism, right?

00:13:45 So this entire process is differentiable.

00:13:51 Back propagation is very easy because all we did was using a lot of the matrix product, right?

00:13:56 Linear algebra.

00:13:57 And the model choose the best attention ways by learning to minimize the loss function.

00:14:03 And there is no need for you to determine that, right?

00:14:07 And this is one example, right, to show you the benefit of attention.

00:14:14 Because attention is not only for accuracy, right?

00:14:17 It's also helping the interpretability.

00:14:20 Something we want to talk about is interpretable machine learning.

00:14:24 If you look at it in this case, right, so we have this image, right?

00:14:29 And we are doing the captioning, the output should be a bird flying over a body of water, right?

00:14:37 Every word corresponding to one time point T, right?

00:14:42 T minus one, T, T plus one, and so on.

00:14:44 At every time T, we do have the attention map, right?

00:14:48 So in the top row, we have the soft attentions like we showed before using softmax, right?

00:14:54 And if you look at the brighter pixels are showing where, you know, the higher attentions were paid.

00:15:00 You see sort of a bird flying over a body.

00:15:03 You see sort of where attentions were paid more and more, right?

00:15:07 When it's bird, you see more attentions on the bird.

00:15:11 When it's water, it's more attentions outside of the bird, right?

00:15:15 And that's what the soft attention is doing, the softmax we are using.

00:15:21 But instead of using the softmax, you are using, for instance, the maximum attention for the output eventually.

00:15:30 In that case, you create a more crispy, right, more crisp and focused attention that's called a hard attention, right?

00:15:39 Of course, there are some maths you have to introduce to make that happen.

00:15:42 And in this case, you see even more focused.

00:15:45 I think this is probably similar to lasso, right?

00:15:49 When you introduce the sparsity, right?

00:15:51 So lasso tries to make this feature selection more sparse, right?

00:15:55 This is like the lasso analog, right?

00:15:59 When you introduce the hard attention to the soft attention.

00:16:01 So I should stop right here and the next time we continue here.

00:16:08 Next time we'll talk about, you know, still attention.

00:16:12 And I'll give you another example, not in computer vision, but in NLP, that is a machine translation, okay?

00:16:19 Starting from the attention, right?

00:16:22 We can introduce attention to RNNs easily, right?

00:16:26 And that's exactly what is happening here, right?

00:16:29 Using an encoder-decoder architecture.

00:16:32 But on top of that, what can we generalize, right?

00:16:36 What can we generalize and make the model much more expressive?

00:16:40 So that's why we delve into that and get more general attention mechanism, more general.

00:16:48 And then we're going to start with that general attention mechanism and land on something special again.

00:16:55 And that's called self-attention.

00:16:57 And that's what transformers have been using, including GPT or BERT and so on,

00:17:03 and including many of our current protein language models, okay?

00:17:08 [Uncertain recognition] Well, again.