Lecture 20.1: Recurrent Neural Networks
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 Now I think it's time to think a little bit deeper about neural networks.
00:00:06 We also showed you before that there are all kinds of neural networks actually.
00:00:12 We paid a lot of attention last time to
00:00:17 forward neural networks, right?
00:00:21 In these cases, you know, it's like the vanilla neural network. You see these kind of
00:00:26 building blocks that's passed
00:00:30 sequentially, where the information is passed across
00:00:33 sequentially, right?
00:00:35 But there are other possibilities as well, right? So like what is
00:00:40 shown here.
00:00:43 So the first example is like here, right? You have this
00:00:46 input, right?
00:00:48 There's the
00:00:49 let me call it a hidden layer for the moment, but these are the hidden blocks, right?
00:00:53 And there is actually a sequential information passed across them.
00:00:58 And they have the corresponding output, right? So you may wonder what do you do with this one, right?
00:01:06 Well, this is one example for instance for image captioning. You have input image here
00:01:13 but the output is a sentence that
00:01:17 captures the image, right? So this is a one-to-many kind of
00:01:22 architecture, right?
00:01:24 We also have many-to-one in this case.
00:01:30 Any guess or
00:01:34 what kind of applications? Can you name one?
00:01:37 For many-to-one?
00:01:41 Any volunteers?
00:01:44 One application where the many-to-one
00:01:48 architecture may make sense.
00:01:50 One simple example is just reversing image captioning. Start with the sentence and the synthesize an image, right?
00:02:04 A man walking his dog
00:02:08 along the river and there's, you know, people sailing in the river.
00:02:14 So that's a sentence, but I can, you know, synthesize the image accordingly, right?
00:02:20 Do you want to give another example?
00:02:28 Grouping?
00:02:29 I see. I see.
00:02:31 We got to pay attention that you sort of like, you know, you have the sequence information here.
00:02:41 That is in the nature of the input or output. Then accordingly in the intermediate layers, we use the sequence structure as well.
00:02:53 Right? So I'm not sure that the example you gave has the sequence structure in the input for this many-to-one, okay?
00:03:05 The example that was given was sentiment classification.
00:03:12 So you can look at others' tweets and read a sentence and figure out what is the sentiment.
00:03:19 Are they, you know, happy or angry or frustrated and so on, right? So this is a, like, a sentiment classification.
00:03:26 Because in the end, it's like a categorical label, right? But the input has a sequence of words.
00:03:32 So this is like a sentiment classification.
00:03:34 [Uncertain recognition] Right?
00:03:35 [Uncertain recognition] People use food на 얼بيomp SiNoviii ang processing .
00:03:38 [Uncertain recognition] Veryym地 Huizar за ней coin ,ste исп parler ==
00:03:41 [Uncertain recognition] Veryym地 Huizar vuuttimi tas чис că förUL inn cu güいや颅公за Wereyd氏
00:03:46 [Uncertain recognition] Will hilarious 55 00 tracer Soonmark cri audit
00:03:51 [Uncertain recognition] Many to many .
00:03:53 [Uncertain recognition] That could utilize many to many kind of mean cut mostly .
00:03:59 So that means both the input and the output
00:04:03 have some sequence information.
00:04:05 Can you give one example?
00:04:14 For instance, machine translation.
00:04:17 Start with a sequence of words, and you get a sequence of words.
00:04:21 You may pay attention that you see this input and output are not
00:04:24 really aligned to the same hidden blocks.
00:04:27 We'll talk more about it.
00:04:30 And then there is this kind of many-to-many where they are kind
00:04:32 of aligned.
00:04:33 So if I gave you an example, for instance,
00:04:36 the video classification on the frame level.
00:04:38 Each frame, I give a categorical label for this video.
00:04:43 But the label of the previous frame or frames
00:04:48 would affect my decision for the current frame.
00:04:54 So that's the examples for different kind of.
00:04:59 So I think here you can probably already see two keywords.
00:05:05 One is a sequence.
00:05:07 There's a sequence information.
00:05:09 Either demanded in the input or the output.
00:05:12 And it's actually reflected in architecture.
00:05:14 Right?
00:05:15 The second one is recurrent.
00:05:19 So we'll pay attention to these two keywords.
00:05:22 OK?
00:05:24 So actually, this idea is not something that we haven't learned.
00:05:30 We already learned it.
00:05:32 We just learned from the CNN that it's utilizing some mathematical operators that
00:05:40 actually are not just economic, but also consistent to how our vision systems work.
00:05:47 Right?
00:05:48 Including this localness, including this translational invariance, and so on.
00:05:55 So we look at the convolution at the end because it really gives us what we want.
00:05:59 But keep in mind that using this convolution kernel, it's a very small, right?
00:06:05 3 by 3, 4 by 4, much, much smaller than the image size.
00:06:11 And using these convolutions, we can make this applicable to variable sizes of the images,
00:06:19 and the scalabilities of the large images, and so on.
00:06:23 And we have sparse connections.
00:06:26 Right?
00:06:27 What I mean by sparse connection, do you remember a figure I showed before to illustrate the concept
00:06:32 of the receptive field?
00:06:35 I was saying that if your kernel size was 3 by 3, right, if you look at the first hidden
00:06:41 layer, one neuron there basically received input from three input neurons.
00:06:51 Right?
00:06:52 So the receptive field is actually very wide given the depth.
00:06:57 So, and we accomplish that through just a very sparse connection, just a local, right?
00:06:58 And so, in the second hidden layer, the second hidden layer, each neuron there receives input
00:07:04 from three previous hidden layers neurons, and each of the three receives, again, you know,
00:07:10 three of the input neurons.
00:07:13 So the receptive field is actually very wide given the depth.
00:07:18 So, and we accomplish that through just a very sparse connection, just a local, right?
00:07:24 Three, for instance.
00:07:25 And we use the parameter sharing, right?
00:07:29 So, we can probably consider the same idea, but now not for the grid data, but for the image
00:07:38 data, right?
00:07:39 Can we, can we use some similar ideas for sequence data?
00:07:44 Because we desire the similar applicability and scalability for the sequence data, right?
00:07:50 We also probably, you know, would like to use a parameter sharing idea as well.
00:07:56 Can we borrow similar ideas?
00:07:58 In the convolutional neural networks, we have the convolutions, right?
00:08:01 So here, can we, you know, the convolutional kernels were, the same kernel was, you know, sliding
00:08:08 down the grid, right?
00:08:11 The two-dimensional grid, right?
00:08:12 Can we borrow similar ideas?
00:08:15 Well, the keyword is recurrent, okay?
00:08:22 So, so it's a recurrence formula.
00:08:24 So, it's like the following, okay?
00:08:27 So, let's consider a sequence, right?
00:08:32 Where each position is indexed, right?
00:08:35 Right?
00:08:36 And this index, we can call it a time, t, right?
00:08:39 This could be a position in the sequence.
00:08:43 Let's call it x of t, right?
00:08:45 And let's imagine that when t equal to one, I have h of zero that is initialized somehow, right?
00:08:53 Let's call it an old state because t is the present and t minus one is the, the past, the
00:09:04 immediate past, right?
00:09:05 So, that's an old state.
00:09:08 And then, and then to give the new state, right?
00:09:13 For time t, to calculate the new state h sub t, I'm basically using the old state and the current input vector x sub t.
00:09:24 And I use some function which is parameterized by w.
00:09:29 So, the recurrence is reflected by the fact that I am using the same w for all the time t.
00:09:40 So, that's the recursive.
00:09:41 So, that's the recursive.
00:09:42 You can think of what h of t plus one equals to.
00:09:45 That equals to f sub w of f sub w, right?
00:09:51 Of this one, then comma, x t plus one, right?
00:09:56 In that sense, it's recursive.
00:09:59 Did I get what I mean?
00:10:01 So, the same f of w, f sub w.
00:10:16 Okay?
00:10:17 So, in this sense, it's recursive.
00:10:34 We are using this f w recursively.
00:10:37 We are composing f w.
00:10:39 So, and we are reusing this f w.
00:10:42 Okay?
00:10:43 So, it's a recursive formula.
00:10:44 So, I was asking, you know, how do we share the parameters, right?
00:10:49 So, that's how we share the parameters.
00:10:51 Okay?
00:10:52 Okay.
00:10:53 Let's see.
00:10:54 Okay, good.
00:10:55 All right.
00:10:56 So, if in the convolutional neural network, we use the same small convolutional kernel,
00:11:05 right, over the 2D, 3D spatial grids, here we have a sequence.
00:11:13 So, there is a concept of time.
00:11:15 So, we are using the same formula recurrently over time.
00:11:21 Okay?
00:11:22 So, if we want to visualize this concept, right, the bottom is input x and the top is output
00:11:32 y.
00:11:33 But, you know, you can imagine that this is x t plus, this is x sub t and then this is y
00:11:42 is like h sub t, right?
00:11:44 Then this new h sub t is going to be fed back to this r n, right, with a new x t plus 1.
00:11:52 And then you get the new h t plus 1 and so on.
00:11:55 So, it's like a cycle here, right?
00:11:58 So, I hope you understand that the reason I'm using a cycle to illustrate is with the recurrence
00:12:05 formula mathematically underlying.
00:12:07 So, that's the reason.
00:12:09 Okay?
00:12:10 So, we want to make sure that we understand both the mathematics and the intuition.
00:12:15 Okay?
00:12:16 So, this is how we get the parameter sharing, right?
00:12:19 So, now let's talk about what is the f of f sub w, right?
00:12:24 We have some examples, right?
00:12:25 So, one example is like, you know, this, you know, weighted sum and then hyperbolic tangent.
00:12:34 This is one idea, right?
00:12:36 So, this is sometimes called a vanilla r n, right?
00:12:40 And the y is simply like a linear readout of h t, which is h is for the hidden, right?
00:12:47 Hidden vector, right?
00:12:48 This is a very vanilla idea, right?
00:12:51 So, now let's look at another way, right?
00:12:54 Previously, I showed you that this is what I showed you is like a recurrent diagram, right?
00:13:01 Because I have these cycles, right?
00:13:04 But here, I'm not looking at a time index t anywhere.
00:13:09 I'm just, you know, implicitly I'm telling you that the previous, you know, will be fed for the next state.
00:13:15 This is this, there's a time concept in it, right?
00:13:18 So, in other words, this diagram was not really shown in the time, right?
00:13:25 So, in that sense, I'm saying that it's, you know, it's not unfolded.
00:13:32 It's already folded.
00:13:33 There's, you know, there's, if you think about the time axis, right?
00:13:37 It's like already collapsed, right?
00:13:39 I'm not showing or extending it in the time domain.
00:13:44 So, now let's imagine that we can unfold it or unroll this into like a time domain.
00:13:52 Now we can have this superscript t for the time index now, right?
00:13:58 So, now we're not using a single x and single h.
00:14:01 We're actually looking at x and h at different time point.
00:14:05 So, this is like unfold or unroll, okay?
00:14:09 And here, then we, again, our input is x, x, x, and then this is output.
00:14:15 Actually, this is the hidden state.
00:14:17 And there's no output read out yet.
00:14:19 So, this figure, this unfolded diagram, is what I showed you at the beginning.
00:14:24 Right? Right?
00:14:28 So, that's what this very simple illustration is about, okay?
00:14:34 So, now you know that, you know, you could have the math, you have the, you know, a collapsed view, and you have the unfolded view.
00:14:41 Okay?
00:14:42 So, this is the R-none.
00:14:44 And the read out could be just here, or it could be multiple, right?
00:14:49 Like the examples I just showed you.
00:14:51 And now, let's remind ourselves, we learned in the last lecture, the computation graph, right?
00:15:00 For both forward and backward propagation.
00:15:03 So, what's the computational graph?
00:15:05 Well, if you look at this one, we start with x1 and with a hidden state at time 0.
00:15:13 We put, you know, both of them into this fw, and then we get h1, right?
00:15:19 And we keep doing this the same, right?
00:15:23 Right?
00:15:24 So, this x are entering the same recurrent formula, right?
00:15:28 The fw.
00:15:29 And the w is actually shared for all of them.
00:15:33 All of these building blocks, okay?
00:15:36 So, we reuse the same weight matrix at every time step.
00:15:40 This is the forward propagation for computational graph, okay?
00:15:46 So, now if I add the read out, right?
00:15:49 Here, there's no read out yet, right?
00:15:51 If I add the read out, as I was showing you earlier, right?
00:15:55 I could have y1, y2, y3, y4, and so on read out like here, right?
00:16:02 For instance, machine translation is one example, right?
00:16:06 I read in English and I want to translate into French, right?
00:16:10 So, this is the example I showed you earlier, right?
00:16:14 And that's a many-to-many.
00:16:16 I wanted to give you a more concrete idea about this many-to-many, okay?
00:16:22 So, let's talk about this thing, right?
00:16:27 This is a language model, but it's only at the character level,
00:16:32 meaning that the tokens are characters, okay?
00:16:35 Or the building blocks of the sentence is regarded as the characters, right?
00:16:41 I'm not considering words for the moment, okay?
00:16:44 It's a very simple character-level language model.
00:16:47 And I only have four letters in the vocabulary here, H-E-L-O, right?
00:16:56 And then the training sequence here is H-E-L-L-O, right?
00:17:02 Okay, so this model is auto-regressive model.
00:17:06 I will show you what it means, right?
00:17:08 So, the input is H-E-L-L, and they are one-hot encoded as one, one, one, one here.
00:17:19 So, it's like the vocabulary is only four digits, so it's one-hot encoded in this way, okay?
00:17:27 And then, based on the vanilla RNN, we can calculate the hidden state for every time T, right?
00:17:39 And then, you know, we probably get some numbers here, right?
00:17:42 Given your W, which is to be learned.
00:17:45 And then, given the hidden state, right?
00:17:50 And, you know, I'm going to get some output, right?
00:17:55 And this output is going to be mapped or categorized into one of the letters in the vocabulary, like H-E-L-O.
00:18:05 And what I'm trying to predict here, the true labels are E-L-L-O.
00:18:11 So, for that, I need to learn the W for X and W for H, so, and W-H-H.
00:18:20 I need to learn these Ws so that these numbers eventually, you know, crunch to the optimum loss function that can reproduce E-L-L-O.
00:18:32 So, what is this model trained to do?
00:18:43 What is this model trained to do?
00:18:46 Minimize the difference between the output and the input.
00:18:50 Minimize the difference between the input and output.
00:18:54 And what is the structure correspondence between the input and output here?
00:19:02 [Uncertain recognition] Minimize the Maunaindo speaker using L ele stabbing idea because one could represent where E-L-L-O under multiple threads through ways from Ad Telepatinosaur function or parallel Segjam谢
00:19:16 [Uncertain recognition] دم
00:19:32 The technical word is called autoregressive,
00:19:36 meaning that the previous state is gonna be,
00:19:43 if you have a new prediction, right?
00:19:46 So for instance, if you have a new prediction,
00:19:51 at position t, that prediction is gonna be utilized,
00:19:58 fed back to the model,
00:19:59 and then get the next one t plus one.
00:20:02 So in other words, you're predicting p of x sub t,
00:20:07 given x one, x two, until x t minus one.
00:20:13 So you keep predicting x sub t recurrently in this way.
00:20:20 One application you can think of,
00:20:22 by modeling these probabilities,
00:20:25 so this is actually like a factorization.
00:20:29 It's essentially what you wanna model for this language model
00:20:33 is the joint probability of x one, x two, until x sub t, right?
00:20:39 But you can factorize them to p of x one times p of x two,
00:20:43 given x one times p of x three, given x one and x two,
00:20:47 until all the way to p of x t,
00:20:49 given x one through x t minus one.
00:20:52 You can factorize them, right?
00:20:54 So this kind of factorization is what auto regressive is about.
00:20:59 And one application can be something you encounter every day.
00:21:03 Used to be when you are Googling something,
00:21:05 but now when you are writing an email in Gmail,
00:21:09 you also get some reminders for auto completion, right?
00:21:12 And this is one suitable application for that, right?
00:21:16 But in general, for this language modeling, right?
00:21:19 It's for the language generation, right?
00:21:24 Okay, so,
00:21:27 and then this is a computational graph to many to many, right?
00:21:32 So when we have this output, y one, y two, one, three,
00:21:36 and so on, to y t, capital T is the largest capital T.
00:21:39 Again, compared to the previous computational graph
00:21:42 for the forward propagation,
00:21:44 we have the additional readouts, right?
00:21:48 And be careful, right?
00:21:50 It depends, right?
00:21:52 It depends on how this y,
00:21:54 remember, the forward propagation is all the way
00:21:58 from x and w to j, the loss, right?
00:22:02 The forward propagation is to compute the loss.
00:22:07 The forward propagation is to calculate the gradient
00:22:12 of the loss with regard to different w's
00:22:15 at different layers, right?
00:22:17 So if we look at the forward propagation here,
00:22:20 we have to pay attention to how the y's, right?
00:22:24 The many outputs map to the losses, okay?
00:22:28 It is possible that you have the loss at every time t, right?
00:22:33 Every time lowercase t,
00:22:35 and this loss is probably by comparing the corresponding y i's
00:22:40 and x i's or other kind of ways, right?
00:22:42 So as the answer before that, you know,
00:22:49 you're trying to maximize the correspondence
00:22:53 between the input output.
00:22:54 So the one possibility is that the character
00:22:56 to character level, right?
00:22:57 There's L1, L2, and so on.
00:23:00 And then the loss is the sum over all the characters, right?
00:23:05 So that's the complete forward propagation.
00:23:10 The computational graph for the forward propagation, okay?
00:23:13 Remember, it goes all the way to L.
00:23:15 That's what the forward propagation is about.
00:23:19 And if it's many to one, right, in this case,
00:23:22 maybe you are just trying to figure out the sentiment
00:23:25 behind this sequence, then the final readout is probably,
00:23:31 you know, figured out or mapped from the last hidden state,
00:23:35 H sub t, and then this is your y.
00:23:38 And there's also possibilities that you have one too many,
00:23:43 like you have the image,
00:23:44 and you want to caption the image
00:23:47 with a sequence of characters or words.
00:23:51 And in that case, you have these sequence introduced
00:23:55 in the hidden layers, then they correspond to the words,
00:23:59 the order, or the characters in the desired output.
00:24:04 And, you know, and let's put this side to side again, right?
00:24:09 Side to side again, right, one is many to one,
00:24:13 and one is one to many, right?
00:24:16 So many to one in this case,
00:24:18 you can encode this sequence, right,
00:24:23 into a single vector eventually, right?
00:24:25 Maybe you just read out somewhere here, right?
00:24:30 Just like the example I gave you,
00:24:32 that predicted y, that's for the sentiment prediction.
00:24:36 And in the case of the one to many,
00:24:39 you're producing the output sequence
00:24:41 from a single input vector.
00:24:43 For instance, this is the input is, excuse me,
00:24:46 the single input, where is my x?
00:24:50 I think x should be, where is h zero?
00:24:55 I think x, x one should be fed here, is that right?
00:25:00 Right, there should be x one fed here, right?
00:25:04 There should be an x fed here, right?
00:25:06 And that could be an image, and an output is a sequence, okay?
00:25:14 Okay, let's see whether there is any money points.
00:25:16 So, so far we talked about how RNNs can be utilized
00:25:21 in different ways,
00:25:23 and their corresponding computational graphs.
00:25:26 Okay, again, mapping from the input
00:25:28 all the way to the loss.
00:25:31 But this is a computational graph for forward propagation.
00:25:35 Now we have learned so much from the last lecture that
00:25:38 we also need to do the computation.
00:25:41 That means we need to use gradients, right?
00:25:45 To optimize the loss function.
00:25:48 And for that purpose,
00:25:49 we need to do the back propagation
00:25:51 to calculate the gradients of j with regard to different w's,
00:25:55 using the chain rule, right?
00:25:57 So,