# Lecture 20.1: Recurrent Neural Networks

[Recording](https://www.youtube.com/watch?v=8SgXLPGSuLI)

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Now I think it's time to think a little bit deeper about neural networks.

00:00:06 We also showed you before that there are all kinds of neural networks actually.

00:00:12 We paid a lot of attention last time to

00:00:17 forward neural networks, right?

00:00:21 In these cases, you know, it's like the vanilla neural network. You see these kind of

00:00:26 building blocks that's passed

00:00:30 sequentially, where the information is passed across

00:00:33 sequentially, right?

00:00:35 But there are other possibilities as well, right? So like what is

00:00:40 shown here.

00:00:43 So the first example is like here, right? You have this

00:00:46 input, right?

00:00:48 There's the

00:00:49 let me call it a hidden layer for the moment, but these are the hidden blocks, right?

00:00:53 And there is actually a sequential information passed across them.

00:00:58 And they have the corresponding output, right? So you may wonder what do you do with this one, right?

00:01:06 Well, this is one example for instance for image captioning. You have input image here

00:01:13 but the output is a sentence that

00:01:17 captures the image, right? So this is a one-to-many kind of

00:01:22 architecture, right?

00:01:24 We also have many-to-one in this case.

00:01:30 Any guess or

00:01:34 what kind of applications? Can you name one?

00:01:37 For many-to-one?

00:01:41 Any volunteers?

00:01:44 One application where the many-to-one

00:01:48 architecture may make sense.

00:01:50 One simple example is just reversing image captioning. Start with the sentence and the synthesize an image, right?

00:02:04 A man walking his dog

00:02:08 along the river and there's, you know, people sailing in the river.

00:02:14 So that's a sentence, but I can, you know, synthesize the image accordingly, right?

00:02:20 Do you want to give another example?

00:02:28 Grouping?

00:02:29 I see. I see.

00:02:31 We got to pay attention that you sort of like, you know, you have the sequence information here.

00:02:41 That is in the nature of the input or output. Then accordingly in the intermediate layers, we use the sequence structure as well.

00:02:53 Right? So I'm not sure that the example you gave has the sequence structure in the input for this many-to-one, okay?

00:03:05 The example that was given was sentiment classification.

00:03:12 So you can look at others' tweets and read a sentence and figure out what is the sentiment.

00:03:19 Are they, you know, happy or angry or frustrated and so on, right? So this is a, like, a sentiment classification.

00:03:26 Because in the end, it's like a categorical label, right? But the input has a sequence of words.

00:03:32 So this is like a sentiment classification.

00:03:34 **[Uncertain recognition]** Right?

00:03:35 **[Uncertain recognition]** People use food на 얼بيomp SiNoviii ang processing .

00:03:38 **[Uncertain recognition]** Veryym地 Huizar за ней coin ,ste исп parler ==

00:03:41 **[Uncertain recognition]** Veryym地 Huizar vuuttimi tas чис că förUL inn cu güいや颅公за Wereyd氏

00:03:46 **[Uncertain recognition]** Will hilarious 55 00 tracer Soonmark cri audit

00:03:51 **[Uncertain recognition]** Many to many .

00:03:53 **[Uncertain recognition]** That could utilize many to many kind of mean cut mostly .

00:03:59 So that means both the input and the output

00:04:03 have some sequence information.

00:04:05 Can you give one example?

00:04:14 For instance, machine translation.

00:04:17 Start with a sequence of words, and you get a sequence of words.

00:04:21 You may pay attention that you see this input and output are not

00:04:24 really aligned to the same hidden blocks.

00:04:27 We'll talk more about it.

00:04:30 And then there is this kind of many-to-many where they are kind

00:04:32 of aligned.

00:04:33 So if I gave you an example, for instance,

00:04:36 the video classification on the frame level.

00:04:38 Each frame, I give a categorical label for this video.

00:04:43 But the label of the previous frame or frames

00:04:48 would affect my decision for the current frame.

00:04:54 So that's the examples for different kind of.

00:04:59 So I think here you can probably already see two keywords.

00:05:05 One is a sequence.

00:05:07 There's a sequence information.

00:05:09 Either demanded in the input or the output.

00:05:12 And it's actually reflected in architecture.

00:05:14 Right?

00:05:15 The second one is recurrent.

00:05:19 So we'll pay attention to these two keywords.

00:05:22 OK?

00:05:24 So actually, this idea is not something that we haven't learned.

00:05:30 We already learned it.

00:05:32 We just learned from the CNN that it's utilizing some mathematical operators that

00:05:40 actually are not just economic, but also consistent to how our vision systems work.

00:05:47 Right?

00:05:48 Including this localness, including this translational invariance, and so on.

00:05:55 So we look at the convolution at the end because it really gives us what we want.

00:05:59 But keep in mind that using this convolution kernel, it's a very small, right?

00:06:05 3 by 3, 4 by 4, much, much smaller than the image size.

00:06:11 And using these convolutions, we can make this applicable to variable sizes of the images,

00:06:19 and the scalabilities of the large images, and so on.

00:06:23 And we have sparse connections.

00:06:26 Right?

00:06:27 What I mean by sparse connection, do you remember a figure I showed before to illustrate the concept

00:06:32 of the receptive field?

00:06:35 I was saying that if your kernel size was 3 by 3, right, if you look at the first hidden

00:06:41 layer, one neuron there basically received input from three input neurons.

00:06:51 Right?

00:06:52 So the receptive field is actually very wide given the depth.

00:06:57 So, and we accomplish that through just a very sparse connection, just a local, right?

00:06:58 And so, in the second hidden layer, the second hidden layer, each neuron there receives input

00:07:04 from three previous hidden layers neurons, and each of the three receives, again, you know,

00:07:10 three of the input neurons.

00:07:13 So the receptive field is actually very wide given the depth.

00:07:18 So, and we accomplish that through just a very sparse connection, just a local, right?

00:07:24 Three, for instance.

00:07:25 And we use the parameter sharing, right?

00:07:29 So, we can probably consider the same idea, but now not for the grid data, but for the image

00:07:38 data, right?

00:07:39 Can we, can we use some similar ideas for sequence data?

00:07:44 Because we desire the similar applicability and scalability for the sequence data, right?

00:07:50 We also probably, you know, would like to use a parameter sharing idea as well.

00:07:56 Can we borrow similar ideas?

00:07:58 In the convolutional neural networks, we have the convolutions, right?

00:08:01 So here, can we, you know, the convolutional kernels were, the same kernel was, you know, sliding

00:08:08 down the grid, right?

00:08:11 The two-dimensional grid, right?

00:08:12 Can we borrow similar ideas?

00:08:15 Well, the keyword is recurrent, okay?

00:08:22 So, so it's a recurrence formula.

00:08:24 So, it's like the following, okay?

00:08:27 So, let's consider a sequence, right?

00:08:32 Where each position is indexed, right?

00:08:35 Right?

00:08:36 And this index, we can call it a time, t, right?

00:08:39 This could be a position in the sequence.

00:08:43 Let's call it x of t, right?

00:08:45 And let's imagine that when t equal to one, I have h of zero that is initialized somehow, right?

00:08:53 Let's call it an old state because t is the present and t minus one is the, the past, the

00:09:04 immediate past, right?

00:09:05 So, that's an old state.

00:09:08 And then, and then to give the new state, right?

00:09:13 For time t, to calculate the new state h sub t, I'm basically using the old state and the current input vector x sub t.

00:09:24 And I use some function which is parameterized by w.

00:09:29 So, the recurrence is reflected by the fact that I am using the same w for all the time t.

00:09:40 So, that's the recursive.

00:09:41 So, that's the recursive.

00:09:42 You can think of what h of t plus one equals to.

00:09:45 That equals to f sub w of f sub w, right?

00:09:51 Of this one, then comma, x t plus one, right?

00:09:56 In that sense, it's recursive.

00:09:59 Did I get what I mean?

00:10:01 So, the same f of w, f sub w.

00:10:16 Okay?

00:10:17 So, in this sense, it's recursive.

00:10:34 We are using this f w recursively.

00:10:37 We are composing f w.

00:10:39 So, and we are reusing this f w.

00:10:42 Okay?

00:10:43 So, it's a recursive formula.

00:10:44 So, I was asking, you know, how do we share the parameters, right?

00:10:49 So, that's how we share the parameters.

00:10:51 Okay?

00:10:52 Okay.

00:10:53 Let's see.

00:10:54 Okay, good.

00:10:55 All right.

00:10:56 So, if in the convolutional neural network, we use the same small convolutional kernel,

00:11:05 right, over the 2D, 3D spatial grids, here we have a sequence.

00:11:13 So, there is a concept of time.

00:11:15 So, we are using the same formula recurrently over time.

00:11:21 Okay?

00:11:22 So, if we want to visualize this concept, right, the bottom is input x and the top is output

00:11:32 y.

00:11:33 But, you know, you can imagine that this is x t plus, this is x sub t and then this is y

00:11:42 is like h sub t, right?

00:11:44 Then this new h sub t is going to be fed back to this r n, right, with a new x t plus 1.

00:11:52 And then you get the new h t plus 1 and so on.

00:11:55 So, it's like a cycle here, right?

00:11:58 So, I hope you understand that the reason I'm using a cycle to illustrate is with the recurrence

00:12:05 formula mathematically underlying.

00:12:07 So, that's the reason.

00:12:09 Okay?

00:12:10 So, we want to make sure that we understand both the mathematics and the intuition.

00:12:15 Okay?

00:12:16 So, this is how we get the parameter sharing, right?

00:12:19 So, now let's talk about what is the f of f sub w, right?

00:12:24 We have some examples, right?

00:12:25 So, one example is like, you know, this, you know, weighted sum and then hyperbolic tangent.

00:12:34 This is one idea, right?

00:12:36 So, this is sometimes called a vanilla r n, right?

00:12:40 And the y is simply like a linear readout of h t, which is h is for the hidden, right?

00:12:47 Hidden vector, right?

00:12:48 This is a very vanilla idea, right?

00:12:51 So, now let's look at another way, right?

00:12:54 Previously, I showed you that this is what I showed you is like a recurrent diagram, right?

00:13:01 Because I have these cycles, right?

00:13:04 But here, I'm not looking at a time index t anywhere.

00:13:09 I'm just, you know, implicitly I'm telling you that the previous, you know, will be fed for the next state.

00:13:15 This is this, there's a time concept in it, right?

00:13:18 So, in other words, this diagram was not really shown in the time, right?

00:13:25 So, in that sense, I'm saying that it's, you know, it's not unfolded.

00:13:32 It's already folded.

00:13:33 There's, you know, there's, if you think about the time axis, right?

00:13:37 It's like already collapsed, right?

00:13:39 I'm not showing or extending it in the time domain.

00:13:44 So, now let's imagine that we can unfold it or unroll this into like a time domain.

00:13:52 Now we can have this superscript t for the time index now, right?

00:13:58 So, now we're not using a single x and single h.

00:14:01 We're actually looking at x and h at different time point.

00:14:05 So, this is like unfold or unroll, okay?

00:14:09 And here, then we, again, our input is x, x, x, and then this is output.

00:14:15 Actually, this is the hidden state.

00:14:17 And there's no output read out yet.

00:14:19 So, this figure, this unfolded diagram, is what I showed you at the beginning.

00:14:24 Right? Right?

00:14:28 So, that's what this very simple illustration is about, okay?

00:14:34 So, now you know that, you know, you could have the math, you have the, you know, a collapsed view, and you have the unfolded view.

00:14:41 Okay?

00:14:42 So, this is the R-none.

00:14:44 And the read out could be just here, or it could be multiple, right?

00:14:49 Like the examples I just showed you.

00:14:51 And now, let's remind ourselves, we learned in the last lecture, the computation graph, right?

00:15:00 For both forward and backward propagation.

00:15:03 So, what's the computational graph?

00:15:05 Well, if you look at this one, we start with x1 and with a hidden state at time 0.

00:15:13 We put, you know, both of them into this fw, and then we get h1, right?

00:15:19 And we keep doing this the same, right?

00:15:23 Right?

00:15:24 So, this x are entering the same recurrent formula, right?

00:15:28 The fw.

00:15:29 And the w is actually shared for all of them.

00:15:33 All of these building blocks, okay?

00:15:36 So, we reuse the same weight matrix at every time step.

00:15:40 This is the forward propagation for computational graph, okay?

00:15:46 So, now if I add the read out, right?

00:15:49 Here, there's no read out yet, right?

00:15:51 If I add the read out, as I was showing you earlier, right?

00:15:55 I could have y1, y2, y3, y4, and so on read out like here, right?

00:16:02 For instance, machine translation is one example, right?

00:16:06 I read in English and I want to translate into French, right?

00:16:10 So, this is the example I showed you earlier, right?

00:16:14 And that's a many-to-many.

00:16:16 I wanted to give you a more concrete idea about this many-to-many, okay?

00:16:22 So, let's talk about this thing, right?

00:16:27 This is a language model, but it's only at the character level,

00:16:32 meaning that the tokens are characters, okay?

00:16:35 Or the building blocks of the sentence is regarded as the characters, right?

00:16:41 I'm not considering words for the moment, okay?

00:16:44 It's a very simple character-level language model.

00:16:47 And I only have four letters in the vocabulary here, H-E-L-O, right?

00:16:56 And then the training sequence here is H-E-L-L-O, right?

00:17:02 Okay, so this model is auto-regressive model.

00:17:06 I will show you what it means, right?

00:17:08 So, the input is H-E-L-L, and they are one-hot encoded as one, one, one, one here.

00:17:19 So, it's like the vocabulary is only four digits, so it's one-hot encoded in this way, okay?

00:17:27 And then, based on the vanilla RNN, we can calculate the hidden state for every time T, right?

00:17:39 And then, you know, we probably get some numbers here, right?

00:17:42 Given your W, which is to be learned.

00:17:45 And then, given the hidden state, right?

00:17:50 And, you know, I'm going to get some output, right?

00:17:55 And this output is going to be mapped or categorized into one of the letters in the vocabulary, like H-E-L-O.

00:18:05 And what I'm trying to predict here, the true labels are E-L-L-O.

00:18:11 So, for that, I need to learn the W for X and W for H, so, and W-H-H.

00:18:20 I need to learn these Ws so that these numbers eventually, you know, crunch to the optimum loss function that can reproduce E-L-L-O.

00:18:32 So, what is this model trained to do?

00:18:43 What is this model trained to do?

00:18:46 Minimize the difference between the output and the input.

00:18:50 Minimize the difference between the input and output.

00:18:54 And what is the structure correspondence between the input and output here?

00:19:02 **[Uncertain recognition]** Minimize the Maunaindo speaker using L ele stabbing idea because one could represent where E-L-L-O under multiple threads through ways from Ad Telepatinosaur function or parallel Segjam谢

00:19:16 **[Uncertain recognition]** دم

00:19:32 The technical word is called autoregressive,

00:19:36 meaning that the previous state is gonna be,

00:19:43 if you have a new prediction, right?

00:19:46 So for instance, if you have a new prediction,

00:19:51 at position t, that prediction is gonna be utilized,

00:19:58 fed back to the model,

00:19:59 and then get the next one t plus one.

00:20:02 So in other words, you're predicting p of x sub t,

00:20:07 given x one, x two, until x t minus one.

00:20:13 So you keep predicting x sub t recurrently in this way.

00:20:20 One application you can think of,

00:20:22 by modeling these probabilities,

00:20:25 so this is actually like a factorization.

00:20:29 It's essentially what you wanna model for this language model

00:20:33 is the joint probability of x one, x two, until x sub t, right?

00:20:39 But you can factorize them to p of x one times p of x two,

00:20:43 given x one times p of x three, given x one and x two,

00:20:47 until all the way to p of x t,

00:20:49 given x one through x t minus one.

00:20:52 You can factorize them, right?

00:20:54 So this kind of factorization is what auto regressive is about.

00:20:59 And one application can be something you encounter every day.

00:21:03 Used to be when you are Googling something,

00:21:05 but now when you are writing an email in Gmail,

00:21:09 you also get some reminders for auto completion, right?

00:21:12 And this is one suitable application for that, right?

00:21:16 But in general, for this language modeling, right?

00:21:19 It's for the language generation, right?

00:21:24 Okay, so,

00:21:27 and then this is a computational graph to many to many, right?

00:21:32 So when we have this output, y one, y two, one, three,

00:21:36 and so on, to y t, capital T is the largest capital T.

00:21:39 Again, compared to the previous computational graph

00:21:42 for the forward propagation,

00:21:44 we have the additional readouts, right?

00:21:48 And be careful, right?

00:21:50 It depends, right?

00:21:52 It depends on how this y,

00:21:54 remember, the forward propagation is all the way

00:21:58 from x and w to j, the loss, right?

00:22:02 The forward propagation is to compute the loss.

00:22:07 The forward propagation is to calculate the gradient

00:22:12 of the loss with regard to different w's

00:22:15 at different layers, right?

00:22:17 So if we look at the forward propagation here,

00:22:20 we have to pay attention to how the y's, right?

00:22:24 The many outputs map to the losses, okay?

00:22:28 It is possible that you have the loss at every time t, right?

00:22:33 Every time lowercase t,

00:22:35 and this loss is probably by comparing the corresponding y i's

00:22:40 and x i's or other kind of ways, right?

00:22:42 So as the answer before that, you know,

00:22:49 you're trying to maximize the correspondence

00:22:53 between the input output.

00:22:54 So the one possibility is that the character

00:22:56 to character level, right?

00:22:57 There's L1, L2, and so on.

00:23:00 And then the loss is the sum over all the characters, right?

00:23:05 So that's the complete forward propagation.

00:23:10 The computational graph for the forward propagation, okay?

00:23:13 Remember, it goes all the way to L.

00:23:15 That's what the forward propagation is about.

00:23:19 And if it's many to one, right, in this case,

00:23:22 maybe you are just trying to figure out the sentiment

00:23:25 behind this sequence, then the final readout is probably,

00:23:31 you know, figured out or mapped from the last hidden state,

00:23:35 H sub t, and then this is your y.

00:23:38 And there's also possibilities that you have one too many,

00:23:43 like you have the image,

00:23:44 and you want to caption the image

00:23:47 with a sequence of characters or words.

00:23:51 And in that case, you have these sequence introduced

00:23:55 in the hidden layers, then they correspond to the words,

00:23:59 the order, or the characters in the desired output.

00:24:04 And, you know, and let's put this side to side again, right?

00:24:09 Side to side again, right, one is many to one,

00:24:13 and one is one to many, right?

00:24:16 So many to one in this case,

00:24:18 you can encode this sequence, right,

00:24:23 into a single vector eventually, right?

00:24:25 Maybe you just read out somewhere here, right?

00:24:30 Just like the example I gave you,

00:24:32 that predicted y, that's for the sentiment prediction.

00:24:36 And in the case of the one to many,

00:24:39 you're producing the output sequence

00:24:41 from a single input vector.

00:24:43 For instance, this is the input is, excuse me,

00:24:46 the single input, where is my x?

00:24:50 I think x should be, where is h zero?

00:24:55 I think x, x one should be fed here, is that right?

00:25:00 Right, there should be x one fed here, right?

00:25:04 There should be an x fed here, right?

00:25:06 And that could be an image, and an output is a sequence, okay?

00:25:14 Okay, let's see whether there is any money points.

00:25:16 So, so far we talked about how RNNs can be utilized

00:25:21 in different ways,

00:25:23 and their corresponding computational graphs.

00:25:26 Okay, again, mapping from the input

00:25:28 all the way to the loss.

00:25:31 But this is a computational graph for forward propagation.

00:25:35 Now we have learned so much from the last lecture that

00:25:38 we also need to do the computation.

00:25:41 That means we need to use gradients, right?

00:25:45 To optimize the loss function.

00:25:48 And for that purpose,

00:25:49 we need to do the back propagation

00:25:51 to calculate the gradients of j with regard to different w's,

00:25:55 using the chain rule, right?

00:25:57 So,
