Lecture 20.3: Modern RNNs - LSTM & GRU

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Talking about how to learn from sequence data.

00:00:04 And again, as a reminder, we talked about RNNs, right?

00:00:13 So we talked about the utility of the RNNs

00:00:18 in different kinds of problems.

00:00:20 We'll have a review of that very quickly.

00:00:22 And also what is the corresponding computational graphs,

00:00:25 right, for the forward propagation, right?

00:00:28 And we also did a little bit backward propagation

00:00:31 to see where the vanishing gradient

00:00:34 or exploding gradient problems might become, right?

00:00:39 So we talked about what is the problem, why does it happen,

00:00:42 how is it really addressed, and why is it bad for RNNs?

00:00:46 So I think then we should start talking about

00:00:49 some modern RNNs, especially RSDM and GRUs

00:00:53 that are essentially solving that problem to some extent, okay?

00:01:01 But again, I want to give you a reminder that,

00:01:04 you know, why do we care about such cases, right?

00:01:08 So we knew that the vanishing gradient problem is that,

00:01:13 you know, the information from far away is probably lost

00:01:18 lost in the back propagation, the gradient information, right?

00:01:21 But that such long-term information

00:01:24 actually can be interesting and can be important

00:01:27 because as in this case is, right?

00:01:30 So we find out that what we need to fit,

00:01:34 filling here is actually very far away,

00:01:37 it's actually tickets, right?

00:01:40 So in other words, there are such information

00:01:44 which appears to be far in the sequence,

00:01:48 but actually very close, very important in the syntax.

00:01:54 So for those kind of syntactically close,

00:01:58 but sequence-wise far, right?

00:02:01 So for that kind of information,

00:02:04 if you are facing the vanishing gradient problem,

00:02:07 then it's the last thing you want, you desire, right?

00:02:12 So I think now it's time to think about

00:02:15 how to, you know, address this problem more fundamentally

00:02:19 than just try to, you know, mitigate them, right?

00:02:22 So again, the sequential recency is not as important

00:02:28 as a syntactic recency, which actually is sequentially far away,

00:02:33 right?

00:02:35 So instead of just trying to, you know, mitigate these issues,

00:02:39 like what I said before,

00:02:40 you can clip the gradient when it's exploding

00:02:42 or use some activation function or dropout,

00:02:46 but can we address this issue a little bit more fundamentally?

00:02:49 In other words, you do want to consider

00:02:53 the long-term information,

00:02:55 which can be syntactically recent or close,

00:02:59 but you're facing the vanishing gradient problem, right?

00:03:03 So what can you do?

00:03:05 So how to fix the problem for RNNs?

00:03:08 There are a few observations.

00:03:10 First is that sometimes an early observation

00:03:14 is very significant in predicting future,

00:03:18 like that ticket was way early, right?

00:03:21 So we need a memory, right?

00:03:24 A memory mechanism to store such important

00:03:29 but early information, right?

00:03:31 The Valena RNNs do not have the memories

00:03:34 to keep updating the hidden states

00:03:36 and keep some early observations in mind, right?

00:03:39 So that's one thing we need, a memory mechanism, right?

00:03:43 But if you, we all know that one important

00:03:49 function of our brain is to, you know, memorize,

00:03:52 but another important function is that it can forget.

00:03:57 Actually, it's a very important mechanism,

00:04:01 and it actually helps the long-term cognition well-being.

00:04:07 You know, you have to remember that,

00:04:10 even though the early observations can be important,

00:04:14 but sometimes, or many of them, are just not even relevant.

00:04:19 So in those cases, you need to memorize,

00:04:21 but you also have to forget, right,

00:04:24 in the latent space, those early.

00:04:26 So it's not for forgetting the early observations,

00:04:30 it's forgetting the corresponding latent variables

00:04:33 or the hidden variables in the latent space, okay?

00:04:37 And lastly, sometimes there is a logical break

00:04:40 between parts of the sequence,

00:04:42 such like chapters of a book or major paragraphs.

00:04:46 In those cases, we need to reset the internal states.

00:04:50 So these are the three keywords, right?

00:04:53 Memory, forgetting, and resetting.

00:04:56 So these are the three things that are important

00:04:58 to one of the most important modern RNNs,

00:05:02 that is RSTM.

00:05:05 Okay, RSTM is abbreviation for long, short-term memory, okay?

00:05:10 So as you can see, it's trying to use a memory mechanism

00:05:15 to use some sequentially long or sequentially far away information,

00:05:20 right, but it's using that in the short-term way.

00:05:22 We'll see what that means, right?

00:05:24 So this was proposed in 1997

00:05:27 as a solution to the vanishing gradient problem, okay?

00:05:31 It's not just mitigating the problem a little bit.

00:05:38 It's a solution, right?

00:05:40 So let's look at what the solution is, right?

00:05:43 So on step T, there is a hidden state, H sub T,

00:05:47 and the cell state, C of T, right?

00:05:51 And I put parentheses to indicate this T is the time index, right?

00:05:58 So both are vectors of length M,

00:06:01 and the cell, the cell state, stores long-term information, right?

00:06:06 And the cell stores long-term information,

00:06:09 which is encoded in both.

00:06:11 And then this RSTM as a building block,

00:06:13 it can erase, write, and read information from the cell, okay?

00:06:18 So it can forget, it can add, and it can load, right?

00:06:23 So the selection of which information is erased or read,

00:06:27 is controlled by three corresponding gates.

00:06:32 So these gates are very important concepts.

00:06:35 So these gates, you know, are corresponding to these three mechanisms, right?

00:06:39 So forgetting, right?

00:06:42 Memory, and resetting, okay?

00:06:44 So these gates are of length N vectors,

00:06:48 and on each time step, each element of the gates can be open, closed,

00:06:54 or somewhere in between.

00:06:55 So this, this case can have different kind of states,

00:07:00 and they are dynamic, their value is computed based on the current context.

00:07:05 Okay, let's, let's now understand them in more details.

00:07:08 So, let's first look at these mathematics, okay?

00:07:15 Okay, so what we have is, remember, right, this, you know, gates have inputs of H and X.

00:07:24 Remember, H is the hidden state from the previous time point.

00:07:29 X is the state of the current time point, right?

00:07:34 They are fed into this function, which is a forget gate, parametrized by WF, UF, and BF.

00:07:42 Right, this is a function, which is actually a sigmoid function.

00:07:45 Remember, all gates values are between 0 and 1.

00:07:49 So that sigmoid function is, you know, natural choice.

00:07:55 The second gate, so that's a forgetting gate.

00:07:58 The second gate is to write, right?

00:08:03 And, to read, excuse me, to read.

00:08:06 So we need to, you know, what parts of the new cell will be written to the cell.

00:08:15 Okay, what part of the new cell content.

00:08:18 So in this case, again, we have H in the previous time point, X of the current time point,

00:08:25 now parametrized by different parameters.

00:08:28 Okay, this gate parameters are WI, UI, and BI.

00:08:31 And then this is, the output is again fed to a sigmoid function,

00:08:37 and so that the gate stays are between 0 and 1.

00:08:41 And lastly, we have the output gate.

00:08:43 Okay, very similar, but now parametrized by WO, UO, and VO.

00:08:49 So we have these three gates, forgetting gate, input gate, and output gate.

00:08:55 Okay, so now, now that we see, right, how these H and X are fed into this RSDM building block,

00:09:06 and now we have, you know, three outputs, right?

00:09:10 And how do we deal with this, right?

00:09:12 How do we deal with this, right?

00:09:14 So there are three things to keep in mind.

00:09:17 The first thing is the new cell content.

00:09:20 So this is the new cell content to be written to the cell,

00:09:24 which is a hyperbolic tangent function with very similar, you know, input of H and X.

00:09:31 So this is very similar format, as you have seen before,

00:09:36 but this activation function is a hyperbolic tangent.

00:09:40 Okay, and the cell state, right?

00:09:44 So you have to somehow erase or forget something from the last and write something from the new, right?

00:09:51 So you have to forget something from the last, so you forget from the last, right?

00:09:57 Forget gate controls, so this is a gating function, right?

00:10:03 So it's forgetting some of the last cell state because it's T minus one, okay?

00:10:10 And in the meantime, you're inputting or writing, right?

00:10:15 This is an input gate, some new content to the cell, right?

00:10:22 And then the hidden state is gonna be, you know, using an output gate

00:10:31 on top of the hyperbolic tangent of this cell state at time T.

00:10:37 So these circles, as I said, this gating mechanism is through the element-wise product, okay?

00:10:46 Any questions so far? Any questions?

00:10:51 Okay, so that's what the LSTM is doing.

00:10:54 So as you can see that we have three gates that are basically determining, right,

00:10:59 whose states are between zero and one.

00:11:01 They're determining what previous cell state, right?

00:11:08 What portion of the previous cell state to be forgotten,

00:11:13 what portion of the new cell state or content to be written,

00:11:18 and how they should be output to these new hidden state, H sub T, H of T, okay?

00:11:26 So this is what LSTM is doing.

00:11:29 As you can see that, you know, the way we are trying to utilize, right,

00:11:35 this previous memory is we are only looking at T minus one,

00:11:42 the most immediate, the last cell state, right?

00:11:46 Because the last cell state has some memory, right?

00:11:51 Some part is forgotten, but it has some memory from, you know, T minus two, H of T minus two, right?

00:11:59 So if you recall in the convolutional neural network, we talked about the receptive field.

00:12:07 So in this case, in that case, receptive field is talking about the spatial case.

00:12:14 In this case, you can see that even though we are just looking at the previous state,

00:12:19 the last cell state, but that is also determined by the previous state.

00:12:23 So we actually have memory of the past, right?

00:12:27 But maybe the longer ones are forgotten more, but we don't know that.

00:12:31 Remember, these gates are having parameters to be learned, right?

00:12:35 So it's not going to be determined by you.

00:12:38 It's going to be learned from the data, which portion to be forgotten, okay?

00:12:43 And now let's look at that was just the one LSTM block, right?

00:12:49 So now, right, if you think about putting more and more into there, right?

00:12:54 As I see what we had before was this, you know, LSTM whose input is H T minus one and X T, right?

00:13:06 And the output is H T. And to get this output, we actually use the three gates, right?

00:13:12 Determining how much of this is going to be forgotten, right?

00:13:16 How much of the new cell states would be written, and how much their combination will be output to the next, right?

00:13:25 So that's what we had. And we can do this in a sequential way.

00:13:29 This is like the on-road, right? On-road, recursive formula, right? Over time.

00:13:35 Okay.

00:13:36 Now, let's look at it again, right?

00:13:41 So as a reminder, right?

00:13:44 So here, as I said before, you had the previous cell state, right?

00:13:49 And you have to determine, you know, what portion to be forgotten, and that's controlled by the forgetting gate with some parameters to be learned, right?

00:14:00 And then this, you know, current state is, you know, partially given by the previous hidden state, right?

00:14:09 And it's going to be written, right, into the new state by using parameters from the input gate.

00:14:17 And then we do this new cell content calculation using this hyperbolic tangent, and then these two pieces of information, right?

00:14:27 The, you know, memories, right? The history information after something is forgotten, and the new cell state are put together,

00:14:36 and then went through this output gate, right?

00:14:41 It's, again, parameterized by some w's and v's.

00:14:46 And then we have the output of the new cell content at time t, and the new hidden state at time t.

00:14:55 So this is done recursively, okay?

00:14:57 This is what Y is called long short-term memory.

00:15:01 In other words, you do have the short-term memory, right?

00:15:04 But it's also kind of long because you have this recursive.

00:15:10 Okay, so this is a solution, as I said.

00:15:13 It's a solution to the vanishing gradient problem that is particularly troublesome in Arnon's backpropagation, okay?

00:15:21 And that was in 1997, and again, I redraw this LSTM, right, with the same kind of, you know, forgetting gate, input gate, and output gate here, right?

00:15:38 But two decades later, right, GRU, right, is the newer model, right?

00:15:45 The building block.

00:15:47 Essentially, it's less fancy and less parameters, if you look at it, right?

00:15:51 So you see that we have, you know, the hidden state, right?

00:15:56 So instead of, you know, here, right, we had to determine what portion to be forgotten.

00:16:01 Here we have the hidden state, right?

00:16:03 And then it's gonna be, you know, combined with input.

00:16:08 Here I just want to give you a very intuitive understanding before we look at the math.

00:16:12 And then we have a reset gate here, right?

00:16:16 And then, excuse me, and then this is how we determine the new information, right?

00:16:21 So apparently you see these much less parameters are being used.

00:16:26 And then we have the output again here, right?

00:16:31 So let's try to, you know, compare, right?

00:16:38 So this was in 2014, a simpler alternative to LSTM, right?

00:16:43 So in each time step, T, we have the input XT and HT, but we don't have CT, the cell state anymore, okay?

00:16:54 And we have two gates, right?

00:16:57 Update gate and resetting gate, okay?

00:17:00 And these two gates are taking H, T minus one and XT as the previous formula, right?

00:17:07 It's very similar.

00:17:08 We have this linear and a nonlinear activation function.

00:17:11 We have the U and R.

00:17:13 One is for updating, update gate, which controls what parts of the hidden state are updated versus reserved.

00:17:22 Preserved.

00:17:23 Then we have the resetting gate, which controls what parts of the previous hidden state are used to compute the new, okay?

00:17:31 So this is the update gate and one is the resetting gate.

00:17:35 So what part of the current and what part of the previous hidden state?

00:17:40 Remember, there's no cell state anymore, right?

00:17:43 So let's again compare, right?

00:17:46 So there's no cell state, right?

00:17:48 So actually, I only have the hidden state from the previous time point together with the input, okay?

00:17:57 That's what we have.

00:17:58 We don't have this cell state anymore, right?

00:18:01 And we only have two gates, right?

00:18:04 One is to determine how much we reset, right?

00:18:09 From the previous and how much we update for the current.

00:18:14 So the new hidden state content, right?

00:18:19 Is here.

00:18:20 It's very similar as we have before, but pay attention here.

00:18:24 So we have this elementwise dot product between the resetting gate output, right?

00:18:34 Again, it's this gate are between zero and one, right?

00:18:37 Thanks to the sigmoid function, right?

00:18:41 So we have this sort of elementwise product between the reset gate output and the previous.

00:18:50 So we're basically determining, right?

00:18:52 So you use this whenever you see this circle elementwise product and you see this function in front of it or vector,

00:19:00 which is between zero and one.

00:19:02 Basically, what's in front of it is like a gate to what's after, right?

00:19:07 It's like a gate in the information.

00:19:09 In this case, the previous or the last time step points hidden state, right?

00:19:15 So we gate the previous time points hidden state with the reset gates.

00:19:21 So we have this right gate output, right?

00:19:24 Whose parameters are to be learned, right?

00:19:26 Now we again feed this directly to a function like what we had before.

00:19:34 Again, W, right?

00:19:35 Times that output plus U times that output, right?

00:19:39 So this is the hidden parameters, right?

00:19:42 And plus BH, right?

00:19:44 This is very similar, right?

00:19:46 If you look at RSTM, that's what we had before as well, just right here.

00:19:52 But we don't have cell states anymore.

00:19:54 We directly give it to the output state.

00:19:57 So it's like the forgetting gate is replaced by the resetting gate, and this cell state is directly replaced by the hidden state,

00:20:07 and there's no this thing.

00:20:09 And then you feed this whole thing and replace this part, right?

00:20:13 So that's what exactly you get.

00:20:15 And the output is htl.t, right?

00:20:20 And how do you go from htl.t, right?

00:20:24 The new hidden state content, this is the new content, to the current hidden state, right?

00:20:30 So the ultimate ht, right?

00:20:32 So essentially you are combined, you are utilizing some information from the last step, and some new information from the current step.

00:20:41 Again, this is controlled by the U, that is the updating gate, right?

00:20:45 That's right here.

00:20:46 And this is a convex combination between the last hidden state and the current new content before we do this convex combination and call it the new hidden state.

00:21:01 Okay?

00:21:02 Again, you know, how does this solve vanishing gradient problem?

00:21:06 Just like RSTM, it's much easier to retain the long-term information, right?

00:21:14 So as I said, it's as if saying that this convolutional neural network, even though these convolutional kernels are small, they have very wide receptive field if the depth of the hidden layers is large enough, right?

00:21:36 So in this case, using this recurrent gating mechanisms, we actually are able to combine a lot of short-term memories together into this, you know, very remote memory.

00:21:51 As long as it's relevant, as we will learn from the data, it will be retained, okay?

00:21:58 Any question about RSTM and GRU for now?

00:22:05 Any?

00:22:06 Okay, so the RSTM, right?

00:22:11 So 2013, 2015, that's the time that RSTM starts achieving the state-of-the-art results if you're going to this, you know, this American Conference of Linguistics or other kinds of things.

00:22:27 NLP conferences, right?

00:22:29 So RSTMs and later GRUs are, you know, sort of the state-of-the-art.

00:22:35 Some tasks involved include the handwriting recognition, speech recognition, machine translation, passing or image captioning.

00:22:47 These examples I have shown you before using this very simple diagram that we will show you again later again.

00:22:59 And that has become the dominant approach, RSTM, okay?

00:23:04 But since 2019 in the, you know, in the natural language processing field, actually I'm not completely thorough because since last year, even in the computer vision field, transformers have become more dominant for certain tasks.

00:23:26 Not all the time, so for certain tasks, right?

00:23:29 So you can see that in this very simple example in, you know, WMT, right?

00:23:34 It's MT Conference, Machine Translation Conference, plus competition.

00:23:39 2016, the summary report contains RNN 44 times.

00:23:44 In 2018, report contains RNN 9 times and a transformer 63 times, okay?

00:23:51 But don't get the wrong impression that, oh, we're just going to desert RNNs for transformers, for whatever this NLP tasks are.

00:24:03 You have to keep in mind that, you know, all of them are introduced for some reasons and they have their own niche.

00:24:14 They have something they are very good at, you know, and also they are building blocks that you can utilize in your own scenery.

00:24:22 So it's not like either this or that, if not this, then that.

00:24:29 So there are a lot of things that can be combined together and there are a lot of things you have to choose and, you know, use wisely.

00:24:37 Again, we'll see a little bit more about this in today's lecture about attentions and self attentions, okay?