# Lecture 20.2: The Exploding or Vanishing Gradient Problem

[Recording](https://www.youtube.com/watch?v=_kpBSPH6ARM)

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Probably we'll talk a little bit more about those as well.

00:00:02 But before then, I want to say something more than the vanilla RNNs.

00:00:08 So the modern RNNs compared to the vanilla RNNs, including the encoder decoder and bidirection,

00:00:15 these ones, but there are two representative architectures called LSTM and GRU,

00:00:24 which are in some sense simpler, but it's actually addressing something

00:00:29 that's fundamentally missing in the vanilla RNN, and we'll talk about it.

00:00:33 The fundamental deficiency is related to the computation.

00:00:37 Okay, so let's talk about why do we need to introduce LSTM and GRU, right?

00:00:46 So I'm going to give you the motivation, okay?

00:00:49 The motivation is coming from the vanishing or exploding gradient problem.

00:00:53 I will next tell you what is the problem, why does it happen,

00:00:58 how is it usually addressed, and why is it so bad for RNNs in particular?

00:01:03 Okay, so now let's look at this problem, okay?

00:01:07 It comes from the computation for neural networks, specifically back propagation, okay?

00:01:13 But I think we now understand what back propagations are needed and why.

00:01:18 Okay, so now let's look at a very simple example, just a L layer MLP and do a back propagation, okay?

00:01:27 So, and we have the superscript, right?

00:01:31 This H is like the hidden layer, right?

00:01:33 And we, remember it's a vector, right?

00:01:35 Whose length is the width of that layer, right?

00:01:39 And the superscript gives you the layer index, right?

00:01:43 And the next layer's hidden, the input to the next hidden layer is, you know,

00:01:56 the function of the previous hidden layer, right?

00:02:01 So that is parameterized by W, right?

00:02:04 So, as we said before, it could be a linear and then followed by a nonlinear, right?

00:02:10 But essentially, as we said last time, it's like a composing, right?

00:02:16 So you have F1 and then F1 of X is fed as input to,

00:02:22 sorry, X is fed into the input of this input layer and the output is F1 X and F1 X is fed to the first hidden layer.

00:02:34 So that's like fed to F2.

00:02:36 So we use these circles to indicate this composing operation, okay?

00:02:43 And then what happens is like when we are calculating, you know, the, let's say the output is O, right?

00:02:52 Which is a simple, I'm not going all the way to J because J is a function of O.

00:02:58 So let me just try to calculate the gradient of O with regard to the W sub L, right?

00:03:05 The L-th layer, L-th layer weight matrix, okay?

00:03:13 So as you can see that using the change rule, right?

00:03:18 I can recursively apply this.

00:03:21 So for instance, I can look at what is the L-th layer, right?

00:03:30 So the L-th layer, so I can ask what's the L-th layer's,

00:03:34 this H sub L, right, with regard to WL, right?

00:03:42 And I can also have the L-plus-1's layers H and all the way to this capital L-th layer.

00:03:50 So I'm just using a change rule here, right?

00:03:54 And then if you look at what these partial derivatives are, remember our H, right,

00:04:01 is mapped by this F function.

00:04:04 And F is parameterized by W of that layer L.

00:04:08 So what I'm doing here is essentially, you know, just using, you know,

00:04:14 using some matrix product and activation function, okay?

00:04:21 So now let me denote each of this as matrix M sub index.

00:04:29 This could be capital L and L-1 all the way to M sub L.

00:04:34 The last one is a vector, so let's just call it a V sub L, okay?

00:04:38 So now let's look at this gradient as a product of some matrix and vector, right?

00:04:50 And let's ask what's the magnitude of this gradient can be, right?

00:04:56 Apparently, this magnitude of this gradient is determined by the size of these matrices,

00:05:05 which can be quantified by their eigenvalues, right?

00:05:12 So the eigenvalues can be small or large, and the gradient as a product can be large or very large or small, right?

00:05:20 So this could, if the gradients are very small or very large, then this could lead to the instability to your gradient-based optimization

00:05:30 when you are training the neural networks, is that right?

00:05:33 So that is, you know, one glimpse to this vanishing and exploding gradient, okay?

00:05:42 But this is just MLP, okay?

00:05:45 So how do we address this problem, right?

00:05:53 I'm not saying that I can completely remove the problem for the MLP, but I can mitigate or relieve the problem, right?

00:06:01 One way is to use activation function like, you know, changing the sigmoid to ReLU, right?

00:06:09 That's one possibility, right?

00:06:11 If you use, you know, a different choice of the activation function.

00:06:15 Another way is that you can use better initialization, right?

00:06:21 Instead of random or all zero, you could use different initialization.

00:06:25 This one is following a Gaussian distribution, for instance, right?

00:06:30 And also, empirically, we know that dropout can help, right?

00:06:35 It can help break the permutation symmetry, right?

00:06:39 What do I mean by permutation symmetry?

00:06:41 So layers are permutation symmetry, in the sense that if I just swap some layers, then just their corresponding values,

00:06:51 weight values can be just swapped then, right?

00:06:55 So in this sense, if you initialize, right, all the, you know, parameters the same, then you swap these two layers,

00:07:06 then it doesn't really matter, right?

00:07:09 The outcome is the same, right?

00:07:11 So either you change the way you initialize, right?

00:07:15 But it's possible that you use the same initialization distribution, then, you know, if you want to break that symmetry, right,

00:07:22 after swapping the layers, the same initial parameters are used, then you can break the symmetry by dropout,

00:07:29 because different layers will be dropped out, some random portion of the neurons, right?

00:07:37 Then you will not see these two layers being effectively the same anymore, right?

00:07:43 So do you get what I mean by permutation symmetry?

00:07:48 Okay, especially when you initialize all the parameters the same, right?

00:07:53 So these are the ways you can mitigate the vanishing gradient or the exploding gradient issue, okay?

00:08:01 And, yes?

00:08:06 How does, like, the icon-tree have the permutation symmetry?

00:08:12 Yes.

00:08:13 But I didn't get how dropouts can...

00:08:17 Right.

00:08:18 So what I was trying to say was that layers are permutation symmetry, meaning that if I swap the third layer and the fifth layer, right,

00:08:28 for the model it doesn't really matter.

00:08:30 In the end, you know, the corresponding is as if that your weight vector has some segments swapped, right?

00:08:40 It doesn't really matter.

00:08:42 So for optimization, if they were initialized the same way, right, then that's, you know, it's completely computational symmetry, right?

00:08:53 If you want to break the symmetry, you have to somehow make these two layers, and after swapping the order, it won't be the same.

00:09:02 And how do you make that happen is when these layers...

00:09:08 when these layers cannot be regarded as equivalent for the ultimate mathematical model.

00:09:16 So for that to happen, one way is to sort of make the individual layers incomplete by dropping these neurons, right?

00:09:26 Right?

00:09:27 So this is completely changing the symmetry, okay?

00:09:34 But this...

00:09:35 Am I making myself clear?

00:09:37 It's a great question.

00:09:38 It's not that easy to understand, okay?

00:09:40 A lot of the empirical recipes are just the known, and a few of them have some explanation.

00:09:48 But these are some explanations, but these are just mitigating, okay?

00:09:53 If you want to address the problem, for instance, you know, the gradient explosion issue, right?

00:10:04 If you have the exploding gradient, one of the easiest way is to sort of like in the optimization, if you're going too far, you project back, right?

00:10:17 So this is the same thing that if the norm of the gradient is too large, right, because our training, model training is based on the gradient, if the gradient is too large, then numerically it's unstable.

00:10:29 If it's too large, then you scale it down, then apply SGD update.

00:10:34 So for instance, you have calculated this gradient.

00:10:38 Remember, this is the stochastic gradient, okay?

00:10:41 Last time we talked about gradient descent, stochastic gradient descent, mini batch, stochastic gradient descent, and then in addition, we talked about momentum and other ideas that led to Adam, right?

00:10:55 If you forgot about them, you may want to refresh your memory, about the fact that the mini batch stochastic gradient descent is getting the best of both worlds.

00:11:12 It's an unbiased estimate of the gradient, right?

00:11:17 And it also has a small variance, right?

00:11:21 And you can use the vectorization and exploiting the GPU, right?

00:11:28 So that's really great.

00:11:30 So in this case, it's very simple.

00:11:35 If the norm is above some threshold, then you just, you know, make this gradient to be smaller, the same direction, but the scalar is normalized, right?

00:11:49 So that the norm now is just a threshold, isn't it?

00:11:53 Right?

00:11:54 It's because the denominator is the norm, right?

00:11:58 If you take the norm of the whole thing, you get the threshold, right?

00:12:02 So it's just like, you know, I'm going that direction, but I'm going to take maximum step size, like, threshold.

00:12:11 That's what they're trying to do, okay?

00:12:13 This is the gradient clipping.

00:12:16 So let's see intuitive explanation, right?

00:12:20 Here, let's say we have this landscape, right?

00:12:24 And this is a landscape.

00:12:27 It's actually taken from a loss function from a simple vanilla RNN.

00:12:34 There's a one-dimensional W and a one-dimensional threshold B.

00:12:40 This is a linear term and this is a constant term, right?

00:12:43 So these are the affine parameters.

00:12:47 And we are trying to train the model by minimizing the loss function of W and B.

00:12:52 And this is kind of the landscape.

00:12:56 And imagine that, you know, we are, like, you know, inching toward, right?

00:13:05 So if we are minimizing, we are inching toward this global minimum, right?

00:13:11 But the problem is that sometimes, you know, for instance, if you are here, right?

00:13:16 If you reach here, then this gradient is so high that it can just explode in the sense that, you know,

00:13:26 you follow that gradient direction, but don't forget that this norm is very high.

00:13:31 So you end up going this direction and walk too far and reaches here.

00:13:37 Once you are there, then you get follow the gradient there again and you go here and so on.

00:13:44 So you got no convergence desired, right?

00:13:49 But if you, in other words, this is like a cliff, right?

00:13:54 The cliff is dangerous because the gradient is very steep here, right?

00:13:59 The exploding gradient.

00:14:01 But if you use a clipping, right, right, then this is what you get and this is much better, okay?

00:14:08 So again, on the left, the gradient descent takes two very large steps, right?

00:14:15 Due to the steep gradient resulting in climbing the hill and then shooting off to the right.

00:14:20 Both are bad updates, right?

00:14:22 On the right, the gradient clipping reduces the size of these steps so you see that it's less drastic.

00:14:29 You're not climbing all the way to the cliff and go somewhere else completely, okay?

00:14:36 So this is for exploding gradients, okay?

00:14:41 So now why you understood what vanishing and exploding gradient problems are?

00:14:47 Why they are bad for training neural networks, right?

00:14:51 And how they are usually addressed in general?

00:14:55 So now why are you all of a sudden talking about vanishing and exploding gradient problems when introducing RNNs?

00:15:04 Is there something special about RNNs?

00:15:07 Yes.

00:15:08 I think first we are ready to talk about vanishing and exploding gradients after the last lecture.

00:15:15 The neural network computation.

00:15:17 We understand backpropagation and all these gradients, right?

00:15:23 But another reason is yes, as you guessed, RNN is extremely vulnerable to these problems.

00:15:31 And we can take a look at the question.

00:15:33 Take a look at the problem using this illustration, okay?

00:15:37 And it really comes from the same keyword that gave it some blessing.

00:15:43 The recurrence, okay?

00:15:45 It's both a blessing and a curse.

00:15:47 So let's look at this simple example, right?

00:15:52 So we have again, you know, you have x, right?

00:15:57 Let's say, you know, here we, you know, don't pay too much attention to where x is

00:16:02 because here I want to pay more attention to how the gradients of j with regard to w can be calculated.

00:16:11 Is that right?

00:16:12 And so in this case, let's see.

00:16:19 So j4 is here, right?

00:16:22 So I can go backwards, right?

00:16:26 I can go backwards.

00:16:27 I can calculate how, you know, j4 is changing with regard to h4, right?

00:16:33 And then h4, how it's changing with regard to h3, right?

00:16:38 And so on, right?

00:16:40 All the way back.

00:16:42 And notice that I have these terms, right?

00:16:51 I have these ones, right?

00:16:55 And this is h1, right?

00:16:57 And this is h1.

00:16:59 So the number of such terms is related to which hidden state is corresponding to.

00:17:08 The earlier the time point, the more multiplications I have to give.

00:17:14 Is that right?

00:17:16 So now imagine, what if these terms are very small, right?

00:17:25 And if these terms are very small, like between 0 and 1, if you multiply them together, right?

00:17:32 The more you multiply, the smaller this magnitude becomes.

00:17:36 So for h2, it's a small value, but for h1 is even smaller.

00:17:41 So the more you back propagate, the deeper you back propagate, the smaller the gradient becomes.

00:17:51 Is that right?

00:17:53 So in other words, if that happens, you don't have enough sensitivity from the j with regard to the earlier h's, right?

00:18:05 And then that is not good for your training.

00:18:09 Okay?

00:18:10 So if the gradients, right?

00:18:14 So, right?

00:18:15 So if the gradients become extremely small over long distances, then we cannot tell the difference between two scenarios.

00:18:28 One scenario is that, yes, we calculate the gradient and it's small and it's correct.

00:18:36 It's supposed to be small because there is indeed no dependency between steps t and t plus n when n is large, for instance.

00:18:46 Right?

00:18:47 So we calculate the small gradient because indeed we shouldn't, as the problem dictates, there isn't too much information back from t plus n to t.

00:18:58 Right?

00:18:59 There is no dependence.

00:19:02 But there is a, don't forget that there is a difference between representability and learnability.

00:19:08 Another scenario is there is a dependency, but you learned a wrong parameter w and that w tells you that, oh, there is no dependency.

00:19:16 Right?

00:19:17 So in other words, once the gradients become small over long distances, you don't really know whether it's supposed to be that way and you did it right or it's not that way and you did it wrong.

00:19:31 Right?

00:19:32 So you don't really want to see that gradient vanishing problems occurring.

00:19:36 Okay?

00:19:37 Any questions so far?

00:19:39 Right?

00:19:40 So you see now why it's a bigger problem for our nons.

00:19:43 Don't forget, don't forget all these partial derivatives has w involved, the same w involved.

00:19:53 Okay?

00:19:54 Don't forget about that.

00:19:56 Okay, so now let's look at, again, for you to better understand what I mean by, you know, the gradient signal from far away is lost because it's much smaller than the gradient signal from close by.

00:20:11 And let's try to understand it, right?

00:20:13 The model weights are only updated with respect to the new effects and no long-term effects.

00:20:20 I will give you one example in the language models about these long-term effects, right?

00:20:27 Why it's needed and why the inability to capture it could be a problem.

00:20:33 Okay?

00:20:34 Before I go, it's 4.45.

00:20:38 Hey, Ming, do you have journal club today?

00:20:41 I haven't seen the slides yet.

00:20:43 Okay, then in five minutes?

00:20:46 Five minutes?

00:20:47 Okay.

00:20:48 Okay, so then I'm trying to finish here then.

00:20:51 So let's give you one example about these long-term effects, okay?

00:20:55 Let's read this text for like 30 seconds and fill in the blank.

00:21:01 Okay, so what did she print?

00:21:17 Tickets.

00:21:18 Tickets, right?

00:21:20 But where were tickets?

00:21:23 Very long-term, right?

00:21:26 So apparently, you know, if you have a badly trained model in this case, right?

00:21:33 So this dependency is really far, right?

00:21:35 If the gradient is really small, then the model can't learn this dependency, right?

00:21:39 And unable to predict the long-distance dependency in the test of time, okay?

00:21:47 So, and, you know, then let's talk about a little bit, you know, what RSTM and GRU can do, but I won't be able to talk about the specific architectures.

00:22:01 Let's do that next time together with language models.

00:22:05 So the effect of vanishing gradient in RNN, right?

00:22:08 The RM task is the writer of the book is or are, right?

00:22:13 So if you're only looking at the nearest term, the books may imply that it's R, but you got to really realize that what dictates the choice is actually from writer.

00:22:25 Is that right?

00:22:26 It's from the writer.

00:22:27 It's not from the books, right?

00:22:29 So the due to the vanishing gradient, the RNN language models are better at learning from sequential recency than syntactic recency, right?

00:22:42 The sequential recency means that, you know, the positional proximity, but the syntactic recency is the proximity in the syntax, okay?

00:22:52 So that's why in those early days, the RNN based language models make this kind of mistakes a lot, okay?

00:23:00 And how to fix the size problems, right?

00:23:05 How to fix such problems?

00:23:07 This is actually exactly how RSTM is motivated and invented, and we'll talk more about it, okay?

00:23:16 So you need some memory to store the distant or earlier information, right?

00:23:23 You need some memory mechanism.

00:23:25 You need to remember something.

00:23:27 Then you also need to forget about something that is not pertinent.

00:23:32 The writer of the books, the books is much more recent, but it's not that important, right?

00:23:39 And then sometimes you need to reset the memory for the internal states,

00:23:44 and that is exactly what RSTM is doing.

00:23:47 So next time, let's talk about RSTM and then GRU.

00:23:51 We'll end that and then talk about language models a little bit, okay?

00:23:56 And next time.
