Lecture 18.3: Convolutional Neural Networks (CNNs): WHAT? + Case Study
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 Now let's talk about what corresponding math. I talked about the knowledge and then convolution can give you that. But more than that, what corresponding math or changing the operations are motivated by this knowledge?
00:00:18 So I will talk about convolution we talked about before, but we'll also talk about channels, pooling, strides, and padding. And we'll ask how much can be saved and what else can be saved.
00:00:30 So let's talk about the concepts. So this is what we have learned so far. So first thing I want to talk about is the additional concept. The first thing is the channel.
00:00:48 Okay? So channel is the following. I said before that each image data is consists of a lot of pixels in the 2D grid. And each pixel could be one numerical value or multiple numerical value.
00:01:04 So if it has multiple numerical value, for instance RGB or CYMK, then we can call each like a channel. Right?
00:01:16 So in this case, there are three color channels. Right? Right? But in other cases, you may parameterize your image data in different ways and you have different kind of channels. Right?
00:01:27 So this is what the channel means. Right? So this is another strength of GPU in this kind of multi-channel introduction.
00:01:41 Because if you think about this multi-channel, right? So you can actually, you know, previously if you have a single channel, right, you do this matrix multiplication or convolution in this way. Right?
00:01:54 But if you think about multiple channels, you can, you know, imagine as if there is an additional dimension, which is like a depth of this paper or the screen. Okay?
00:02:07 Now you can overlay these different channels behind this first channel is like the additional dimension. Right? So now you don't just have a matrix, you have a tensor. Right? Right?
00:02:23 And then you can do the convolution or other kind of linear algebra in the tensor space. Right? Right? So, you know, if you think about why some products are called tensor flow, right, it's really very convenient.
00:02:40 And the output could also have multiple channels. Okay? So sometimes, sometimes you want multiple readouts and you could use a multiple output. And how do you do multiple output channels?
00:02:56 That means like in this case, you have like, you know, if you want two output channels, then it's like you are using, you know, two sets of kernels, right, along this dimension.
00:03:07 So, in the most popular neural network architectures, we actually increase the channel dimension as we go higher up. And then this will down sample to trade off the spatial resolution for greater channel depths. Okay?
00:03:23 So, this is what channel means. Any questions so far?
00:03:36 So the next concept, let's look at a bigger picture of, you know, a regular convolutional neural network, right?
00:03:46 So, there's a number of tools that we know about these convolution operators already, right? And then we are more or less familiar with the nonlinearity, right? And then there will be pooling stage involved. And then it will be, you know, fed into the next layer.
00:04:03 layer. So these are the additional concepts we're going to talk about, right?
00:04:07 First, if you remember our 320 by 280 image, I said the output is going to be
00:04:14 319 by 280 because your, you know, kernel will start with the left
00:04:23 boundary and it will stop when its right boundary hits the right boundary of the
00:04:28 image, right? So it will be one less if the kernel is of size 2 in the horizontal
00:04:36 dimension. If it's a size n in the horizontal dimension then your image
00:04:41 will, you know, reduce n minus 1 pixels horizontally, right? So the question is the
00:04:52 following. If we start with 240 by 240 and then we use 10 layers of a 5 by 5
00:04:59 convolution, in the end what's the size of the output image?
00:05:17 If you know what convolution is doing
00:05:22 any volunteers? You must know.
00:05:41 Yes, Ryan? Don't be shy to try.
00:05:45 236. 236 by 236? Okay.
00:05:53 That's going to be no variable.
00:05:55 Uh-huh. Correct. But 10 layers.
00:05:59 The layer I'm not sure.
00:06:01 Ten layers. You know it. Your answer is for one layer.
00:06:05 Minus 4.
00:06:07 Correct. So after each layer you will lose 4 pixels along either direction. Is that right?
00:06:21 So it's going to be 200 by 200, right? Right? After each of the layers you will lose 4 pixels
00:06:31 5 minus 1 and 5 minus 1. Okay? So in other words, after even 10 layers you will lose like 30% of the size and you know, and also it doesn't consider any interesting information on the boundary. So apparently that's not desired. Okay? So for that reason we introduce padding.
00:06:49 Right? So right? So if we have zero padding without padding this is what we have. Right? Right? This is another way to illustrate what we calculated before. Okay?
00:06:59 than we introduce padding, right?
00:07:01 So, right?
00:07:03 So if we have zero padding without padding,
00:07:05 this is what we have, right?
00:07:07 So this is another way to illustrate
00:07:10 what we calculated before, okay?
00:07:13 So that's why it keeps reducing, right?
00:07:17 So here you see one, two, three, four, five, six.
00:07:20 The kernel size is six, so after one layer,
00:07:23 you should have five less, right?
00:07:27 So that's how the size is changing.
00:07:32 But if you use padding, okay,
00:07:36 you pad pixels outside of the boundary
00:07:39 and give zero values, meaning that's what we call zero padding,
00:07:44 then these black points are zero padding, okay?
00:07:47 So that's what you have.
00:07:49 And you see that in each layer,
00:07:50 we keep doing this zero padding,
00:07:52 and then the size does not change, actually, okay?
00:07:57 So that's what padding means.
00:08:00 The third concept is stride.
00:08:05 At the beginning when I was introducing convolution,
00:08:07 I said that if you overlay the kernel
00:08:09 on the local patch of the input image,
00:08:12 and then you move to the right,
00:08:14 one unit, and then another one unit, and so on.
00:08:17 There I already assumed that the stride is one.
00:08:21 So that's what the stride means, right?
00:08:24 But you could have different possibilities,
00:08:28 and apparently the stride also matters, right?
00:08:32 When Ryan was saying that it's four pixels less
00:08:38 each time we pass a convolution without padding, right?
00:08:43 But that's also assuming that we are using a stride of one,
00:08:46 isn't it, right?
00:08:47 Okay.
00:08:48 So you could use different strides, apparently, right?
00:08:52 For instance, in this case, right?
00:08:55 So in this case, I'm using, it's a two-dimensional image.
00:08:59 I use three for the height and two for the width, okay?
00:09:06 So let's think about this.
00:09:07 It's a two by two, right?
00:09:10 It's a two by two.
00:09:11 And then for the width, I'm using two.
00:09:15 So first it's here, and then the next is here, right?
00:09:22 Right?
00:09:24 Right?
00:09:25 So the first time it's here, and then, you know,
00:09:27 move to the right by two, it's here.
00:09:29 And then I stop here, because if I move by two again,
00:09:32 I will hit the boundary.
00:09:34 And right now, I'm not using padding yet, okay?
00:09:39 And vertically, let's try it again.
00:09:41 Vertically, I'm using a stride of three.
00:09:43 So I start from here, and then I move down.
00:09:46 One, two, three, I would start here.
00:09:48 And then that's it.
00:09:50 So in the end, I have a two by two, okay?
00:09:53 So apparently, you have to make the math
00:09:56 a little bit more complicated when you are calculating
00:09:59 the output size when there is no padding,
00:10:02 and the stride is not one,
00:10:03 because you have to consider this division
00:10:05 and the remainders, okay?
00:10:07 So now if we have a convolution with a stride,
00:10:14 what is it like, okay?
00:10:17 So this is the convolution we have seen before, right?
00:10:19 And you can see that this is a kernel, right?
00:10:26 Overlaid on pixel one, two, and a three.
00:10:29 And then I have a stride of one.
00:10:32 So now the kernel is on two, three, and a four.
00:10:34 That's why you have these arrows, right?
00:10:37 And then I move to the right again,
00:10:39 and I have these arrows, right?
00:10:41 If the stride is two instead,
00:10:45 so when it's first here, right?
00:10:47 Then we have these output arrows.
00:10:51 Then we move this kernel by two instead of one.
00:10:56 Then next time we move it here, the center is here.
00:11:00 Then this is the parameters.
00:11:03 This is the operations you will observe.
00:11:06 So this is a clear difference.
00:11:08 It's making things even sparser, right?
00:11:13 So let's try to summarize padding and stride a little bit.
00:11:16 So now you understand the concept, right?
00:11:19 What we do with it.
00:11:21 So padding can increase the height and the width
00:11:25 of the output, right?
00:11:26 This is often used to give the output
00:11:28 the same height and width as the input, right?
00:11:32 That's something that's clear.
00:11:34 The stride apparently can reduce the resolution of the output,
00:11:39 for instance, if the stride is an m bigger than one,
00:11:45 then apparently you're reducing the height and the width
00:11:48 by a factor of one over n, right?
00:11:51 So without padding you're just losing some number of pixels,
00:11:55 but here you're scaling the height and the width.
00:12:00 So it's like reducing the resolution, right?
00:12:04 Much more drastically.
00:12:07 So padding and the stride can be used to adjust the dimensionality
00:12:12 of the data effectively.
00:12:13 So that's something that can be counted on.
00:12:18 The last concept is pooling.
00:12:22 Okay?
00:12:23 So pooling, so let's look at this case, right?
00:12:29 So what we did here is, you know,
00:12:32 previously we have a kernel, let's say two by two,
00:12:38 and let's say a, b, c, d, right?
00:12:40 Four elements.
00:12:41 What we did was multiply a by zero, multiply, you know,
00:12:47 b by one and so on and so forth, right?
00:12:52 So that's kind of, you know, you can also think of it as like a local summary, right?
00:12:57 Right?
00:12:58 So that's what convolution does.
00:13:00 What pooling does is also like a local summary, okay?
00:13:04 But it's not using convolution, like in this case.
00:13:07 It's a two by two which defines the locality,
00:13:11 and the type of pooling is max pooling.
00:13:15 What it does is find the maximum in this two by two.
00:13:20 In this case, it's four, okay?
00:13:22 And if you use a stride of one and then move on next,
00:13:26 then you have it here, then the biggest is five.
00:13:28 That's five here.
00:13:29 And then the biggest here is seven, seven here,
00:13:33 and biggest here is eight.
00:13:35 That's eight here.
00:13:36 Okay?
00:13:37 So just like convolution,
00:13:39 pooling slides a fixed size window, right?
00:13:45 Over all regions according to its stride, right?
00:13:48 If it's maximum, that's what we have seen before.
00:13:52 You can also have average pooling, right?
00:13:54 Which give you the average value and so on.
00:13:56 So, but just like convolution,
00:14:01 that's what's similar, but unlike convolution,
00:14:04 pooling does not have any parameters, right?
00:14:08 I don't have any element here.
00:14:09 I know that it's already defined.
00:14:11 I'm using the maximum or the average.
00:14:13 There's no parameters to learn.
00:14:16 Okay?
00:14:18 Okay.
00:14:19 So, now let's try to see what these operators can bring to us,
00:14:24 right?
00:14:25 So, what does this bring to us?
00:14:29 Don't forget about how we learned from our vision system.
00:14:33 This pooling is very similar to convolution in the sense that it's invariant to translations, right?
00:14:39 Thanks to the same kind of, you know, fixed size window, the same kind of window and sliding over all regions.
00:14:46 That is maintained, but it does not have any parameters.
00:14:51 So, in other words, it can help you reach some multi-scale, right?
00:14:58 For instance, you can, you know, gradually aggregate information from the first,
00:15:04 and then to the last layer, and then you gradually reduce the scale of the hidden dimension.
00:15:09 So, that's like giving you a coarser and a coarser map.
00:15:13 Now, you can actually scale up as well.
00:15:16 That's a unit, right?
00:15:19 Right?
00:15:20 So, let's look at max pooling and its invariance to translate.
00:15:28 Right?
00:15:29 Something I mentioned here.
00:15:31 Right?
00:15:32 But now let's try to visualize it again.
00:15:34 Right?
00:15:35 Right?
00:15:36 So, let's look at, you know, this case.
00:15:42 Right?
00:15:43 So, if we look at this size, again, this max pooling's fixed, what should I call it?
00:15:53 I don't want to call it a window or kernel, right?
00:15:57 But anyway, the size, this max pooling is 1 by 3 here, apparently.
00:16:02 Right?
00:16:03 You can tell right away, right?
00:16:04 It's 1 by 3.
00:16:05 So, the biggest value here is 1, apparently, and the biggest value here is 1, and so on.
00:16:11 Right?
00:16:12 So, let's say, let's say, the input is shifted to the right by 1 unit.
00:16:25 You see that 0.1 is shifted here, 1 is shifted here, 0.2 is shifted here, there's a new value here.
00:16:32 Okay?
00:16:33 Now, let's look at what this max pooling does to the translated original image.
00:16:41 Let's look at the output.
00:16:43 What did you see?
00:16:45 What did you conclude about the output?
00:16:47 Right?
00:16:48 The output is also shifted, right?
00:16:54 1, 1, 1 versus this 1, 1, 1, right?
00:16:57 Right?
00:16:58 And you can do pooling with downsampling, depending on your pooling window size, right?
00:17:07 So, you can, you know, you can, you can downsample, right?
00:17:12 And, in this case, how do you downsample?
00:17:17 You actually do a pooling with a stride, right?
00:17:20 So, what's the stride here?
00:17:22 Here is a max pooling, apparently, and it's a 1 by 3 max pooling.
00:17:28 What's the stride of this 1 by 3 max pooling?
00:17:31 2, right?
00:17:33 1, 2, right?
00:17:34 1, 2, right?
00:17:37 Okay, and this pooling, now we have learned a lot of things, right?
00:17:44 But they can be all put together, make things even more complicated.
00:17:48 We talked about channel, we talked about pooling, we talked about stride, we talked about the kernel size, right?
00:17:58 So, now let's try to mix them even more, right?
00:18:01 You could do pooling cross channel, okay?
00:18:06 You could do pooling across channel, right?
00:18:10 And, you know, let's look at this example, okay?
00:18:16 So, this is, you know, MNIST, we are just detecting 10 digits, right?
00:18:30 It's 0 to 9, this is a 5, right?
00:18:32 So, these are different, you know, handwritten 5s, right?
00:18:37 And then, in this case, let's say we have multiple detectors, you know, multiple channels, right?
00:18:47 And then, you know, in this case, you know, there's a large response in detector unit 1 here, right?
00:18:54 And then, in this case, you have the large response in detector 3 here, right?
00:19:00 So, these are different detectors, like different channels, right?
00:19:05 Then, you can actually pool them together, for instance, the max pool pooling across the channels,
00:19:12 and you get the same large response in the pooling unit, okay?
00:19:17 So, now let's summarize pooling again.
00:19:22 So, let's see what pooling is doing is that it takes the input elements in the pooling window,
00:19:29 and it does something very similar to convolution, right?
00:19:32 For instance, it's doing the maximum pooling or doing the average, right?
00:19:37 But it does not have any parameters, right?
00:19:39 So, one intuitive way to see what they are doing is, in the case of the maximum pooling,
00:19:44 it's bringing out the lighter foreground, right?
00:19:47 If you think about the grayscale image, right?
00:19:50 The lighter pixels are over higher values.
00:19:53 So, if you are doing the max pooling, you are basically bringing out the lighter foreground, right?
00:19:58 Because the foreground is really lighter, right?
00:20:01 And the larger pixel values.
00:20:03 If you do the average pooling, it's like you are smoothing the whole image,
00:20:08 and then what you may lose is the sharp edges, right?
00:20:13 Right?
00:20:14 So, one of the major functions of a pooling layer is to assess the image, right?
00:20:19 The most important thing is to alleviate the excessive sensitivity of the convolution layer to the location, okay?
00:20:26 And we can specify the padding and the stride and combine and use the channels even together, right?
00:20:32 And the max pooling combined with the stride larger than one can reduce the resolution.
00:20:38 And the pooling layer's number of output channels is the same as the number of input channels.
00:20:44 Again, you know, you could have multiple channels as well, okay?
00:20:49 Any questions so far?
00:20:52 We talked about the few important operators in the convolutional neural networks, right?
00:20:58 So, again, this is just another example like a classification architecture, right?
00:21:04 Again, you start with the input image, right?
00:21:07 And you do convolution and then do a ReLU and then this is a new size, right?
00:21:12 And then you do a pooling with the stride of four.
00:21:15 I would strongly recommend you to pay attention to these numbers to make sure that you understand them, right?
00:21:21 If you understood the concepts taught so far, you should be able to, you know, prove or, you know, calculate by yourself
00:21:29 and prove that it's the same as what's shown here, okay?
00:21:35 And this is one example, okay?
00:21:37 Let's say there's an input image.
00:21:38 Again, I'm writing this number two, right?
00:21:41 Then the convolutional kernel is five by five, right?
00:21:44 So it slides with a padding.
00:21:47 Well, actually, sorry, here there's no padding.
00:21:50 So I'm getting four less in each dimension and I'm using N1 channels, okay?
00:21:56 Then I do a max pooling two by two.
00:21:59 Then what it's doing is it's reducing the resolution by a factor of one over two in each dimension.
00:22:06 So then I reduce to 12 by 12 times N1 and I do a max pooling and I do convolution again.
00:22:13 And, you know, and then I do the max pooling and then you see this.
00:22:19 So I'm doing this four less and then scale by half again, right?
00:22:23 But this N1 has changed to N2, okay?
00:22:29 This is another thing you have to pay attention to.
00:22:32 In the end, I do a readout through the fully connected layers, okay?
00:22:36 There's nonlinear activation here, okay?
00:22:42 Again, this was introduced.
00:22:44 So what has happened since Lynette, right?
00:22:47 So there's a lot of the, you know, applications apparently, right?
00:22:51 And it's used for, you know, image recognition, like for each pixel you can classify what it belongs to, right?
00:23:00 And then if you think about what I said, how deep learning has picked up again, thanks to the image net.
00:23:12 This is Alex net, right?
00:23:14 This is Alex net, right?
00:23:16 So this is the same model as Lacoon 98 or LearnNet.
00:23:20 But it is a bigger model.
00:23:23 It has more data.
00:23:24 And also it has already GPU implemented.
00:23:27 And then better regularization it used to drop out.
00:23:30 So if you wonder why Lacoon, Lynette was already born in 98, but why it took until 2004, 12 or 11,
00:23:44 the image net for CNN to be picked up, these are the reasons, right?
00:23:50 One of the key reason is there was no image net back then, and GPU wasn't that powerful, okay?
00:23:58 But this drop out is a great idea.
00:24:01 We talked about regularization, right, before in the case of the shallow models, right?
00:24:06 In the deep models, in this case, drop out is a very effective way to regularize model complexity.
00:24:13 Okay, so after Lynette, what else can be done?
00:24:18 There are a lot of things that can be done, but I'm not able to introduce them into a lot of details, right?
00:24:28 You could have local connection, but without sharing, right?
00:24:32 Compared to the convolution, right?
00:24:34 It's somewhere between the convolution and fully connected, right?
00:24:38 So this A, B, A, B is shared in convolution, but it's no longer shared in this kind of cases.
00:24:45 It's still local, but not parameter sharing, okay?
00:24:49 What else can you do?
00:24:50 You can do tiled convolution.
00:24:52 You can cycle between the groups of shared parameters, right?
00:24:56 So you have A, B, C, D, then A, B, C, D again, right?
00:25:00 So this is a tiled convolution.
00:25:02 And there is a partial connectivity between channels, right?
00:25:06 So the connectivity can be restricted, right?
00:25:10 And there are a lot of things that can be done.
00:25:13 So, but if you look at the ImageNet, the winners after AlexNet, right?
00:25:19 So then you have seen the VGG, GoogleNet, right?
00:25:25 You see this LE here, right?
00:25:28 That's a, you know, standing ovation to the net, right?
00:25:32 And then the ResNet you're familiar with now, and then SENet and so on, right?
00:25:38 Some of you will talk about the, you know, VGGs or UNET.
00:25:42 So these are the summaries, fully convolutional networks, networks using blocks like VGG,
00:25:49 and then Inceptions used in GoogleNet and ResNet using the residual blocks.
00:25:55 So some of them will be mentioned in your journal clubs as well, okay?
00:25:59 And convolution can be done not only on images, but also on graphs.
00:26:06 So we probably want to, you know, talk more about it in a dedicated graph neural works.
00:26:11 But this is an illustration of how graph convolution is done.
00:26:17 It's very similar, but just different structure.
00:26:19 It's no longer a grid structure, but a graph structure.
00:26:22 So you're looking at a local region, which is now defined by the connectivity.
00:26:27 So this is now a neighborhood, not a grid patch anymore.
00:26:32 So let's see what we have learned so far, right?
00:26:37 So we have seen how knowledge drives math.
00:26:40 We have learned that seemingly so, you know, simple ideas like convolution, channels, pooling, strides, padding,
00:26:48 actually have so many great properties.
00:26:51 And you have to think about it.
00:26:53 What we are learning is assuming that we have these properties, these operators,
00:26:58 and we see that great, they have these properties.
00:27:00 But how were they developed?
00:27:02 Again, people actually started with the desired properties, learn from the vision systems,
00:27:07 and then they designed mathematical operators to accomplish that, right?
00:27:12 And we also learned how much can be saved compared to the dense net, and what else can be done, right?
00:27:18 So some summaries of the recent works, right?
00:27:21 So in the end, I'd like to talk about two case studies very quickly.
00:27:27 I'd like to use the same protein structure prediction as an example.
00:27:34 We have learned how it's done in the classical world, right?
00:27:39 The three scenarios and then different formulations, right?
00:27:43 So now, if you think from the data perspective, right, of course we can do it now because there are more data and more computing power and more sophisticated AI models.
00:27:53 But what can we do now, right?
00:27:55 Do you see any images in the problem of protein structure prediction?
00:28:01 Well, I'm going to talk about two simpler versions of protein structure prediction.
00:28:06 The input is the 1D sequence, but the output is not necessarily the 3D structure or the coordinates of all atoms.
00:28:14 The first simple version, I call it a 1D problem.
00:28:18 In other words, we are only predicting which fold the structure belongs to.
00:28:25 Okay?
00:28:26 So it's like a 1D summary of the 3D structure.
00:28:29 So is this a regression problem or a classification problem?
00:28:32 Classification because the labels are discrete, right?
00:28:39 We have thousands of labels.
00:28:40 So in this case, what they do is DeepSF, you know, they start with a sequence G, I, V, and so on.
00:28:48 Then you featurize every position, okay?
00:28:52 What they have is like a 45 dimension.
00:28:55 The first 20 is a one-hot encoding of amino acids.
00:28:59 The next 20 is a position-specific scoring matrix or profiles, both of which have learned before.
00:29:07 Then the next three are categorical predictions of the solvent, sorry, secondary structure, alpha, beta, or loop, right?
00:29:17 Categorical.
00:29:18 And lastly, it's a sequence predicted solvent accessibility, buried or surface.
00:29:24 So in other words, they rearrange this into a L by 45 dimensional matrix.
00:29:32 Then they run through the convolution layers.
00:29:35 They actually have 10 channels, right?
00:29:37 Then L by 2, right?
00:29:41 And 10 filters and then 2 filter sizes.
00:29:47 And then they do a pooling afterwards, right?
00:30:08 So we have the lower layer again.
00:30:09 And then flatten them.
00:30:11 And then in the end you have the dense layer to have the readout.
00:30:14 So the readout is 1 by 100, sorry, 1195.
00:30:21 This number 1195 is the number of folds.
00:30:25 So each output here is essentially, you know, like a probability, okay?
00:30:30 So I assume that they have a softmax at the end, right?
00:30:34 Because the sequence can only belong to one fold, right?
00:30:37 In a simpler setting.
00:30:39 The question here is, where are the images?
00:30:43 Why are they using CNN?
00:30:45 Where is the image here?
00:30:46 What image do you see in this case?
00:30:50 I cannot see you, RuJie, but I will just ask you.
00:30:58 Where is the image in this case?
00:31:03 The input L by 45, right?
00:31:18 So they treat it as image and use CNN.
00:31:23 How much do you agree with that treatment?
00:31:28 Is this image like the natural image?
00:31:31 Is there anything special about it?
00:31:34 So in other words, the structure in the horizontal direction is not as the natural images.
00:31:57 Is that right?
00:31:58 Right.
00:31:59 It's very sophisticated.
00:32:00 It's not even tabular data either.
00:32:02 Because you know that these first 20 are one-hot encoding.
00:32:06 So there's some, you know, dependency, you know, between these 20.
00:32:11 And then this three-dimension is also categorical, right?
00:32:15 It's one-hot again, and SA is one-hot again as well, right?
00:32:19 So it's actually a very special structure.
00:32:22 So that I think it will take the convolutional neural work a lot of data to learn this, right?
00:32:28 But nevertheless, that's what people, you know, still do.
00:32:31 But I think gradually they are being revolutionized.
00:32:34 Another example I want to show you is a 2D version of protein structure prediction.
00:32:41 What I mean by 2D prediction is let's look at the output.
00:32:45 I said that there are different ways to represent a protein structure.
00:32:49 One way is to look at the contact map.
00:32:52 If you have L amino acids in the sequence, then you can look.