Lecture 20.0: CNN Recap
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 Second kind of data and the corresponding neural network architecture.
00:00:06 So we're now talking about text or sequence data.
00:00:11 And the corresponding architecture we're going to talk about is recurrent neural networks or RNNs.
00:00:21 So last time we talked about CNNs.
00:00:26 So we said that if you look at how ImageNet has enabled the development of computer vision models, especially CNNs.
00:00:38 We have seen that from the AlexNet, this is 2012, we see the first big major improvement.
00:00:51 That was thanks to the AlexNet.
00:00:55 But I also reminded you, I think I used one slide to tell you that the first CNN by Yang LeCun or LeNet was in 19...
00:01:09 Let's see.
00:01:13 Late 80s or early 90s, I recall.
00:01:16 So the question is, you know, apparently they didn't have contributions to earlier ImageNet.
00:01:26 Then what happened?
00:01:27 Right?
00:01:28 So I think I used some slide earlier to tell you why that was the time, year 2012, was the time for CNNs to ripen for computer vision.
00:01:40 Right?
00:01:41 One is, you know, the hardware and this vectorization, you know, GPU tensor calculation gets much more effective.
00:01:49 But there's also another fact that is, you know, these neural networks are data driven and you need high quality data.
00:01:57 Right?
00:01:58 ImageNet actually provided a lot of, you know, high quality data that has hierarchical labels.
00:02:05 Right?
00:02:06 So, and over the years, over the years, you know, starting from the AlexNet, we see a lot of improvement.
00:02:17 Right?
00:02:18 So including the VGG in year 2014.
00:02:22 Right?
00:02:23 And this is what VGG 16 or 19.
00:02:27 Right?
00:02:28 Then you also have the Google Net in year 2014.
00:02:34 Right?
00:02:35 So, you know, then this is the performance.
00:02:41 Right?
00:02:42 And if we compare them to the human error, actually they even outperform human errors.
00:02:49 Because sometimes for a random human, it's really hard for you to find out some fine labels.
00:02:57 Right?
00:02:58 For instance, you may not know well the difference between two breeds of dogs.
00:03:04 Right?
00:03:05 Some of you know well, some of you don't.
00:03:07 So, but for computer models, as long as they have enough examples, they can learn that very well.
00:03:14 So, and you know, this revolution of depth, right?
00:03:23 So we see that these models are, you know, getting deeper, but there are more things.
00:03:29 Right?
00:03:30 So for instance, the residual blocks.
00:03:32 So, if you recall last time I talked about the treatment of protein sequence data and the corresponding features as image.
00:03:44 And also the task of contact prediction as a pixel level segmentation, image segmentation.
00:03:54 Because each row is a residue index, each column is a residue index.
00:03:59 So that is artificial image at the end that we need to figure out at each pixel, the corresponding label being one or zero.
00:04:08 For residues being contacting or not.
00:04:11 Or you could even use two channels.
00:04:14 One channel is a distance, one channel is a distance expected value.
00:04:19 One is a channel, one channel is a distance standard deviation or something like that.
00:04:24 Right?
00:04:25 So it's like the image task of pixel level segmentation.
00:04:33 Back then, they already used the residual blocks back then.
00:04:38 Right?
00:04:39 So the residual blocks is allowing you to have these identity operations.
00:04:43 Right?
00:04:44 Going bypass these kind of jump connections.
00:04:46 Right?
00:04:47 And you can use these blocks sequentially in this ResNet.
00:04:53 Right?
00:04:54 So that's for the ResNet.
00:04:55 And there is, of course, in 2017 there is a SENet.
00:05:00 Right?
00:05:01 So if you're interested in, you can look into them into more details.
00:05:07 So now, let's look at what complexity brings in.
00:05:14 Right?
00:05:15 So is this just a game of bigger and bigger models?
00:05:20 Right?
00:05:21 Let's look at some analysis.
00:05:23 Right?
00:05:24 And let's see what implications we reach.
00:05:27 So this is the top one accuracy for different models.
00:05:32 Right?
00:05:33 The higher the better.
00:05:34 Right?
00:05:35 And you see that starting from the AlexNet and all the way into later like a VGG, ResNet, and Inception.
00:05:42 The accuracy level is increasing.
00:05:45 But now let's compare, right?
00:05:49 These accuracy still the y-axis.
00:05:51 But now let's change the x-axis.
00:05:53 Right?
00:05:54 Instead of ordering these models in a scale-free way, now let's order the models x-axis by the number of operations.
00:06:04 Right?
00:06:05 By the number of operations.
00:06:07 In the meantime, the dot, you know, or the point here.
00:06:11 Right?
00:06:12 We have the coordinate of the number of operations versus accuracy.
00:06:16 Right?
00:06:17 In the meantime, we also can, you know, change the radius of the dot based on how many parameters we have.
00:06:24 For instance, the VGG 16, right?
00:06:25 Right?
00:06:26 It has about 125 million parameters.
00:06:27 Right?
00:06:28 And so on.
00:06:29 Right?
00:06:30 So now we can compare the performance versus the model complexity.
00:06:39 But of course, as I said, counting the number of parameters is not, you know, right away.
00:06:47 Right?
00:06:48 But the number of operations is something that we can calculate all the time.
00:06:53 And the number of parameters, that's something we can figure out.
00:06:56 But whether it's too complex or not, that is relative to the data.
00:07:00 Right?
00:07:01 So AlexNet is here.
00:07:05 It's actually here.
00:07:06 It actually have 60 million parameters and accuracy is not that high.
00:07:11 Right?
00:07:12 And, but the number of operations is not that high either.
00:07:16 Right?
00:07:17 So you see that the later networks such as GoogleNet, ResNet, right?
00:07:24 So they still don't have, you know, a lot of parameters.
00:07:28 Because they keep reusing some building blocks of the same parameters.
00:07:34 So they do have these, you know, parameter sharing ideas kept used.
00:07:42 Right?
00:07:43 And operations increase a little bit because you still have to compose these repeating units together.
00:07:49 Right?
00:07:50 And then you see some, some other cases like Inception.
00:07:55 Inception's number of operations and the number of parameters are both, you know, very high.
00:08:04 Then, you know, VGG is, you know, is even higher in both the number of parameters and the number of operators.
00:08:12 [Uncertain recognition] Operations.
00:08:13 [Uncertain recognition] Sorry.