Lecture 20.0: CNN Recap

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Second kind of data and the corresponding neural network architecture.

00:00:06 So we're now talking about text or sequence data.

00:00:11 And the corresponding architecture we're going to talk about is recurrent neural networks or RNNs.

00:00:21 So last time we talked about CNNs.

00:00:26 So we said that if you look at how ImageNet has enabled the development of computer vision models, especially CNNs.

00:00:38 We have seen that from the AlexNet, this is 2012, we see the first big major improvement.

00:00:51 That was thanks to the AlexNet.

00:00:55 But I also reminded you, I think I used one slide to tell you that the first CNN by Yang LeCun or LeNet was in 19...

00:01:09 Let's see.

00:01:13 Late 80s or early 90s, I recall.

00:01:16 So the question is, you know, apparently they didn't have contributions to earlier ImageNet.

00:01:26 Then what happened?

00:01:27 Right?

00:01:28 So I think I used some slide earlier to tell you why that was the time, year 2012, was the time for CNNs to ripen for computer vision.

00:01:40 Right?

00:01:41 One is, you know, the hardware and this vectorization, you know, GPU tensor calculation gets much more effective.

00:01:49 But there's also another fact that is, you know, these neural networks are data driven and you need high quality data.

00:01:57 Right?

00:01:58 ImageNet actually provided a lot of, you know, high quality data that has hierarchical labels.

00:02:05 Right?

00:02:06 So, and over the years, over the years, you know, starting from the AlexNet, we see a lot of improvement.

00:02:17 Right?

00:02:18 So including the VGG in year 2014.

00:02:22 Right?

00:02:23 And this is what VGG 16 or 19.

00:02:27 Right?

00:02:28 Then you also have the Google Net in year 2014.

00:02:34 Right?

00:02:35 So, you know, then this is the performance.

00:02:41 Right?

00:02:42 And if we compare them to the human error, actually they even outperform human errors.

00:02:49 Because sometimes for a random human, it's really hard for you to find out some fine labels.

00:02:57 Right?

00:02:58 For instance, you may not know well the difference between two breeds of dogs.

00:03:04 Right?

00:03:05 Some of you know well, some of you don't.

00:03:07 So, but for computer models, as long as they have enough examples, they can learn that very well.

00:03:14 So, and you know, this revolution of depth, right?

00:03:23 So we see that these models are, you know, getting deeper, but there are more things.

00:03:29 Right?

00:03:30 So for instance, the residual blocks.

00:03:32 So, if you recall last time I talked about the treatment of protein sequence data and the corresponding features as image.

00:03:44 And also the task of contact prediction as a pixel level segmentation, image segmentation.

00:03:54 Because each row is a residue index, each column is a residue index.

00:03:59 So that is artificial image at the end that we need to figure out at each pixel, the corresponding label being one or zero.

00:04:08 For residues being contacting or not.

00:04:11 Or you could even use two channels.

00:04:14 One channel is a distance, one channel is a distance expected value.

00:04:19 One is a channel, one channel is a distance standard deviation or something like that.

00:04:24 Right?

00:04:25 So it's like the image task of pixel level segmentation.

00:04:33 Back then, they already used the residual blocks back then.

00:04:38 Right?

00:04:39 So the residual blocks is allowing you to have these identity operations.

00:04:43 Right?

00:04:44 Going bypass these kind of jump connections.

00:04:46 Right?

00:04:47 And you can use these blocks sequentially in this ResNet.

00:04:53 Right?

00:04:54 So that's for the ResNet.

00:04:55 And there is, of course, in 2017 there is a SENet.

00:05:00 Right?

00:05:01 So if you're interested in, you can look into them into more details.

00:05:07 So now, let's look at what complexity brings in.

00:05:14 Right?

00:05:15 So is this just a game of bigger and bigger models?

00:05:20 Right?

00:05:21 Let's look at some analysis.

00:05:23 Right?

00:05:24 And let's see what implications we reach.

00:05:27 So this is the top one accuracy for different models.

00:05:32 Right?

00:05:33 The higher the better.

00:05:34 Right?

00:05:35 And you see that starting from the AlexNet and all the way into later like a VGG, ResNet, and Inception.

00:05:42 The accuracy level is increasing.

00:05:45 But now let's compare, right?

00:05:49 These accuracy still the y-axis.

00:05:51 But now let's change the x-axis.

00:05:53 Right?

00:05:54 Instead of ordering these models in a scale-free way, now let's order the models x-axis by the number of operations.

00:06:04 Right?

00:06:05 By the number of operations.

00:06:07 In the meantime, the dot, you know, or the point here.

00:06:11 Right?

00:06:12 We have the coordinate of the number of operations versus accuracy.

00:06:16 Right?

00:06:17 In the meantime, we also can, you know, change the radius of the dot based on how many parameters we have.

00:06:24 For instance, the VGG 16, right?

00:06:25 Right?

00:06:26 It has about 125 million parameters.

00:06:27 Right?

00:06:28 And so on.

00:06:29 Right?

00:06:30 So now we can compare the performance versus the model complexity.

00:06:39 But of course, as I said, counting the number of parameters is not, you know, right away.

00:06:47 Right?

00:06:48 But the number of operations is something that we can calculate all the time.

00:06:53 And the number of parameters, that's something we can figure out.

00:06:56 But whether it's too complex or not, that is relative to the data.

00:07:00 Right?

00:07:01 So AlexNet is here.

00:07:05 It's actually here.

00:07:06 It actually have 60 million parameters and accuracy is not that high.

00:07:11 Right?

00:07:12 And, but the number of operations is not that high either.

00:07:16 Right?

00:07:17 So you see that the later networks such as GoogleNet, ResNet, right?

00:07:24 So they still don't have, you know, a lot of parameters.

00:07:28 Because they keep reusing some building blocks of the same parameters.

00:07:34 So they do have these, you know, parameter sharing ideas kept used.

00:07:42 Right?

00:07:43 And operations increase a little bit because you still have to compose these repeating units together.

00:07:49 Right?

00:07:50 And then you see some, some other cases like Inception.

00:07:55 Inception's number of operations and the number of parameters are both, you know, very high.

00:08:04 Then, you know, VGG is, you know, is even higher in both the number of parameters and the number of operators.

00:08:12 [Uncertain recognition] Operations.

00:08:13 [Uncertain recognition] Sorry.