Lecture 18.1: Convolutional Neural Networks (CNNs): WHY?
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 So now let's talk about a bigger picture of deep learning.
00:00:04 A bigger picture. Because if you think about where it's coming from.
00:00:08 So we've got to think bigger than just the feed-forward neural network.
00:00:13 So what else can we do?
00:00:17 So deep feed-forward neural networks, or MLPs, opened the Pandora's box of deep learning in the early 1990s.
00:00:25 And their training became practical in early 2000s. Back propagation.
00:00:31 Remember that was not apparent when Minsky wrote the book in 1969, criticizing, well critiquing, not criticizing, critiquing the perceptions.
00:00:42 Those algorithms were not known. We know that they were actually available but just not known.
00:00:48 Until 1986. And we know that there are a lot of articles about the real history of back propagation.
00:00:58 But that aside, right? Another thing I want to say is that MLPs are often used for tabular data.
00:01:04 It's like a table format where each row is an example and each column is a feature.
00:01:09 So in other words, there's not too much structure. You can swap the rows. It doesn't matter.
00:01:15 Right? Because each row is an example. You can swap the columns. Doesn't matter.
00:01:21 Because each column is a feature. There's no order among the examples.
00:01:25 And there's no order among the features. So there's not a lot of structure.
00:01:30 Right? So for this kind of tabular data, there's not a lot of structure.
00:01:34 And MLPs are often used. And actually, if you look at the Kaggle competition, whenever there's tabular data, right?
00:01:42 The frequent winners are random forest and XGBoost. Right? So not very surprising. Right?
00:01:51 But then we see that in the image or text, there are more structures than tabular data.
00:01:56 Then we can talk about how to utilize the structures. Okay?
00:02:01 So what else? As I said, the first example is image data. Right?
00:02:08 So what are the image data? Right? So image data are actually very interesting.
00:02:15 Because they are discrete samples of continuous signals. You see what I mean? Right?
00:02:21 In nature, right, these are continuous. Right?
00:02:26 But when I take a picture of the continuous object, I can only have a discrete sample. Right?
00:02:32 For instance, 800 by 600 image. Right?
00:02:37 Meaning that I have 800 pixels each on the width and on the length and the 600 on the width, for instance.
00:02:48 So I have 800 times 600 pixels arranged on the grid. Right?
00:02:53 Two-dimensional grid. So these are what image data are.
00:02:58 So apparently, this is very different from tabular data. Right?
00:03:03 Because there is order between rows and columns. Right?
00:03:08 And for each grid, right, out of 600 times 800, I can give one or multiple numerical values. Right?
00:03:17 For instance, if it's a gray scale between 0 and 1. Or it could have RGB, three values. Right?
00:03:24 Red, green, blue. Right? Or CY and MK. Right? In printer. Right?
00:03:29 Sometimes they use CY and MK. Right? Cyan, yellow, magenta, and black. Right?
00:03:35 And for each color, or later you will see that they are called channels in neural network. Right?
00:03:42 There is a single numerical value for each color or channel.
00:03:48 So now we understand that, well, this is what image data are. Right?
00:03:53 So 2D grid, each grid is one or multiple numerical values.
00:03:59 So now what if we treat images or photos as tabular data sets? Right?
00:04:07 Well, each row of a megapixel photo can be flattened.
00:04:14 Okay? So instead of one megapixel photo, right, is, for instance, one megapixel.
00:04:21 For instance, 1,000 by 1,000. Okay?
00:04:24 So what if I just flattened the rows, converted this 2D matrix into a 1D feature vector? Right?
00:04:34 Row by row. So I have six to 10 to the power of six dimensions. Right?
00:04:40 And then I just feed these 10 to the six dimension feature to the MLP. Right?
00:04:46 So this is, you know, if we just treat image data as tabular data.
00:04:52 So can I do that? Well, of course, you can do whatever you want.
00:04:56 But do you see the consequences?
00:04:59 Is that something you like to see or not?
00:05:02 What are the consequences?
00:05:04 Do you like to see this consequence or not?
00:05:06 I see someone shaking heads.
00:05:10 Ryan, any comments about this approach of treating image data as tabular data and use MLP?
00:05:21 Do you like to see the MLP to learn the image data?
00:05:28 The order in the image is important?
00:05:31 The order in the image is important.
00:05:34 It's one source of information.
00:05:37 That's one source of information?
00:05:39 Yes.
00:05:40 Yes.
00:05:41 And I more or less respected it.
00:05:44 That within each row I kept the order.
00:05:47 Just also across different rows I also kept its order.
00:05:53 The only thing that artificial is that, you know, between the last grid of the previous row and the first grid of the current row, I introduced something artificial.
00:06:06 Is that right?
00:06:10 But no matter what I do, MLP does not really utilize that order information.
00:06:15 Not to mention that I introduced some artificial order, right?
00:06:18 MLP does not care about such order information.
00:06:21 These neurons are interchangeable within each layers, right?
00:06:25 Okay, that's one critique.
00:06:27 What others?
00:06:28 So your critique is saying that the structure of the image data is not utilized.
00:06:35 Any other critique?
00:06:36 Look at this number, 10 to 6.
00:06:47 How many input neurons do you need?
00:06:51 10 to 6.
00:06:53 Is that something you desire?
00:06:57 Probably not.
00:07:01 Right?
00:07:02 So let's say you have one hidden layer with 10 to 3 neurons.
00:07:06 How many parameters do you end up having?
00:07:11 How many parameters?
00:07:16 You only have one hidden layer.
00:07:18 Okay, input layer has 10 to 6 neurons.
00:07:21 Hidden layer is 10 to 3.
00:07:24 Small, okay.
00:07:25 And output, let's say you're doing the binary classification, whether it's a cat or not.
00:07:31 So let's just say there's a single O.
00:07:34 So how many parameters do you roughly have?
00:07:40 What's the dimensionality of your W?
00:07:42 Huh?
00:07:46 Multiplication of between 10 to 6 and 10 to 3.
00:07:53 So that's the number of edges here.
00:07:56 Is that right?
00:07:57 Plus this multiplication of 10 to 3 times one, but that's much smaller.
00:08:04 So it's order of 10 to 9 times some constant because there's bias as well.
00:08:09 Right?
00:08:10 But it's in the order of 10 to 9.
00:08:12 Right?
00:08:13 So for just a megapixel photo, right?
00:08:16 You need the 10 to 9 model.
00:08:18 That means what?
00:08:19 To train this extremely large model accurately.
00:08:24 You need a lot of data.
00:08:27 A lot of GPUs and extreme parallelization for optimization.
00:08:33 Right?
00:08:34 You can do that, right?
00:08:36 But it comes with a cost.
00:08:38 So instead, right?
00:08:43 So if we conclude that MLPs for image data are extremely complex models that demand data and computing, then what do we do?
00:08:55 We learned before that, you know, if your data is not something you can change, right?
00:09:04 And your model is too complex for the given data, you have to control the complexity or regularize the complexity.
00:09:12 Right?
00:09:13 So how do you regularize MLPs in this case?
00:09:23 In other words, how do you change MLPs so that it works for the image data?
00:09:27 Right?
00:09:30 So to answer that question, let's go back to the biology and ask how do our vision system works so well.
00:09:47 Right?
00:09:48 Our vision systems works so well.
00:09:49 How do they work?
00:09:50 Can we emulate that and then regularize or change our MLPs accordingly?
00:09:55 So that's a question, right?
00:09:58 Right?
00:09:59 Because we cannot just, you know, pat our heads and they come up with a new architecture.
00:10:04 You see what I mean?
00:10:06 So, this is one example, right?
00:10:09 How our vision systems work.
00:10:10 This is our target called a Waldo.
00:10:13 Okay?
00:10:14 Well, I'll give you one minute to tell me where Waldo is in this image.
00:10:25 Let me move away this to be less distracting.
00:10:33 Okay, you're going to pay attention to the pattern of the target and ask yourself, what's the best way to search?
00:10:43 Right?
00:10:44 You are actually having a neural network in your brain now, which is being trained previously.
00:10:52 Right?
00:10:53 Now you are transferred that neural network to identify Waldo.
00:11:00 Where is Waldo?
00:11:02 Any luck?
00:11:16 Let's give it another 30 seconds.
00:11:36 Anyone with success?
00:11:41 I remember last year it was Randy who found it first.
00:11:51 Some of you know Randy.
00:11:54 Okay, let's stop here.
00:12:04 And the important thing is not about the answer.
00:12:09 Okay?
00:12:10 The important thing is you rationally get aware of how your vision systems work.
00:12:24 That's the point.
00:12:25 Okay?
00:12:26 If you ask about the answer, the answer is here.
00:12:28 Do you see it?
00:12:29 Do you see it?
00:12:31 Do you see him?
00:12:34 Caught by the savage?
00:12:38 And bonded?
00:12:40 See him now?
00:12:42 Okay.
00:12:43 So, that said, the answer is not about where is Waldo.
00:12:52 The question is how did our eyes and brain work together to find Waldo?
00:12:58 Right?
00:12:59 Right?
00:13:00 So, there are a few things.
00:13:03 Right?
00:13:04 So, how did we really find Waldo?
00:13:08 Right?
00:13:09 First of all, right?
00:13:10 This, there's got to be something, right?
00:13:15 You know, there's got to be something about the fact that the target is invariant to something.
00:13:24 Right?
00:13:25 So, in other words, it doesn't matter where the target appears translationally in the image.
00:13:32 Right?
00:13:33 Or it's perturbed a little bit.
00:13:36 Right?
00:13:37 And it doesn't matter.
00:13:40 So, there's something invariant to the transformations.
00:13:43 In this case, translations or rotations.
00:13:46 Right?
00:13:47 You see these red and white stripes and the hats.
00:13:51 Right?
00:13:52 And this part is known in the target and it's invariant to where this image is.
00:14:01 Rotationally or translationally.
00:14:03 Right?
00:14:04 And secondly, you're going to have to see through the destructor.
00:14:08 In this case, you know, he has been attached, you know, he's bound with a lot of the strings
00:14:14 and then that is distracting you.
00:14:16 You have to discriminate against such distractions.
00:14:19 Right?
00:14:20 And then you will have to do something more efficiently.
00:14:23 Somehow, you have to figure out which blocks and how do I search block by block or, you know, how do I search?
00:14:29 Right?
00:14:30 So, this.
00:14:31 And a little bit prior training.
00:14:33 Right?
00:14:34 Because you have learned how to search for objects.
00:14:37 But in this case, it's a novel object.
00:14:39 It's Waldo.
00:14:40 But then it's a little bit transferring from your prior to the new task.
00:14:46 Right?
00:14:47 This is what we learned about how our vision systems work.
00:14:49 So now let's translate it.
00:14:51 So now, you know, the reason that eventually we went from the dense layers to convolutions, right, is based on the two observations or knowledge about our vision systems.
00:15:08 Okay?
00:15:09 First is that they respond to similarly to the same object, in this case, Waldo, regardless of where it appears in the image.
00:15:19 Okay?
00:15:20 No matter where it appears, as long as our eyes see that pattern, you know, it knows that it's there.
00:15:27 Right?
00:15:28 So, in other words, it's translational invariant.
00:15:31 Okay?
00:15:32 So the output signal is invariant to the translation of Waldo's positions.
00:15:40 Secondly, they focus on local regions without regard for what else is happening in the image at a greater distance.
00:15:48 So, in other words, you know, when finding this object, right, you very often, you know, restrict to this, you know,
00:15:56 restrict to this locality pattern, right?
00:15:59 Right?
00:16:00 Where you need to find this small object, right?
00:16:03 So, these are the two observations, translation, invariance, and locality, right?
00:16:09 So now, is the question.
00:16:12 If we want to emulate the vision systems, how do we translate this two knowledge into mathematics and regularize MLPs?
00:16:24 If convolutions haven't been invented yet, how would you figure that out?
00:16:32 Right?
00:16:33 So, before I give you the answer, right, I'd like to say that, you know, deep learning has so many different architectures, as we know now.
00:16:46 There is feedforward that is already introduced, but there is also feedback or bidirectional or recurrent neural networks, right?
00:16:56 A lot of possibilities.
00:16:58 And we talked about perceptrons and feedforward, and there are other possibilities, right?
00:17:08 [Uncertain recognition] There are lots of possibilities, right?