# Lecture 18.1: Convolutional Neural Networks (CNNs)： WHY？

[Recording](https://www.youtube.com/watch?v=PfELfF53mOU)

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 So now let's talk about a bigger picture of deep learning.

00:00:04 A bigger picture. Because if you think about where it's coming from.

00:00:08 So we've got to think bigger than just the feed-forward neural network.

00:00:13 So what else can we do?

00:00:17 So deep feed-forward neural networks, or MLPs, opened the Pandora's box of deep learning in the early 1990s.

00:00:25 And their training became practical in early 2000s. Back propagation.

00:00:31 Remember that was not apparent when Minsky wrote the book in 1969, criticizing, well critiquing, not criticizing, critiquing the perceptions.

00:00:42 Those algorithms were not known. We know that they were actually available but just not known.

00:00:48 Until 1986. And we know that there are a lot of articles about the real history of back propagation.

00:00:58 But that aside, right? Another thing I want to say is that MLPs are often used for tabular data.

00:01:04 It's like a table format where each row is an example and each column is a feature.

00:01:09 So in other words, there's not too much structure. You can swap the rows. It doesn't matter.

00:01:15 Right? Because each row is an example. You can swap the columns. Doesn't matter.

00:01:21 Because each column is a feature. There's no order among the examples.

00:01:25 And there's no order among the features. So there's not a lot of structure.

00:01:30 Right? So for this kind of tabular data, there's not a lot of structure.

00:01:34 And MLPs are often used. And actually, if you look at the Kaggle competition, whenever there's tabular data, right?

00:01:42 The frequent winners are random forest and XGBoost. Right? So not very surprising. Right?

00:01:51 But then we see that in the image or text, there are more structures than tabular data.

00:01:56 Then we can talk about how to utilize the structures. Okay?

00:02:01 So what else? As I said, the first example is image data. Right?

00:02:08 So what are the image data? Right? So image data are actually very interesting.

00:02:15 Because they are discrete samples of continuous signals. You see what I mean? Right?

00:02:21 In nature, right, these are continuous. Right?

00:02:26 But when I take a picture of the continuous object, I can only have a discrete sample. Right?

00:02:32 For instance, 800 by 600 image. Right?

00:02:37 Meaning that I have 800 pixels each on the width and on the length and the 600 on the width, for instance.

00:02:48 So I have 800 times 600 pixels arranged on the grid. Right?

00:02:53 Two-dimensional grid. So these are what image data are.

00:02:58 So apparently, this is very different from tabular data. Right?

00:03:03 Because there is order between rows and columns. Right?

00:03:08 And for each grid, right, out of 600 times 800, I can give one or multiple numerical values. Right?

00:03:17 For instance, if it's a gray scale between 0 and 1. Or it could have RGB, three values. Right?

00:03:24 Red, green, blue. Right? Or CY and MK. Right? In printer. Right?

00:03:29 Sometimes they use CY and MK. Right? Cyan, yellow, magenta, and black. Right?

00:03:35 And for each color, or later you will see that they are called channels in neural network. Right?

00:03:42 There is a single numerical value for each color or channel.

00:03:48 So now we understand that, well, this is what image data are. Right?

00:03:53 So 2D grid, each grid is one or multiple numerical values.

00:03:59 So now what if we treat images or photos as tabular data sets? Right?

00:04:07 Well, each row of a megapixel photo can be flattened.

00:04:14 Okay? So instead of one megapixel photo, right, is, for instance, one megapixel.

00:04:21 For instance, 1,000 by 1,000. Okay?

00:04:24 So what if I just flattened the rows, converted this 2D matrix into a 1D feature vector? Right?

00:04:34 Row by row. So I have six to 10 to the power of six dimensions. Right?

00:04:40 And then I just feed these 10 to the six dimension feature to the MLP. Right?

00:04:46 So this is, you know, if we just treat image data as tabular data.

00:04:52 So can I do that? Well, of course, you can do whatever you want.

00:04:56 But do you see the consequences?

00:04:59 Is that something you like to see or not?

00:05:02 What are the consequences?

00:05:04 Do you like to see this consequence or not?

00:05:06 I see someone shaking heads.

00:05:10 Ryan, any comments about this approach of treating image data as tabular data and use MLP?

00:05:21 Do you like to see the MLP to learn the image data?

00:05:28 The order in the image is important?

00:05:31 The order in the image is important.

00:05:34 It's one source of information.

00:05:37 That's one source of information?

00:05:39 Yes.

00:05:40 Yes.

00:05:41 And I more or less respected it.

00:05:44 That within each row I kept the order.

00:05:47 Just also across different rows I also kept its order.

00:05:53 The only thing that artificial is that, you know, between the last grid of the previous row and the first grid of the current row, I introduced something artificial.

00:06:06 Is that right?

00:06:10 But no matter what I do, MLP does not really utilize that order information.

00:06:15 Not to mention that I introduced some artificial order, right?

00:06:18 MLP does not care about such order information.

00:06:21 These neurons are interchangeable within each layers, right?

00:06:25 Okay, that's one critique.

00:06:27 What others?

00:06:28 So your critique is saying that the structure of the image data is not utilized.

00:06:35 Any other critique?

00:06:36 Look at this number, 10 to 6.

00:06:47 How many input neurons do you need?

00:06:51 10 to 6.

00:06:53 Is that something you desire?

00:06:57 Probably not.

00:07:01 Right?

00:07:02 So let's say you have one hidden layer with 10 to 3 neurons.

00:07:06 How many parameters do you end up having?

00:07:11 How many parameters?

00:07:16 You only have one hidden layer.

00:07:18 Okay, input layer has 10 to 6 neurons.

00:07:21 Hidden layer is 10 to 3.

00:07:24 Small, okay.

00:07:25 And output, let's say you're doing the binary classification, whether it's a cat or not.

00:07:31 So let's just say there's a single O.

00:07:34 So how many parameters do you roughly have?

00:07:40 What's the dimensionality of your W?

00:07:42 Huh?

00:07:46 Multiplication of between 10 to 6 and 10 to 3.

00:07:53 So that's the number of edges here.

00:07:56 Is that right?

00:07:57 Plus this multiplication of 10 to 3 times one, but that's much smaller.

00:08:04 So it's order of 10 to 9 times some constant because there's bias as well.

00:08:09 Right?

00:08:10 But it's in the order of 10 to 9.

00:08:12 Right?

00:08:13 So for just a megapixel photo, right?

00:08:16 You need the 10 to 9 model.

00:08:18 That means what?

00:08:19 To train this extremely large model accurately.

00:08:24 You need a lot of data.

00:08:27 A lot of GPUs and extreme parallelization for optimization.

00:08:33 Right?

00:08:34 You can do that, right?

00:08:36 But it comes with a cost.

00:08:38 So instead, right?

00:08:43 So if we conclude that MLPs for image data are extremely complex models that demand data and computing, then what do we do?

00:08:55 We learned before that, you know, if your data is not something you can change, right?

00:09:04 And your model is too complex for the given data, you have to control the complexity or regularize the complexity.

00:09:12 Right?

00:09:13 So how do you regularize MLPs in this case?

00:09:23 In other words, how do you change MLPs so that it works for the image data?

00:09:27 Right?

00:09:30 So to answer that question, let's go back to the biology and ask how do our vision system works so well.

00:09:47 Right?

00:09:48 Our vision systems works so well.

00:09:49 How do they work?

00:09:50 Can we emulate that and then regularize or change our MLPs accordingly?

00:09:55 So that's a question, right?

00:09:58 Right?

00:09:59 Because we cannot just, you know, pat our heads and they come up with a new architecture.

00:10:04 You see what I mean?

00:10:06 So, this is one example, right?

00:10:09 How our vision systems work.

00:10:10 This is our target called a Waldo.

00:10:13 Okay?

00:10:14 Well, I'll give you one minute to tell me where Waldo is in this image.

00:10:25 Let me move away this to be less distracting.

00:10:33 Okay, you're going to pay attention to the pattern of the target and ask yourself, what's the best way to search?

00:10:43 Right?

00:10:44 You are actually having a neural network in your brain now, which is being trained previously.

00:10:52 Right?

00:10:53 Now you are transferred that neural network to identify Waldo.

00:11:00 Where is Waldo?

00:11:02 Any luck?

00:11:16 Let's give it another 30 seconds.

00:11:36 Anyone with success?

00:11:41 I remember last year it was Randy who found it first.

00:11:51 Some of you know Randy.

00:11:54 Okay, let's stop here.

00:12:04 And the important thing is not about the answer.

00:12:09 Okay?

00:12:10 The important thing is you rationally get aware of how your vision systems work.

00:12:24 That's the point.

00:12:25 Okay?

00:12:26 If you ask about the answer, the answer is here.

00:12:28 Do you see it?

00:12:29 Do you see it?

00:12:31 Do you see him?

00:12:34 Caught by the savage?

00:12:38 And bonded?

00:12:40 See him now?

00:12:42 Okay.

00:12:43 So, that said, the answer is not about where is Waldo.

00:12:52 The question is how did our eyes and brain work together to find Waldo?

00:12:58 Right?

00:12:59 Right?

00:13:00 So, there are a few things.

00:13:03 Right?

00:13:04 So, how did we really find Waldo?

00:13:08 Right?

00:13:09 First of all, right?

00:13:10 This, there's got to be something, right?

00:13:15 You know, there's got to be something about the fact that the target is invariant to something.

00:13:24 Right?

00:13:25 So, in other words, it doesn't matter where the target appears translationally in the image.

00:13:32 Right?

00:13:33 Or it's perturbed a little bit.

00:13:36 Right?

00:13:37 And it doesn't matter.

00:13:40 So, there's something invariant to the transformations.

00:13:43 In this case, translations or rotations.

00:13:46 Right?

00:13:47 You see these red and white stripes and the hats.

00:13:51 Right?

00:13:52 And this part is known in the target and it's invariant to where this image is.

00:14:01 Rotationally or translationally.

00:14:03 Right?

00:14:04 And secondly, you're going to have to see through the destructor.

00:14:08 In this case, you know, he has been attached, you know, he's bound with a lot of the strings

00:14:14 and then that is distracting you.

00:14:16 You have to discriminate against such distractions.

00:14:19 Right?

00:14:20 And then you will have to do something more efficiently.

00:14:23 Somehow, you have to figure out which blocks and how do I search block by block or, you know, how do I search?

00:14:29 Right?

00:14:30 So, this.

00:14:31 And a little bit prior training.

00:14:33 Right?

00:14:34 Because you have learned how to search for objects.

00:14:37 But in this case, it's a novel object.

00:14:39 It's Waldo.

00:14:40 But then it's a little bit transferring from your prior to the new task.

00:14:46 Right?

00:14:47 This is what we learned about how our vision systems work.

00:14:49 So now let's translate it.

00:14:51 So now, you know, the reason that eventually we went from the dense layers to convolutions, right, is based on the two observations or knowledge about our vision systems.

00:15:08 Okay?

00:15:09 First is that they respond to similarly to the same object, in this case, Waldo, regardless of where it appears in the image.

00:15:19 Okay?

00:15:20 No matter where it appears, as long as our eyes see that pattern, you know, it knows that it's there.

00:15:27 Right?

00:15:28 So, in other words, it's translational invariant.

00:15:31 Okay?

00:15:32 So the output signal is invariant to the translation of Waldo's positions.

00:15:40 Secondly, they focus on local regions without regard for what else is happening in the image at a greater distance.

00:15:48 So, in other words, you know, when finding this object, right, you very often, you know, restrict to this, you know,

00:15:56 restrict to this locality pattern, right?

00:15:59 Right?

00:16:00 Where you need to find this small object, right?

00:16:03 So, these are the two observations, translation, invariance, and locality, right?

00:16:09 So now, is the question.

00:16:12 If we want to emulate the vision systems, how do we translate this two knowledge into mathematics and regularize MLPs?

00:16:24 If convolutions haven't been invented yet, how would you figure that out?

00:16:32 Right?

00:16:33 So, before I give you the answer, right, I'd like to say that, you know, deep learning has so many different architectures, as we know now.

00:16:46 There is feedforward that is already introduced, but there is also feedback or bidirectional or recurrent neural networks, right?

00:16:56 A lot of possibilities.

00:16:58 And we talked about perceptrons and feedforward, and there are other possibilities, right?

00:17:08 **[Uncertain recognition]** There are lots of possibilities, right?
