Lecture 18.2: Convolutional Neural Networks (CNNs): HOW?
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 Okay, now as described earlier,
00:00:04 we're gonna finish the convolutional neural networks.
00:00:09 And as a reminder, what we talked about in the last lecture
00:00:12 was artificial neural networks, right?
00:00:15 So we have learned what are the key concepts
00:00:18 about neurons, layers, activation functions.
00:00:21 We also learned about the special kind
00:00:24 of artificial neural networks, including linear regression,
00:00:28 including multinomial logistic regression,
00:00:31 including MLP, including SVM, right?
00:00:35 So we learned those special cases of artificial neural networks
00:00:39 from the key aspects of optimization, right?
00:00:43 There is a formulation that involves the data
00:00:46 and the model form, right?
00:00:49 Linear or other kind.
00:00:51 Then there is a objective function or loss function
00:00:55 in the case of machine learning or training, right?
00:00:58 And lastly, you need an optimizer,
00:00:59 an optimization algorithm that can solve the problem, right?
00:01:02 Don't forget that, you know, even for MLP,
00:01:07 this algorithm was not made aware of until 1986, right?
00:01:13 Even though MLP has been known much longer than that.
00:01:18 So you do need all these elements of the optimization
00:01:24 when you think about deep learning.
00:01:28 Besides those, we also look at deep learning neural networks
00:01:32 from different perspectives, right?
00:01:34 We look at that, besides mathematical formulation,
00:01:38 we also talked about the implementation, right?
00:01:42 So in the code, how you should scale the calculations, right?
00:01:47 It's not desirable to use, you know, some or other things,
00:01:51 but it's much more desirable to use those functions,
00:01:55 including the matrix product that can be already well-paralyzed
00:01:59 in the architecture of GPUs, right?
00:02:01 So that also matters.
00:02:03 And lastly, with math and coding in mind,
00:02:09 you will be able to understand what these illustrations
00:02:13 of these architectures mean, right?
00:02:16 So now, today, let's go back to where we started
00:02:22 and finish the convolutional neural networks.
00:02:26 So we started with this motivation question, right?
00:02:29 Motivating question.
00:02:31 Is it desirable to use MLP for image data?
00:02:37 Right, as we know, MLP is very suitable for tabular data.
00:02:41 But we also know that there is a fundamental difference
00:02:44 between the tabular data and the image data.
00:02:46 There's no row order or column order in tabular data,
00:02:51 but there is such structure in the image data, right?
00:02:55 And we also learned that it's not really desirable
00:02:57 to use an MLP for image data,
00:03:01 not only because you are ignoring such structure,
00:03:04 but also because you have to use, like, you know,
00:03:09 a million input neurons if your image is like a million pixel,
00:03:14 right, and then you have these, you know, parameters,
00:03:18 which is, you know, roughly equal to the product
00:03:21 of the previous layer's neurons number
00:03:24 and the next layer's neuron number, right?
00:03:27 In other words, you have such a complex model
00:03:30 that demand, right, a lot of data,
00:03:35 a lot of the training resources and the parallelization, right?
00:03:39 Of course, you know, there are more sophisticated research
00:03:44 about it, especially about the over-parametrized
00:03:47 the deep learning models, but, you know,
00:03:50 let's not go there for the moment.
00:03:52 So then how do we know from the shallow machine learning
00:03:57 that when the model is over-complex with regard to the data,
00:04:01 you want to regularize it, right?
00:04:03 But the question is, how do you regularize?
00:04:05 Back then, you have L1 or L2 or LP norm, right?
00:04:09 We knew about that.
00:04:11 Is that the only thing we can do?
00:04:12 Because if you just use the LP norm on top of this MLP,
00:04:16 you still have too much to compute, right?
00:04:20 So how else can you regularize, right?
00:04:25 So the answer is introducing new architectures
00:04:28 of neural networks.
00:04:29 I have shown some figures before telling you
00:04:33 that essentially, you know, artificial neural networks
00:04:35 really, really very broad, very broad, right?
00:04:40 These are just a few examples that, you know,
00:04:43 if you see them all in the light of neural networks, right?
00:04:47 The question is, you know, how do you come up with it?
00:04:50 Or if you are a rational user,
00:04:51 in what case do you use what, right?
00:04:55 So the answer for the image data comes from the following.
00:04:59 Can we learn from how our vision systems work
00:05:03 and translate that knowledge into the desired mathematics, okay?
00:05:12 And then translate that into a new layer.
00:05:14 So that was the answer.
00:05:16 And that was how the convolutional neural networks
00:05:20 were, you know, brought forth.
00:05:22 And if you know about this gentleman...
00:05:25 Okay, so can anyone tell me who this gentleman is?
00:05:40 You must know.
00:05:42 Don't you follow his Twitter?
00:05:45 Yang-Lakun.
00:05:50 Yang-Lakun.
00:05:51 Okay, Yang-Lakun.
00:05:54 Lacun, et cetera.
00:05:56 So if you know about the history of convolutional neural networks,
00:06:02 you know that this is the so-called LENET, right?
00:06:06 So it starts with something I will introduce, right?
00:06:10 It started with the input image, but what you do is not like MLP,
00:06:13 which is dot product, you're actually doing a convolution.
00:06:17 Then you keep subsampling and convolution.
00:06:20 In the end, you have a fully connected readout, right?
00:06:25 So, and then, you know, the convolutional neural networks,
00:06:31 a few key concepts is that it's a feed-forward neural network,
00:06:35 but what you do to the input is to do the convolution, okay?
00:06:41 And you can still use the nonlinearity, like the activation function.
00:06:45 At the end, you will also have some pooling, right?
00:06:48 So that's the key concepts, and we'll talk about these parts.
00:06:52 And as we said before, these filters are not designed by principle,
00:06:58 but they are actually learned from data, right?
00:07:01 Specifically by optimizing the loss functions through backpropagating, right?
00:07:09 Okay, so what are really, you know, desired in this convolutional neural networks, right?
00:07:16 So there are a few key things here, right?
00:07:19 So they can scale up to process different size of these images.
00:07:25 They have very sparse connections, like, you know, not too many parameters.
00:07:29 Better than that, they even share parameters, okay?
00:07:34 And then they can automatically generalize across different spatial translations, right?
00:07:39 Translational invariant, and they're applicable to input that is laid out on a grid,
00:07:44 no matter what it is, 1D, 2D, and 3D.
00:07:46 As in our journal club, we have all kinds of examples.
00:07:51 And the key idea,
00:07:53 So I'm giving you directly the answer to my earlier question, right?
00:07:57 What kind of properties are demanded from the mathematics to obey or conform to the knowledge of our vision system, right?
00:08:08 So what you do here is that you replace the matrix multiplication MLP with convolution, right?
00:08:15 And everything else stays the same, right?
00:08:18 For instance, if your loss function, if it was maximum likelihood, or it was some other objective functions, that stays the same.
00:08:29 Your optimizer stays the same, back propagation and so on, right?
00:08:32 So it's a very simple idea, but there's way more than that.
00:08:38 So now let's talk about the first key concept, right?
00:08:42 That is replace the matrix multiplication with convolution, okay?
00:08:46 So this is a very simple example of matrix products that is often seen, right?
00:08:51 In the MLP, right?
00:08:54 So if you have A and B, you have the matrix product, right?
00:08:58 So there is a summation here somehow, it's missing.
00:09:02 And then you know that Cij, right, is essentially, you know, the sum of the products.
00:09:09 It's like the inner, this element is the inner product of this row and this column, right?
00:09:15 So this is what matrix product is.
00:09:17 And then this is what matrix transpose is, right?
00:09:21 These are the operators you use a lot in the MLP.
00:09:26 And what about the convolution, right?
00:09:29 Convolution, here I'm using a 2D convolution as an example, okay?
00:09:36 So let's say here you have an input, right?
00:09:39 And then you have a kernel or a filter, right?
00:09:44 This is not the kernel we talked about in a support vector machine, okay?
00:09:50 It's used to be called a filter in signals and systems, okay?
00:09:56 And what it does is that you have this, you know, kernel matrix, right?
00:10:00 And then you have the input.
00:10:02 And what this kernel matrix is doing that apparently it's, you know,
00:10:05 it's a smaller size compared to the input image.
00:10:08 And it's essentially extracting a local summary, right?
00:10:12 Right?
00:10:13 It's summarizing this local.
00:10:15 Why do we want to do this local summary?
00:10:17 Don't forget about our summary of the vision systems.
00:10:21 I will remind you again later on, okay?
00:10:24 Right?
00:10:25 That's something we have learned, right?
00:10:26 Remember?
00:10:27 One is the invariance, one is the locality, right?
00:10:31 So in this case, what you do here is if, you know, you can imagine that you overlay this
00:10:38 kernel on top of the input and then you keep sliding the window, right?
00:10:43 Imagine doing that, right?
00:10:45 So the first time it's overlaid like here, the summary is like element-wise product and
00:10:52 then summed together, right?
00:10:54 Right?
00:10:55 And then you can, you know, move this kernel to the right by one unit.
00:11:00 So now it's overlaid here.
00:11:02 So now B is multiplied to W, C is multiplied to X, and then you add all of them together.
00:11:06 So this is what convolution means in imaging, okay, in computer vision or in convolution neural
00:11:14 works.
00:11:15 I have to remind you, this is not the convolution you learn in signals and systems.
00:11:21 What is the convolution in the signals and systems?
00:11:26 So what you are told here is like, you know, I have an input, let's say it's a, you know,
00:11:37 it's a K and then there is input I and then there's kernel K.
00:11:43 What it's doing is that, you know, it's doing this element-wise product and then sum all of
00:11:50 them together, right?
00:11:51 That's what this illustration is telling.
00:11:55 It's actually not convolution.
00:11:59 What convolution is actually this expression.
00:12:03 Notice that I'm using I minus M or J minus M here.
00:12:08 So that is very different.
00:12:10 So in other words, what you are doing here is that, you know, you are actually not really
00:12:16 doing the convolution in the signals and systems.
00:12:19 What you're doing is, you know, you are not doing the convolution between the input and
00:12:24 the kernel.
00:12:25 You're actually doing the convolution between the input and the time-reversed kernel.
00:12:30 Is that right?
00:12:32 Right?
00:12:33 Because you think about this K is as a function of M and N, right?
00:12:37 Now it's minus M minus N is time-reversed, right?
00:12:42 Because K is a function of M and N, right?
00:12:48 And we call it, you know, in the signals and system, we call this variable time, no matter
00:12:54 whether in this case it's actually 2D space, right?
00:12:57 So in other words, K is time-reversed compared to what is defined in the convolution, right?
00:13:05 So, then does it matter or not?
00:13:11 Well, you know that there is a difference between what we are actually doing here, which is a
00:13:16 cross-correlation between the input and the kernel versus convolution, right?
00:13:21 Right?
00:13:22 There are a lot of properties about convolutions, right?
00:13:26 Right?
00:13:27 So, but that is not satisfied by the cross-correlation.
00:13:31 For instance, convolution is commutative, okay?
00:13:35 The convolution of K and I and I and K are the same, but you cannot say the same for cross-correlation.
00:13:42 That's essentially what we are doing here, okay?
00:13:45 Then do you mind?
00:13:46 Well, not really, because, you know, you've got to think about the whole system you have.
00:13:52 You have the convolution, but then you have the follow-up layers, which do not necessarily
00:13:57 define the commutative operator anyway, right?
00:14:02 So if the commutative is mattering to you, it does matter to you.
00:14:05 So it doesn't really matter, okay?
00:14:09 But you know, another question for you to say is, you know, for you to answer is do we really
00:14:15 need commutative properties in this case?
00:14:18 Something to think about it, okay?
00:14:19 I put a question mark here for you to think about.
00:14:22 So this is what convolution is doing, and we have the convolution, which is very much similar
00:14:29 to the matrix multiplication.
00:14:32 And there is a d-convolution, which is like the multiplication by transpose, and then there
00:14:38 is some computation for these weights, which are the parameters, and then we use gradients
00:14:44 along the weights, and specifically we use the backpropagation.
00:14:49 And we'll talk about these backpropagation when we talk about a neural network computation,
00:14:56 okay?
00:14:57 And definitely they utilize the fact that we are using parameter sharing in this special
00:15:02 case.
00:15:03 So now I can summarize my answer to the initial question, right?
00:15:10 So what mathematical operators do we desire for grid data seen in image, right?
00:15:18 So that we can revolutionize MLP to something we need?
00:15:22 So the answer is essentially what convolution can bring, okay?
00:15:26 So there's sparse interactions, right?
00:15:29 And then it's parameter sharing because you keep using the same kernel and you move it along
00:15:35 in different pieces, right?
00:15:37 So it's equivalent, right?
00:15:38 It's equivalent, and because it doesn't matter where this target image is, right?
00:15:46 And your signal would be translational invariant.
00:15:49 It doesn't matter where that cat or traffic light pixels are translationally, right?
00:15:57 So your output of the system will always give you the same signal, right?
00:16:03 And then it's very adaptable to the variant input size, right?
00:16:08 So now let's talk about one by one, okay?
00:16:11 The first thing, the sparse interactions.
00:16:13 So let's look at, you know, this case, right?
00:16:19 So what we learned before was dense connections essentially in MLP, right?
00:16:24 But in our case, right, what is the connection here?
00:16:30 Think about it.
00:16:32 So you should be very familiar with these kind of illustrations now, right?
00:16:38 Now the question is, you know, let's look at input, you know, this is one pixel, right?
00:16:45 X3, for instance, right?
00:16:48 So the question is, right, how many outputs would be affected by this pixel?
00:16:59 How many?
00:17:07 Here I'm drawing three, what do I mean?
00:17:10 It means that this pixel, for instance, B, right, will affect three such outputs in the
00:17:23 hidden layer.
00:17:25 What does it mean?
00:17:29 It means that if I move this kernel, like I'm talking about one dimension in this case first,
00:17:36 okay, if I move horizontally, right, unit after unit, one unit after another unit, this B will
00:17:45 appear within this kernel, right, so that it can impact the output three times in total.
00:17:54 What does it mean?
00:18:00 What does it mean?
00:18:06 It means the kernel is of size three, right, if it's one dimension.
00:18:11 Is that right?
00:18:13 Right?
00:18:15 That's why you see these three arrows.
00:18:19 Okay?
00:18:20 So in other words, in this 1D case, you can already tell that the kernel size is one by
00:18:27 three.
00:18:28 Okay?
00:18:29 So that tells you, you know, this is a small convolution kernel that defines the sparse connectivity,
00:18:36 connected.
00:18:37 So that's the first thing.
00:18:38 Okay?
00:18:39 And then the second thing is, you know, you can also look reversely.
00:18:47 Look at, you know, one output, right?
00:18:50 How many input elements or the pixels in this case will affect that output?
00:18:56 Again, it's, you know, it's the three, right?
00:19:01 But something else to think about is these three parameters, actually, is the same as these three,
00:19:10 you know, parameters in some sense, because you are sharing the kernels.
00:19:14 We'll talk about the parameter sharing next.
00:19:17 Okay?
00:19:19 One thing to think about is that, you know, let's think about this, okay?
00:19:24 You may think, well, do I desire this sparse connectivity, right?
00:19:28 It feels like, you know, if I look at this, you know, hidden neuron, right, it receives inputs
00:19:38 from only three neurons from the previous layer, which is input layer, right?
00:19:44 We can call it the receptive field, and can we, do we worry about this receptive field being
00:19:51 narrow?
00:19:53 Not really, if you consider more layers.
00:19:56 Now if you include more layers, look at the latter layers.
00:19:59 Yes, yes, G3 is only, you know, receiving inputs from three previous neurons, right?
00:20:08 But each of the three previous neurons also receive their three information from the last
00:20:14 layer.
00:20:15 So in other words, this so-called receptive field, right?
00:20:19 If you sit here and receive the signals from all the way to from the input, your receptive
00:20:25 field is actually growing toward the input.
00:20:27 So, you know, this is, this deep layer indirectly interact with a large portion of the input.
00:20:34 This is a really nice property.
00:20:36 I mean, it's easy to say it's convolution, but if you think about the consequence, there are
00:20:41 so many nice properties brought forth.
00:20:45 So that's just about the sparse interactions alone, because, you know, it's easy to say,
00:20:49 I can have a sparse model, but it's, you know, it's another thing to have a sparse interaction
00:20:56 with this receptive field, right?
00:20:59 That's growing.
00:21:00 So the second property, right, is the parameter sharing.
00:21:05 Okay?
00:21:06 Not only we have the sparse connections, so small number of parameters, right?
00:21:10 But also these parameters are shared.
00:21:13 Okay?
00:21:15 So the thing I talked about before, right?
00:21:17 So in other words, the parameters here are what?
00:21:21 The kernel matrix, isn't it?
00:21:24 And this kernel matrix is used, you know, anywhere, right?
00:21:28 It's used anywhere.
00:21:29 So in other words, you know, if we look at this illustration, this illustration, right?
00:21:36 So the parameters are actually shared, right?
00:21:41 So in other words, you know, if you look at this illustration, right?
00:21:46 If you look at this illustration.
00:21:48 So in the case of the MLP, this parameter, right, which is X3 to S3, is different from
00:21:56 X2 to S2, because you have all these fully connection, which is element of a big matrix,
00:22:03 right?
00:22:04 Which is five by five, right?
00:22:06 But in our case, we have what?
00:22:08 In this case, we have, if the receptive field, sorry, excuse me, if the kernel size is three
00:22:15 in this case, then it means what?
00:22:19 This means what?
00:22:20 It means this parameter, okay, associated with X1 to S1, is the middle of the one by three,
00:22:31 is the middle element of the one by three kernel.
00:22:34 Is it?
00:22:35 Do you agree?
00:22:37 And then if you look at here, X2, again, it's, you know, it's, you know, getting the output
00:22:45 to S1, S2, S3, and X2 to S2.
00:22:50 This parameter is, again, that middle element of the one by three kernel.
00:22:57 Now, in other words, all these solid arrows have the same parameter, right?
00:23:08 So that's, again, from, you know, the source to the next layers, right?
00:23:16 So let's look at one example that is something you don't even have to learn from the data,
00:23:22 that we can design by principle, right?
00:23:25 Okay.
00:23:26 So let's say we have a 320 by 280 input image, right?
00:23:31 And our kernel is one by two, which is one and minus one.
00:23:35 If you look at what one and minus one is, you can clearly figure out what it's doing, right?
00:23:41 It's taking, you know, each time we are looking at two neighboring pixels, horizontally neighboring
00:23:47 pixels, and then subtract the right pixels value from the left, right?
00:23:55 And then what it's doing essentially is telling the difference between the horizontally neighboring
00:24:01 pixels, which is doing what?
00:24:05 Edge detection, right?
00:24:06 This edges should be, this is what, what kind of edge?
00:24:10 Vertical edges, right?
00:24:13 So the question is, how many operations are we doing going from this 320 by 280 input to
00:24:20 this output?
00:24:22 How many operations do we have?
00:24:25 Apparently you are doing multiplication, you are doing the sum.
00:24:31 How many operations do you have?
00:24:33 What's the complexity?
00:24:38 How many operations?
00:25:03 So if you don't pad and you only consider 320 times 280, right?
00:25:10 Let's see.
00:25:11 It looks like this side is 320 and this is 280, seemingly, right?
00:25:17 By my eyes.
00:25:19 So that means you start with a one by two and then you move it to the right.
00:25:27 One unit and another unit and all the way.
00:25:29 How many times can you traverse?
00:25:33 319, right?
00:25:37 And each time you do what?
00:25:39 You do two multiplication and one sum, right?
00:25:43 And then you do that for all the rows, that's 280.
00:25:47 So you can calculate how many operations do you have?
00:25:50 The question is, what about fully connected matrix multiplication?
00:25:55 If you have to do a matrix multiplication, how many operations do you have?
00:26:09 So let's say this is 320 by 280 and then let's say this is still 300 by 280, right?
00:26:16 So to make sure that this input matrix times another matrix equals to output matrix of size 320 by 280, this second matrix to multiply to the input matrix needs to be 280 by 280.
00:26:31 And you have 280 squared elements, right?
00:26:36 So how many operations do you have to make?
00:26:40 You have to do a lot of, for each of the 280, so let's look at each of the 320 times 280 elements.
00:26:52 To get each of the elements, we need to do how many?
00:26:56 We need to do a 280 multiplication and then sum them together, right?
00:27:08 That's for each of the 320, right?
00:27:12 So you can calculate, so I think I already reached the answer and you can compare these two numbers.
00:27:18 How many operations you are saving using the convolutions, even in a simple case of edge detection.
00:27:26 That's the answer here, okay?
00:27:28 So as I said before, if you use the convolution, right?
00:27:36 So this is your input size, this is your kernel size, and this is the output size.
00:27:41 It's one less because there's no padding, okay?
00:27:45 And in the case of storage, right, this is like the space complexity, right?
00:27:53 The storage, the dense matrix is of this size and the sparse matrix is of this size, right?
00:27:59 This is for the storage.
00:28:02 And in the case of the multiplication and the summation, this is the number I, you know, talked about before, right?
00:28:11 For the dense matrix, this is actually 10 to 9, so there is a huge difference, okay?
00:28:16 So that's the answer to my earlier question and the final answer.
00:28:26 So that's the second property.
00:28:28 The third property is equivalent representation, okay?
00:28:32 Something we have learned before was that no matter where that, you know, that object is, right?
00:28:41 When we are trying to find that guy in a bunch of savages, right?
00:28:47 So we knew that we actually have this translational invariant, no matter where local patches, where these, which local patch,
00:28:58 that this object reside in, our vision system will, you know, generate the same output signal, right?
00:29:06 So that's actually what we desire.
00:29:08 So in this case, it mathematically, you know, it's rooted in the group theory, but let's look at the shallow interpretation and look at the equivalence to translations first, okay?
00:29:22 In general, you can say this is the equivalent to some perturbation of the input, okay?
00:29:28 And you will see later on, like, the SE3 equivalent means rotational and translational invariant, okay?
00:29:37 So in the case of the convolution, the particular form of parameter sharing causes the layer to have a property called equivalence to translation, meaning the following.
00:29:48 Let's say you have a perturbation of X, okay?
00:29:52 That's quite a G.
00:29:54 This G could be translation, rotation, or something, okay?
00:29:58 If the, you know, function of the perturbed input equal to the perturbed function of the input,
00:30:08 then this function F is equivalent to the perturbation G, okay?
00:30:14 So in our case, G is translation, is that right, right?
00:30:18 You have learned in signals and system, if G is an affine function, what do you call F?
00:30:30 If G is affine and if F have this property, it means that F is a linear system, right?
00:30:38 Okay.
00:30:40 The next question is, so, you know, this is an intuitive understanding of equivalence, okay?
00:30:48 And some people sometimes use different definitions of, you know, there's another concept called invariant, okay?
00:30:58 So, so, so, so there's a invariance.
00:31:02 If the F of G of X equal to F of X still, then you can say that F is invariant to G, okay?
00:31:10 The next question is, convolution is equivalent to translation?
00:31:15 That's good, right?
00:31:16 Because that's how our vision system works.
00:31:18 But you should also know that to find that little object, your brain or your vision system also know that,
00:31:25 if the object gets bigger or smaller, it still is that object, all right?
00:31:29 I don't care whether this cat image is small or big or the pixels are, if out of this image, right,
00:31:36 three by three pixels are this cat or, you know, 100 by 100 pixels corresponding to the cat,
00:31:42 I can still call it a cat, right?
00:31:44 So the output should be invariant to the scaling, right?
00:31:49 And also, it should be invariant to the rotations, right?
00:31:54 It doesn't matter how I rotate, right?
00:31:57 So, sorry, I should say equivalent because this is before you do the readout, okay?
00:32:03 In the end, when you do the readout, you have the invariant.
00:32:06 But right now, we're talking about the equivalent means no matter where that kernel moves to, right?
00:32:11 The corresponding operation will give you the same perturbation.
00:32:15 In other words, if the input changes, the output changes in the same way, okay?
00:32:19 So the question is, not only our vision system works like the translational equivalent, but also scale and the rotation equivalent.
00:32:30 Does convolution give us that property?
00:32:33 So I would strongly suggest you first prove that this equivalence applies when G is translation and F is convolution.
00:32:47 And then you will find out that if G is a scaling or rotation, then F convolution is not equivalent to such G, okay?
00:33:00 So that's why the convolutional neural networks actually does not know about the rotational equivalence, okay?
00:33:10 So that's why people have to spend extra efforts to teach convolutional neural networks this knowledge.
00:33:17 One of the less efficient but easily adaptable approach is to give the system a lot of examples to enforce that knowledge, right?
00:33:30 For instance, you have one image, you can rotate it many, many different times, right?
00:33:35 Different angles, and it gives the same label, right?
00:33:38 In this way, it's less than not efficient, but it's just one way, okay?
00:33:42 That also means how important it is to directly teach a neural network and knowledge, then try to ask it to learn from the supervised examples.
00:33:54 That's a very difficult task, okay?
00:33:56 Okay, so now let's try to summarize, right?
00:34:02 So what knowledge drives model reduction compared to the MLP?
00:34:07 And actually we learned that from the vision systems, right?
00:34:13 We learned that there's sparse interactions, parameter sharing, equivalence to translations.
00:34:20 All these properties are broad force thanks to convolution, okay?
00:34:24 And we also look at how much cost reduction that is broad force.