Lecture 17: First Take on (Artificial) Neural Networks

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Well, before the spring break, we already started talking about artificial intelligence and more specifically, deep learning and its impact to a lot of fields.

00:00:13 As you can see, in the recent decade, we see how artificial intelligence has progressed in multiple fields, including games.

00:00:29 That's a perfect testbed for Turing test, right? Including image or computer vision tasks, including text, natural language processing.

00:00:45 And the early waves of computer vision or natural language processing products have already shown their impacts to the healthcare field, right?

00:00:59 Of course, there are also fallbacks, right?

00:01:01 We know IBM Watson and also that it started collaboration with MD Anderson quite early, but also we saw that the outcome isn't as perfect as we anticipated, right?

00:01:19 So now like the fairytale, right? After that, AI and healthcare are married happily forever. It's not like that, right?

00:01:27 This shows the challenges that are remaining as well as more foundational AI changes. You cannot just do AI to healthcare.

00:01:39 You actually, healthcare inspires a lot of AI for healthcare. So let's bear those in mind and then we'll start talking about this week, artificial neural networks and computer vision, especially image data.

00:01:54 So we don't assume that you know anything about neural networks, but also if you have already learned about neural networks, right?

00:02:06 Many of you have already. Again, I want you to think about them from the perspective of optimization, what are important for neural networks, as well as understanding neural networks from not only the intuitive illustration,

00:02:22 like those neurons and the connections, but also the mathematics and as well as programming. Okay?

00:02:31 So that's what we will try to address. And at the end, we'll include two case studies. Again, it's about protein function, sorry, excuse me, structure prediction.

00:02:42 But you see that the formulation is very different and the way the problem is solved is also very different.

00:02:50 Anyway, ask yourself, why are these problems? What is the problem of protein structure prediction can now be formulated in this way?

00:02:58 What images are there and what are we utilizing? Okay?

00:03:02 Okay, so let's get started. I think at the beginning of this week, we'll mostly talk about the neural networks and then we will branch into the convolutional neural networks.

00:03:18 Okay? And then we'll spend quite some time on the math and underlying of CNN. And in the end, we include a couple of case studies. Okay?

00:03:31 Okay? So let's talk about neural networks, right? So as you have learned before, right, deep learning is actually the poster child of artificial intelligence.

00:03:44 A lot of people say that I'm doing AI, but essentially, it's deep learning. Okay? AI is a buzzword, but we also knew that from what you learned before, AI is much wider than machine learning.

00:03:57 Right? And so what is machine learning, right? So what is machine learning? I think this is a good comic, right? So you have the data feeding into the models and within the models, you have a lot of, you know, operations, mathematical operators in the end, you make a prediction.

00:04:21 Right? And this is actually a very funny comic, right? That's machine learning. And we knew better that, you know, for instance, from the perspective of regression, what we care about is not how the model fits observed data.

00:04:43 It's actually about how the model fits the data. It's actually about how the model fits the data. Right? That's why we had the bias variance trade-off, right?

00:04:53 So if you look at machine learning, it's actually, you know, it's very wide concept, right? So if you look at this principle axis of deep versus shallow, we have already learned a lot of shallow machine learning models before.

00:05:10 We talked about the support vector machines, we talked about boosting, including AdaBoost. We'll briefly mention the single layer perceptron before we talk about multi-layer perceptron this week, right?

00:05:23 But these are supervised machine learning that are, you know, shallow. They can be deeper, right? Using neural networks, convolutional neural networks, or recurrent neural networks or language models, this will be described.

00:05:38 But also you can think of it from another perspective, that is whether it's supervised versus unsupervised, right? In other words, whether there are labels being utilized in this machine learning, right?

00:05:53 So in the unsupervised shallow models, right? We have some early forms of neural networks such as the restricted Boltzmann machines, right?

00:06:04 Or the shallow autoencoders, or the Gaussian mixture modeling, right? So those are shallow unsupervised models that you might have learned.

00:06:14 The deep ones are including this deep autoencoders, or GAN, right? This will be learned at the end about the generative models, right?

00:06:25 Deep belief networks or, you know, base neural networks and so on.

00:06:31 So this is a rough idea about what is the landscape of machine learning, right?

00:06:39 And you have to see that we've previously covered this, and now we are going to the deep side of the supervised machine learning, right?

00:06:47 So deep learning, right? Deep learning. It's not, let's put it in this way, neural network itself is really an old concept, right?

00:07:02 But probably early days, we, you know, we utilize just the one hidden layer between the input and output layers, okay?

00:07:13 So for a lot of reasons, not just the fact of the hardware, but also the fact of solving these problems, actually.

00:07:23 I will revisit this again. Just keep in mind that this machine learning problems are optimization problems.

00:07:30 In the end, you need optimization algorithms to solve these optimization problems, right?

00:07:35 So it's not like, you know, I can draw more layers and then there are algorithms that can readily solve the problem, okay?

00:07:45 It takes a lot to go from the single to multiple hidden layers. It's not a conceptual change. Do you see what I mean?

00:07:53 It's not like nobody can visualize the multiple hidden layers. That's not a problem.

00:07:57 The problem is what can support this model, including the hardware, including the software, specifically solving this optimization problem, okay?

00:08:07 I wouldn't name the technique, but you should know what technique I'm referring to.

00:08:11 I'm going to ask again in today's lecture, or when we talk about a multiple layer perception.

00:08:16 I think it should be today. In this example, you know, this deep learning neural work is a computer vision kind of task, right?

00:08:25 You have an image input, and the output is like a classification of who this person is, right? George.

00:08:33 So, you can see that what's really interesting about this deep learning neural work is that what at different layers, what are picked up?

00:08:48 What parts of the images or what signals of the images are picked up by different layers, right?

00:08:56 Because you can see at the beginning, right? It's just about each pixel and how light, how dark it is, right?

00:09:02 That's at the input layer. But the intermediate neurons are picking up different things, right?

00:09:08 Like edge detection or combination of edges or more sophisticated features like eyes and nose and so on, right?

00:09:17 So, in signals and systems, you have learned before that we can actually artificially, I'm sorry, manually design the filters to detect, for instance, edges, right?

00:09:30 Right? So, we can make sure that, for instance, a filter that takes the difference between two neighboring pixels, and that is one way for edge detection, right?

00:09:40 You have the horizontal or vertical edge detection, right?

00:09:44 So, there are a lot of ways for you to manually design those filters from principle, right?

00:09:51 But in this case, you are actually automatically learning much higher dimensional and nonlinear filters that are capable of detecting much more sophisticated features.

00:10:05 So, that's a difference, okay, compared to the earlier signal processing.

00:10:11 So, one is principle driven and one is data driven, okay?

00:10:16 And let's start talking about neural networks, right?

00:10:22 I'm going to make this claim. You have already learned some neural networks in this course already.

00:10:28 Well, you can say these are special cases, right?

00:10:34 Okay, so what have we learned? What neural networks have we learned so far?

00:10:39 Can you recall some?

00:10:42 Have we learned any neural networks so far in this course?

00:10:47 Yes? No?

00:10:58 Have we learned any neural networks yet?

00:11:01 No.

00:11:02 No.

00:11:03 No.

00:11:04 No.

00:11:05 No.

00:11:06 No.

00:11:07 No.

00:11:08 No.

00:11:09 No.

00:11:10 No.

00:11:11 No.

00:11:12 No.

00:11:13 No.

00:11:14 No.

00:11:15 No.

00:11:16 No.

00:11:17 [Uncertain recognition] No.

00:11:18 [Uncertain recognition] No.

00:11:19 [Uncertain recognition] No.

00:11:20 [Uncertain recognition] No.

00:11:21 [Uncertain recognition] No.

00:11:22 [Uncertain recognition] No.

00:11:23 [Uncertain recognition] No.

00:11:24 [Uncertain recognition] No.

00:11:25 [Uncertain recognition] No.

00:11:26 [Uncertain recognition] No.

00:11:27 [Uncertain recognition] No.

00:11:28 [Uncertain recognition] No.

00:11:29 [Uncertain recognition] No.

00:11:30 [Uncertain recognition] No.

00:11:31 [Uncertain recognition] No.

00:11:32 [Uncertain recognition] No.

00:11:33 [Uncertain recognition] No.

00:11:34 [Uncertain recognition] No.

00:11:35 [Uncertain recognition] No.

00:11:36 [Uncertain recognition] No.

00:11:37 [Uncertain recognition] No.

00:11:38 [Uncertain recognition] No.

00:11:39 [Uncertain recognition] No.

00:11:40 [Uncertain recognition] No.

00:11:41 [Uncertain recognition] No.

00:11:42 [Uncertain recognition] No.

00:11:43 [Uncertain recognition] No.

00:11:44 [Uncertain recognition] No.

00:11:45 [Uncertain recognition] No.

00:11:46 about but then I want you to also keep in mind that you know machine learning is

00:11:52 about formulating and solving optimization problems so I want you to

00:11:56 keep in mind that when we're talking about this machine learning neural

00:12:00 networks what are the elements important for these models okay from the

00:12:06 perspective of optimization what elements are important decision variables

00:12:11 constraints and optimization in the end there is optimization algorithm for the

00:12:17 for the corresponding formulation right I want you to keep those in mind so again

00:12:23 when we talk about the first example linear regression okay I want you to think

00:12:31 about it as optimization problem so what are the elements important first there's

00:12:37 the data and there's model right so the data are the paired X and Y right input

00:12:44 and output right so this is for any arbitrary X I I have the Y hat which is

00:12:49 the prediction so this Y hat is it's not bold-faced indicating that it's a scalar

00:12:56 and X is bold-faced indicating it's a vector and the W and X are bold-faced

00:13:04 and lowercase indicating that there are vectors right when we have the bold-faced

00:13:11 uppercases we indicate that this matrix okay so you can put a subscript I here to

00:13:17 indicate that this is the I's example right but if you consider all the data all

00:13:23 the pairs of X I's and Y I's we have this expression right this is a you know

00:13:28 lumping everything into linear algebra we have this Y vector of all observed labels

00:13:34 equal to capital X w plus B and the decision variables are w and B right okay you can

00:13:43 you can even put one into this X feature right and then w is enough for all the

00:13:51 decision variables okay now that you have the data to to fit and the model right so

00:14:01 this is a model right next question is what is an optimal model right so for

00:14:07 optimality you need a objective function so the objective function here is again summed over all the

00:14:06 [Uncertain recognition] x i plus B minus Y i plus B minus Y i plus B minus right here if from x i yes plus 1

00:14:24 of w and b, the decision variables, right?

00:14:28 And what you see here is the w transpose xi

00:14:32 plus b minus yi.

00:14:34 So this is this error, right?

00:14:39 This is just the error and then squared.

00:14:41 So in other words, this is like the mean squared error.

00:14:44 There's one half there which is a constant

00:14:49 and I think we put it there usually

00:14:51 for elegant expression of the analytical optimal solution.

00:14:56 So that in this case your Hessian is just

00:15:04 something related to w without having to consider

00:15:08 this non-1 constant.

00:15:13 Okay, now that we have the data and the model

00:15:15 and we have the loss and you probably will include

00:15:18 the regularization as well, okay?

00:15:21 You probably have to include the regularization.

00:15:22 Here this is just the mean squared error.

00:15:26 And then once that is done,

00:15:29 we have to solve this optimization problem

00:15:31 using some optimizer.

00:15:32 We know already that this is something

00:15:36 with analytical solution already.

00:15:38 If your regularization is L2,

00:15:40 that's something that we can solve analytically as well, right?

00:15:44 Because it's a quadratic optimization problem.

00:15:48 If the regularization is L1,

00:15:51 we are still facing a convex optimization problem,

00:15:55 but we cannot count on gradients.

00:15:59 But first of all, we don't have the analytical solution

00:16:02 for the optimum.

00:16:04 Secondly, we cannot count on the gradient-based algorithms, right?

00:16:08 Why?

00:16:09 Because the gradients are not continuous, right?

00:16:14 Because of the absolute value of w.

00:16:18 And if your regularization LP norm has p between 0 and 1,

00:16:25 then that optimization is not even convex, right?

00:16:30 Keep in mind, that's what we talked to you before, okay?

00:16:33 All right, so that's from the perspective optimization, right?

00:16:37 So you can see that this can be already represented

00:16:40 as a one-layer neural network, right?

00:16:42 Let's try to see that in programming, right?

00:16:45 Because I feel, I'd like to show all this programming

00:16:48 in NumPy, because I feel like with a lot of libraries,

00:16:52 you just, you know, you feel like all this symbolic expression

00:16:56 of the neural network and the losses,

00:16:59 and sometimes you just forget about what's behind it.

00:17:02 I think I'd rather use a numerical Python

00:17:05 to show it right here, okay?

00:17:08 One thing I want to emphasize is that

00:17:10 why we would like to use this kind of expression,

00:17:14 because you have to think about one of the reasons

00:17:16 that deep learning got popular and successful.

00:17:20 That's because of what?

00:17:24 The operations are relatively friendly for GPUs.

00:17:28 What kind of operations?

00:17:29 This kind of linear algebra operations, right?

00:17:32 A lot of parallelization can be done.

00:17:33 Think about how many threads are there within a GPU core,

00:17:36 right?

00:17:37 So a lot of things can be parallelized.

00:17:39 So in this case, you don't want to write something

00:17:43 that's just a sum over all the example i's like this.

00:17:47 You know?

00:17:49 That's not a good idea, right?

00:17:50 So what you want to do is somehow make it scalable,

00:17:54 right?

00:17:55 Easy to compute.

00:17:56 So in this case, for instance,

00:17:58 this is how you define a model,

00:18:00 which is a linear regressor, right?

00:18:02 So you are using the dot product of the X matrix

00:18:05 and the W vector plus B, right?

00:18:09 Remember that this is over all the examples,

00:18:12 i from one to n.

00:18:14 Well, if you talk about the squared loss

00:18:16 without a regularization, for instance, in this case, right?

00:18:18 Again, you have Y hat minus Y reshape,

00:18:24 Y hat shape,

00:18:26 Y hat shape, and then squared, and then divide by two.

00:18:30 Again, I'm using linear algebra expression.

00:18:32 I'm not using scalar and sum or anything.

00:18:35 All of this are for the linear algebra scalability, okay?

00:18:39 Lastly, what about optimization?

00:18:42 As I said, you know, you essentially,

00:18:47 if you are just looking at the squared loss,

00:18:50 there's already analytical expression of that, right?

00:18:52 But yearly, we have other losses,

00:18:56 and we have to count on gradient-based algorithms.

00:19:00 Well, use one lecture to dedicate

00:19:04 to optimization in deep learning,

00:19:06 but here, let's say we're considering the gradient descent,

00:19:10 but it's not like the traditional sense

00:19:14 of gradient descent in the sense that

00:19:16 you are looking at the stochastic gradient descent.

00:19:19 So in other words, because of the big data issue,

00:19:22 you have a lot of training examples, right?

00:19:25 And your models might be large as well.

00:19:27 Your parameters can be large, right?

00:19:29 And then there's a learning rate

00:19:30 that is a hyperparameter that you can tune.

00:19:33 So in this case, what if we split our data, right,

00:19:39 into a lot of batches, right?

00:19:41 Well, within each batch,

00:19:43 we have something that we can handle,

00:19:45 both for the time complexity and the space complexity, right?

00:19:53 Especially the space complexity,

00:19:54 if you think about the memory.

00:19:56 Time complexity, if your model has a lot of parameters,

00:19:59 it takes a long time to train.

00:20:01 So in other words, I'm just looking at the loss

00:20:04 calculated over the current batch, right?

00:20:09 And there's a way to calculate the gradient

00:20:13 of the parameters in this stochastic sense, right?

00:20:18 And then we keep updating the parameters, right?

00:20:22 We keep updating the parameters.

00:20:23 So this is how you compute.

00:20:26 So again, I'd like to keep these two views side to side.

00:20:31 One is the math.

00:20:33 One is the Python programming.

00:20:35 Okay, so let's make sure that you both understand them

00:20:38 mathematically and are able to program them,

00:20:42 to implement them, okay?

00:20:44 I think this scalability issue is very important, okay?

00:20:47 But now, with the math and the coding, right?

00:20:53 Now we can see them as what you usually see

00:20:57 in illustrations, right?

00:20:59 So again, we are looking at the linear regression.

00:21:02 So you have these input layers,

00:21:04 these input neurons, right?

00:21:07 With D.

00:21:08 D is the dimension of what?

00:21:19 Features, right?

00:21:20 Or attributes in this case.

00:21:22 And for each attribute or feature,

00:21:25 you multiply it by wi,

00:21:28 and you have the sum,

00:21:30 and then you have this logistic function.

00:21:31 This is a linear regression, by the way.

00:21:33 Not logistic yet, just the linear, okay?

00:21:36 So it's just the linear regression in this case.

00:21:38 So this is illustration of what we have talked about so far,

00:21:44 about math and the coding.

00:21:49 Now we can move on to more sophisticated,

00:21:53 from linear regression to logistic regression.

00:21:57 Now let's make logistic regression a little bit more general,

00:22:01 more than just binary classification.

00:22:03 Let's look at multi-class classification, right?

00:22:06 Again,

00:22:09 you have a o,

00:22:11 which is output,

00:22:13 which is wx plus b,

00:22:14 which is again linear, right?

00:22:16 Which is again linear.

00:22:18 But then you use a softmax function over o,

00:22:22 to get the ultimate output y hat.

00:22:25 So this y hat is essentially,

00:22:28 as you can see,

00:22:29 for each example i,

00:22:31 what y hat is,

00:22:33 is this exponential of o,

00:22:35 i, this output of, for example, i,

00:22:38 divided by the sum of example,

00:22:42 of the exponential of o, j's.

00:22:45 So this is a softmax function, right?

00:22:49 So in this case,

00:22:52 these i's and j's are,

00:23:00 the dimensions for the,

00:23:04 for the labels.

00:23:05 So if you have k classes,

00:23:07 j goes from one to k, okay?

00:23:10 And then you can vectorize them.

00:23:11 So this is a multinomial logistic regression,

00:23:14 which is essentially,

00:23:16 you know, a linear,

00:23:17 linear regression,

00:23:18 but then plus a softmax for classification, okay?

00:23:23 So this is the data and the model.

00:23:26 And what is the loss function?

00:23:28 The loss function here,

00:23:29 as we have explained before,

00:23:31 actually in the diagnosis, right?

00:23:34 What we are trying to maximize is the likelihood.

00:23:37 The likelihood of observing y,

00:23:40 given the input x and the w's, okay?

00:23:44 I should put a semicolon w's.

00:23:47 But essentially,

00:23:48 if you treat all the examples i's, right?

00:23:51 I put a superscript i with the parentheses here,

00:23:55 indicating that this is i's example, okay?

00:23:58 This i is i's label class.

00:24:01 Don't get confused, okay?

00:24:03 Here I didn't indicate which example it is.

00:24:06 I used the generic x and the generic y hat.

00:24:10 I didn't use any index of the example, okay?

00:24:13 So the index here, i in the subscript,

00:24:16 and the index here, j and i in the subscript,

00:24:19 indicating the output for the label class i or j, okay?

00:24:27 And here I put a superscript here to indicate

00:24:29 that this is example i from one to n, okay?

00:24:33 So don't get confused here.

00:24:36 So if I assume again that these examples are iid,

00:24:42 are, you know, independently and identically distributed,

00:24:46 actually I only used independence here.

00:24:48 Then I can rewrite this probability as a product

00:24:53 of individual examples, conditional probabilities, right?

00:24:58 And then for, and then I can, you know,

00:25:02 write the loss function as minus log,

00:25:04 that I can maximize this loss function, right?

00:25:07 Then log of the product becomes the sum, right?

00:25:11 And then we can rewrite them as yj times log of the y hat j,

00:25:16 where yj is the,

00:25:23 observed, I'm sorry, the ground truth

00:25:25 or the observed label, for example, x.

00:25:30 This is observed label for class j, okay?

00:25:34 Again, this is a class, don't get confused, okay?

00:25:37 Here I didn't, for simplicity,

00:25:39 I didn't put the superscript i here

00:25:41 to indicate that this is example i,

00:25:43 but for any examples, okay, for any examples,

00:25:47 I can write them in this way, right?

00:25:50 So what is this?

00:25:52 Again, y hat is the probability

00:25:55 for this example in class j, right?

00:25:58 So what we are writing here is what?

00:26:00 It's nothing but a multi-class cross entropy, isn't it?

00:26:06 Right?

00:26:07 So, and then you can add this over all the example i's, okay?

00:26:12 This is L for the example i.

00:26:15 So the optimization problem is to find the optimal w star,

00:26:19 right, this w parameters here,

00:26:24 so that we can optimize the L over all the examples, right?

00:26:28 So, over all the examples.

00:26:32 So, coding-wise, let's see what it does, right?

00:26:36 So we have w that we can randomly initialize them

00:26:40 to be a normal distribution with zero mean

00:26:43 and 0.01 standard deviation, for instance, right?

00:26:46 This is how you initialize your parameters, right?

00:26:50 And you can initialize your w to be zeros.

00:26:54 There are, you know, different ways

00:26:55 to initialize your parameters, right?

00:26:59 This normal distribution is what, you know,

00:27:03 is something related to what Javier,

00:27:05 X-A-V-I-R, Javier initialization is doing, right?

00:27:09 Using a random normal distribution with zero mean, okay?

00:27:15 A wide noise, basically.

00:27:17 Some other cases use zeros.

00:27:19 When is better, when is it better

00:27:23 to use Javier initialization?

00:27:26 When is it better to use zero initialization?

00:27:28 Depends on your layers

00:27:30 and depends on your activation functions.

00:27:31 I suggest you to look into a lot of articles about it.

00:27:35 It's more like,

00:27:38 it's more like experiences or empirical observations, right?

00:27:43 And let's look at the softmax, for instance, in this case.

00:27:48 Again, I want to show you that, you know,

00:27:50 you essentially want to vectorize.

00:27:54 You want to vectorize your mass expression

00:27:57 and make sure that your implementation

00:27:58 or coding is scalable, right?

00:28:02 So if you vectorize, for instance, in this case,

00:28:05 you see that you now, instead of a single observation O,

00:28:10 right, now you have a matrix O, right?

00:28:14 Equals to X times W plus B, right?

00:28:18 And then Y-hat is now, instead of a single scalar,

00:28:22 for one example, now is a matrix.

00:28:24 I'm sorry, instead of a vector, excuse me,

00:28:27 because you have K classes, right?

00:28:29 The length of Y-hat for one example is one times K

00:28:32 or K times one.

00:28:33 And then in this case, if you look at all the examples,

00:28:36 then Y-hat is now a matrix, right?

00:28:38 Again, uppercase, bold-faced.

00:28:40 And it's a matrix which is a softmax over the big O matrix, right?

00:28:44 So if you think in this way, you can see that

00:28:47 the way you implement a softmax is the following.

00:28:50 First of all, it's over a matrix, right?

00:28:52 So if you think in this way, you can see that

00:28:54 the matrix is the following.

00:28:55 First of all, it's over the X matrix, right?

00:28:59 So this is, you know, this X exponential,

00:29:05 which is exponential of this numerator, right?

00:29:09 And the partition function is the denominator, right?

00:29:13 Denominator, right?

00:29:14 So it's summed over axis one, right?

00:29:18 Because it's summing over all the exponentials, right?

00:29:22 And then you return the numerator divided by the denominator.

00:29:26 So that's, again, what's the definition of a softmax, right?

00:29:32 And then let's look at this network, right?

00:29:36 Essentially, it's returning what?

00:29:40 It's returning the softmax of what?

00:29:43 Well, it's a softmax of O, right?

00:29:47 So we need to give O there, right?

00:29:50 Right?

00:29:51 What is our O?

00:29:52 Our observed output, excuse me,

00:29:56 a neural network output, O, right?

00:29:58 Is XW plus B, right?

00:30:02 So what you're doing is like what you have seen before

00:30:05 in the last time, using a dot product of the W matrix

00:30:10 and X matrix plus B.

00:30:12 But you've got to be careful about this three shape,

00:30:15 minus one num inputs.

00:30:17 Okay?

00:30:18 You've got to be careful about this one.

00:30:20 Got to make sure that what dimension is multiplied

00:30:23 to what dimension.

00:30:24 This is very important.

00:30:25 If you don't fully understand this,

00:30:27 I strongly suggest you to look into this much more carefully.

00:30:30 Okay?

00:30:31 Because you have to control, you have to understand the math

00:30:35 and you have to control the dimensionality, right?

00:30:38 A lot of the errors can occur there

00:30:40 and we cannot count on libraries to do everything for us, right?

00:30:45 And lastly, the last function, the last function,

00:30:48 as I said, is cross entropy, right?

00:30:51 And this cross entropy is in this expression, right?

00:30:56 Again, I'm considering not just the one example.

00:31:01 And in my cases, these are vectors for all the examples.

00:31:06 And then I'm returning this value, right?

00:31:08 So this is how the cross entropy is defined for a single example.

00:31:16 And then when you are calling this cross entropy to maximize

00:31:20 this loss function for all the examples,

00:31:22 you're going to make sure that you give the correct y hat and y, right?

00:31:26 In your case, it's going to be a vector

00:31:31 and you're going to make sure that you are going to use the products

00:31:35 or some very carefully, okay?

00:31:40 Any questions so far about these two examples?

00:31:43 They are very simple, but there's a lot of things about them, all right?

00:31:47 If you want to now think about them as neural networks, right?

00:31:52 And then you need to understand them with regard to the optimization formulations,

00:31:59 these three elements, as well as the scalability in the programming.

00:32:04 Make sure that you have a deep understanding of this, okay?

00:32:07 These are just very simple neural networks, okay?

00:32:11 Any questions so far?

00:32:15 Any questions?

00:32:16 Okay, now let's move on.

00:32:18 Again, as I said, you know, you could represent them now,

00:32:22 illustrate them as this multinomial logistic regression, right?

00:32:27 So I have O's there, and I didn't put the softmax here,

00:32:30 and you can add one more softmax layer, and the output is y hat, right?

00:32:36 y1 hat to y3 hat, y4 hat, and so on, okay?

00:32:41 Depending on how many classes do you have.

00:32:43 Okay, so now let's talk about, you know, multi-layer perceptron.

00:32:51 Let's start with a single-layer perceptron, okay?

00:32:54 Perceptron is a very important concept in the early days of artificial intelligence, right?

00:33:00 By Rosenblatt.

00:33:01 This is not just a model, but actually a real machine, a single-layer perceptron that was done in the 50s or 60s, okay?

00:33:10 So we will talk about the biological reasons or motivations underlying the single-layer perceptron in a few slides,

00:33:21 but now I just want you to regard them as this very simple illustration, right?

00:33:28 Because we already gone through the math and the programming to understand these illustrations, right?

00:33:35 So this is, you know, a single-layer perceptron in the early days.

00:33:40 Remember that, you know, this is...

00:33:45 Sometimes people use these interchangeably with DNN, but this is, you know, for the multi-layer perceptrons.

00:33:53 In the case of the single-layer perceptron, you can think of it as a linear classifier, right?

00:33:59 Depending on how many classes you have.

00:34:01 What you have here, these are just the O, the intermediate outputs.

00:34:05 Remember, in the end, my prediction is y-hat, right?

00:34:09 y-hat, where this example belongs to class i, okay?

00:34:13 It's the probability of the example belonging to class i, okay?

00:34:18 All right, so this is a single-layer perceptron.

00:34:21 It's, you know, a linear classifier.

00:34:24 It could be a multinomial, multi-class.

00:34:28 And we can add one hidden layer between the input or output layer, okay?

00:34:42 Like this h, right?

00:34:44 And you can determine the width or the number of neurons in the hidden layer, okay?

00:34:51 Well, now is the question, right?

00:34:55 So, let's ask this question, right?

00:35:00 So, if you keep stacking more and more hidden layers, right?

00:35:08 Do you think that your model is much more powerful?

00:35:14 Or do you think that your model is still limited in some way?

00:35:18 So, imagine that I want to empower my model, right?

00:35:23 Going to the multi-layer perceptrons, and I can stack more and more hidden layers, right?

00:35:28 Make it deeper and deeper.

00:35:31 Do you think that this is making your model much more powerful,

00:35:38 or do you still think that it's limited somehow?

00:35:40 Hmm?

00:35:41 It's still limited, why?

00:35:55 I feel like adding, it's not the same, but I feel adding the layers are similar to, what's that, like changing 1D feature to 2D feature with what's that name?

00:36:14 So, you feel like adding the layers is as if we are expanding the feature dimensions without changing the operator over the features, is that right?

00:36:31 Yeah.

00:36:32 So, let's look at one example, okay?

00:36:37 In this case, we use two layers, right?

00:36:44 Your O is W2H plus the B2, right?

00:36:49 And then your H is the output of, is input to the second hidden layer, which is the output of the first hidden layer, right?

00:37:01 So, you can replace H by W1X plus B1, right?

00:37:06 And if you rearrange them, you will find out that it's still in the form of WX plus B,

00:37:15 where W is this two layers W's product, and B is this affine function.

00:37:22 So, in other words, you know, you can keep stacking the hidden layers, right?

00:37:29 But, it won't change the fact that your classifier is linear, okay?

00:37:40 That's one point to pay attention, okay?

00:37:44 This is one thing to think about.

00:37:47 Another thing to think about is the following, right?

00:37:51 Why do you worry about these linear classifiers, right?

00:37:58 If you recall, when we talked about support vector machines, we said that, you know,

00:38:03 when you find out that the linear support vector machines are inadequate, what would you do, right?

00:38:09 Right?

00:38:10 So, because sometimes, you know, there's no perfect way to separate two classes with linear classifiers, right?

00:38:20 Then you may say, well, then I allow this software penalties, right?

00:38:25 I allow violation of these, you know, red goes to the other side of the linear classifier,

00:38:32 but I make sure that it turns into a penalty, right?

00:38:36 Another thing we said was that you may also consider a linear classifier in the kernel space that is nonlinear in the original feature space, right?

00:38:47 Something to think about.

00:38:50 Here, this is one example, right?

00:38:52 So, it's a very simple operation we are trying to emulate, okay?

00:38:58 So, the natural process to emulate is AND or OR.

00:39:03 So, in the case of AND, right?

00:39:06 So, we have, you know, 0, 0, 0, 1, and 0, 1, 0, all giving us 0, and only 1, 1 give us 1.

00:39:18 So, you can use a linear classifier to separate them, right?

00:39:21 The same for OR.

00:39:24 But for this exclusive OR, right?

00:39:28 Which is only, which is, you know, which is only 1 when these two are different, 0, 1, or 1, 0, exclusive OR,

00:39:39 would you get the output of 1, right?

00:39:42 Circles.

00:39:43 So, in this case, it's really hard for you to use a single layer perceptron, right?

00:39:52 Right?

00:39:53 There's no single, in this case, linear classifier that can separate these two perfectly, right?

00:40:01 So, this was actually related to the first winter of AI, okay?

00:40:07 So, basically, after the powerful perceptron, there are a lot of research into it, right?

00:40:12 So, in 1969, Marvin Minsky, if you recall the Dartmouth workshop, how this proposal was submitted,

00:40:22 which people?

00:40:23 Shannon, right?

00:40:24 Right?

00:40:25 Was one of them.

00:40:26 Minsky was one of them, I was asking you, right?

00:40:29 And Marvin Minsky, right?

00:40:32 Whose name is now in the MIT C-cell lab, right?

00:40:36 Computer Science and Artificial Intelligence Laboratory.

00:40:39 He basically proved that, you know, you cannot, okay, you cannot train a single layer perceptron for this very simple XOR, exclusive OR function.

00:40:59 So, in other words, you think about, you know, the ambition of artificial intelligence back then was to figure out how brains work in a quantitative way so that we can empower machines to do something similar, right?

00:41:14 But now, it cannot even do something for exclusive OR, right?

00:41:18 So, that was, you know, the beginning of the first winter for AI, right?

00:41:24 So, that was 1969 when Marvin Minsky and the co-author published their paper to critique on Rosenblatt's single layer perception, right?

00:41:36 And it's possible that you can include more layers, right, more hidden layers to have multi-layer perceptions, right?

00:41:46 But not exactly the same multi-layer perception we're talking about so far.

00:41:51 It actually, it's not like Marvin Minsky did not know that, you know, if you include more layers, this may become separatable.

00:42:01 The real reason is that there's no optimization algorithm available for this multi-layer, this kind of perceptron.

00:42:11 When was it eventually born?

00:42:14 1969, you know, Marvin Minsky critiqued two perceptrons or single layer perceptrons, right?

00:42:23 Using this example, right, theoretically proved that you cannot even emulate exclusive OR, right?

00:42:29 But it's not like they didn't know multi-layer perceptrons can deal with it, right?

00:42:34 But why that was still not as, you know, saving grace?

00:42:39 Why was it not that useful?

00:42:42 As I said, there's no optimization algorithm for multi-layer, this kind of linear neural network available.

00:42:50 It was officially born 17 years later, a nature paper, and this algorithm is called what?

00:43:04 You know it and use it every day if you're doing deep learning.

00:43:18 It's called backpropagation.

00:43:22 Okay?

00:43:24 In fact, backpropagation was not born in 1986.

00:43:29 Actually, backpropagation was rediscovered later that actually it was even born before the Book of Perceptron.

00:43:39 You know, it's just really funny, okay?

00:43:42 Mathematicians can be really ahead of, but it's, you know, it's common that we see a lot of rediscoveries, okay?

00:43:51 But so, you know, history aside, so our conclusion here is that, first of all, right, to enable this multiple hidden layers,

00:44:01 you really need optimization algorithm to support them.

00:44:06 But in the meantime, you're still in a class of linear functions or linear classifiers, right?

00:44:13 So that leads to what we know today as multi-layer perception, right?

00:44:19 The only thing you change here is that instead of directly taking the output, right, the linear output,

00:44:28 you take the linear output and do a nonlinear transformation, and that is called a nonlinear activation function.

00:44:36 We'll understand why it's called activation function when we talk about the actual neurons,

00:44:40 but this is what the activation function is doing, right?

00:44:44 So this H is putting a nonlinear transformation to W1X plus B1, okay?

00:44:49 And then this H is fed to O, okay?

00:44:53 This is just a single hidden layer.

00:44:56 If you add more and more hidden layer, basically at the end of each hidden layer, right,

00:45:03 the last mathematical operation is this nonlinear transformation, right?

00:45:07 So-called activation function.

00:45:10 So you can stack much, you can stack more such hidden layers.

00:45:16 Now this model can be much more expressive, okay?

00:45:20 So and then if you assume a fixed width, the number of neurons,

00:45:25 and then if you increase the depths, then you can mathematically prove that the power of the model is,

00:45:33 you know, growing in some way.

00:45:35 It's very easy to understand if you think about, you know, what we are having here,

00:45:39 is actually using a lot of piecewise linear facets to approximate a nonlinear manifold.

00:45:50 You see what I mean?

00:45:51 Right?

00:45:52 Right?

00:45:53 And the depths is, you know, something like the power, you know,

00:45:59 it's like the exponential term, like the power that gives you the number of such piecewise surfaces or facets you could have.

00:46:10 Okay?

00:46:11 So that is some way to quantify the expressiveness of neural networks, right?

00:46:15 You're basically asking yourself, how many pieces can I have, right?

00:46:21 So in this case, we have, you know, two hidden layers, and each of them is a nonlinear transform of the linear output,

00:46:30 which is called activation function sigma.

00:46:33 Okay?

00:46:34 And you have a lot of choices, right?

00:46:38 For your activation function, right?

00:46:42 ReLU, sigmoid, right?

00:46:44 This hyperbolic tangent, and so on.

00:46:49 There's also, you know, this other kind of choices, right?

00:46:55 Like a swoosh, right?

00:46:57 And how do you choose them?

00:47:01 And it actually needs a lot of knowledge about the neural network computation, right?

00:47:07 So we'll talk about them when we get to the neural network computation.

00:47:12 We're not talking about computation so far, right?

00:47:14 Remember, in the end of the optimization, we talked about computation.

00:47:18 Numerical optimization, right?

00:47:20 So some of this knowledge will be brought back when we talk about the neural network computation.

00:47:26 But if you are interested in knowing them, the conclusions so far, you can read this block.

00:47:35 Again, if you directly click the link, it may not work because every time when there's a hyphen,

00:47:41 somehow the PowerPoint converted PDF does not represent this hyphen correctly.

00:47:48 It's using a different ASCII.

00:47:50 So I apologize.

00:47:51 So I hope, you know, you can simply replace those ASCII with hyphens, okay?

00:47:57 Or Google this link.

00:47:59 Any questions so far?

00:48:01 Okay, so there's more.

00:48:06 There's more because I was asking at the beginning, right?

00:48:10 Have you learned neural networks?

00:48:12 Now we know that, yes, we have learned them.

00:48:14 We have learned linear regression is a neural network.

00:48:17 Logistic regression is a neural network.

00:48:20 And then MLP, right, is a neural network that we haven't learned.

00:48:25 But support vector machines like Nami was saying are actually neural networks as well.

00:48:32 So this was a 2002 paper, 2004 ICML paper, right?

00:48:38 So if you read this abstract, it's very interesting.

00:48:42 We first studied ways to control the capacity of perceptrons, right?

00:48:47 Mainly regularization parameters and early stopping.

00:48:51 Using the margin idea introduced with the margin idea.

00:48:56 Remember, support vector machines is about the margin maximization.

00:48:59 Showing that under simple conditions, a perception is equivalent to an SVM.

00:49:06 This is a very interesting, a lot of the proofs from that paper.

00:49:11 But under some simple conditions, a perceptron is equivalent to an SVM.

00:49:18 Moreover, under some assumptions, it also appears that multilayer perceptrons

00:49:24 are a kind of mixture of support vector machines, maximizing the margin in the hidden layer space.

00:49:30 Very nice results.

00:49:31 So if you want to improve your understanding of support vector machines and MLPs

00:49:38 and see the connection of them from the theoretical place,

00:49:43 I strongly recommend you to read this paper again from ICML.

00:49:48 Okay?

00:49:49 So these are the neural networks we have already learned, right?

00:49:53 Now let's write to summarize artificial neural networks, right?

00:49:57 But see them now from biological perspective.

00:50:02 Recall how AI as a word was created, right?

00:50:07 It's about modeling how humans think in a way that can enable machines to do something similar.

00:50:19 So if you start with the smallest computing unit, well arguably smallest computing unit,

00:50:30 because there's more underlying, molecular basis underlying this nucleus.

00:50:37 But if you're thinking about a single brain cell, right?

00:50:43 In this case, right?

00:50:46 We have the nucleus, right?

00:50:48 We have the cell body, we have the dendrites that are making connections, right?

00:50:53 And then these axons, right?

00:50:55 These impulses are then carried away from the cell body,

00:50:59 and then going through the branches of axons, and then reaching the terminals, right?

00:51:05 So this is actually how neural network gets started, right?

00:51:11 What is called a neural network, right?

00:51:13 It's called for that for a reason.

00:51:15 And then we are essentially, you know, using feed forward neural network, right?

00:51:21 Actually artificial neural network, but nobody is using artificial now because, you know,

00:51:26 it's getting so popular.

00:51:27 To emulate how a neuron compute, so this is a very simple model, right?

00:51:33 This is a very simple model, right?

00:51:35 So this could be the input axon from another neuron, the input x0, right?

00:51:43 And then reaching the synapse, right?

00:51:47 Parameterized by w0, and this could be a linear relationship, right?

00:51:54 Then there could be axons from other neurons, right?

00:51:58 Or other axons.

00:51:59 So there are different input x sub i's, and there are weighted sum and reached the cell body.

00:52:06 So that's the linear weighted sum.

00:52:11 And then it's nonlinearly activated.

00:52:16 That's your sigma function, right?

00:52:18 That's, this f is the sigma function you have seen before.

00:52:21 This is the classical, you know, feed forward neural network, right?

00:52:26 If you look at how this neuron is, you know, defined as a class,

00:52:32 and then forward neural network, feed forward neural network,

00:52:36 is a function defined for this class neuron, right?

00:52:41 So we basically have the cell body sum, which is the sum over all the inputs, right?

00:52:48 Times the self-weights, right?

00:52:50 Plus the self-bias.

00:52:52 Weights are w, bias are b, right?

00:52:55 And then there's some firing rate, which is essentially a sigmoid activation function, right?

00:53:02 Then you return the sigma of this linear function.

00:53:06 So that's really the foundation or the motivation,

00:53:11 as well as the mathematical foundation for feed forward neural networks, right?

00:53:18 Okay.

00:53:19 So now we know that, why we say this is a neural network, right?

00:53:25 Right?

00:53:26 At the beginning we said that we want you to understand these concepts.

00:53:33 Neurons, layers, and activation functions, right?

00:53:38 So I think that's a summary of what are they in the neural networks.

00:53:45 But I want you to see them as something very natural,

00:53:48 because if you look at how AI as a word or a field has been born,

00:53:55 you will see that, well, that's a natural choice, okay?

00:53:59 Okay.

00:54:00 Okay.

00:54:01 So, so far, any questions so far, we talked about the first take on neural networks.

00:54:08 We talked about the basic concepts, neurons, layers, and as well as some special cases, right?

00:54:20 Neurons, layers, activation functions, as well as some special cases, like linear regression,

00:54:26 or soft mass regression, or multinomial logistical regression, as well as MLP, right?

00:54:34 We also said that support vector machines are also equivalent to perceptrons.

00:54:39 So, MLP is a mixture of SVMs.

00:54:43 [Uncertain recognition] Ah.

00:54:44 [Uncertain recognition] Alright.