Lecture 15: Machine Learning in Practice - Diagnostics and Fixes

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 Okay, so let me continue. And as I said, we have talked about machine learning in concepts. We have talked about machine learning in theories. And one of the most important theory to me is a bias variance tradeoff, right? Which says that we know machine learning is not about fitting observed data points. It's about making predictions for unobserved data points.

00:00:28 So in that sense, we could look at the expected error, right? So the expected error on the test data. But where is that expectation coming from? The expectation is based on the fact that our training data, the observed data, could be sampled from a distribution, right? So we draw samples from that distribution.

00:00:52 So if we calculate the expected value of the generalization error, okay? The error is mean squared error. If you're doing regression, it's the generalization error because the error was evaluated on the test set, right?

00:01:09 And it's expected generalization error because that was expected or averaged over the distribution of the training set, right?

00:01:18 So we derived that. Again, I recommend you to look at the derivation and try to do that by yourself. It should be straightforward.

00:01:31 That you will see that there are three terms. One is squared bias and one is the variance and one is uncertainty in the measurement that we cannot control.

00:01:42 So if you want to reduce the generalization error, we can try to reduce the sum of the first two terms, right? But in the meantime, it's just too idealistic and actually impossible to reduce both simultaneously.

00:02:03 I asked you to look at the theory behind it. Have you taken a look at it?

00:02:07 Have you?

00:02:08 Yeah, the Kramer-Rey inequality, right? Which is relating the variance, right, to the information, right?

00:02:20 So I strongly recommend you to look into that. That actually tells you why there is a trade-off, right?

00:02:26 So then, in reality, we can make our models significantly reduce the variance part while sacrificing the bias a little bit, right?

00:02:39 How do we do that? Well, if you make the model a little bit regularized in its complexity, right?

00:02:47 Then the bias, you know, the complexity decreases, the bias increases, but hopefully the variance decreases much faster, right?

00:02:54 And we also introduced how do you quantify the model complexity, right?

00:03:00 Which is not only determined by the model, but actually also dependent on determined by the data, right?

00:03:07 So we used the what?

00:03:09 The...

00:03:09 What method did we use to quantify?

00:03:15 The shrinkage method, the LP norm, right? So that can be used as a penalty in your loss function, so that you can simultaneously do the minimization of the loss to the observed data and reducing the model complexity simultaneously, right?

00:03:42 So those are the theoretical concepts that are very important.

00:03:47 And we're actually going to connect to them again, right?

00:03:51 But today we're going to focus on, you know, practical machine learning.

00:03:55 Again, just like what I said in the case of the optimization,

00:04:00 actually you want to have a theoretical guideline when you are developing or even choosing optimization algorithms, right?

00:04:10 You actually have seen that when the LP norm is L2, it's actually a very simple optimization problem if the loss is mean squared error.

00:04:18 You actually have an analytical expression of the mean squared error plus L2 norm, right?

00:04:25 If it's L1 norm, well, it's a little bit more complicated.

00:04:31 It's still convex, but there's no smooth gradient anymore, right?

00:04:35 And then in that case, there are still algorithms, right?

00:04:40 Just not gradient-based algorithms.

00:04:42 And then if the P is between 0 and 1, then it's even more difficult.

00:04:47 It's non-convex, right?

00:04:49 But if P is less than or equal to 1 in general, you have the benefit of having the sparse solution.

00:04:54 So you actually have to decide, right, a lot of things depending on your goals, right?

00:05:01 How you choose the loss function and how do you do the regularization, okay?

00:05:06 So we'll connect with those a little bit, but today, you know, we would like to talk about machine learning in practice, okay?

00:05:14 This is largely based on the great slides, advice for applying machine learning from Android, okay?

00:05:24 I think it was used to shared somewhere.

00:05:27 I'll probably change the link in somewhere, but I thought this is very useful, so I'd like to talk about it a little bit, okay?

00:05:36 So I'm going to start with, you know, whether it's machine learning or deep learning in future, I think these are very useful.

00:05:44 So I'm going to start with a classification machine learning model, right, that I haven't talked about, but hear me out so that, you know, you can get it, right?

00:05:58 So previously, you remember that we use, for instance, a linear classifier, right?

00:06:06 W transpose times X.

00:06:08 If X also include a feature 1, then that corresponding W is W naught, right?

00:06:16 So then there is a linear decision boundary, right?

00:06:21 And then we have a linear classifier, right?

00:06:25 Okay, so that's what we have seen before, okay?

00:06:29 So in the case of the logistic regression compared to the linear regression, sorry, not linear regression, linear classifier, okay, sorry.

00:06:37 So in the case of the logistic regression, we still have a linear decision boundary, but that linear decision boundary is not about the score of each sample, okay?

00:06:53 So previously, we have a score of each sample, W transpose times X, okay?

00:06:58 But here, I still have W transpose times X, but it's not about the score, okay?

00:07:04 It's about the log odds, okay?

00:07:09 So this is a log of the odds.

00:07:11 What is an odd?

00:07:12 Odd is the probability of Y equal to 1, given X, divided by probability of Y equal to 0, given X.

00:07:18 So log odds, okay?

00:07:21 So the log odds is approximated by a linear decision boundary, equal to 0, okay?

00:07:28 And then if you know that the, you know, numerator and denominator add up to 1, right?

00:07:36 Because that's one of the three principles or three laws of probability, right?

00:07:42 Normalization.

00:07:44 Then you can calculate P of Y equal to 1, given X, the conditional probability that a sample X belonged to class 1, right?

00:07:54 And then you will find that it's a logistic function of this form, okay?

00:08:00 So sometimes people also write it as sigma of this function, of this linear function, right?

00:08:08 Right?

00:08:09 So as a reminder, the logistic function is like sigma of X equal to 1 over 1 plus E2 minus X, okay?

00:08:17 So you replace the X by this linear function in the logistic.

00:08:22 So that's the name logistic regression is coming from, okay?

00:08:26 So you still have a linear function, but it's no longer the decision boundary in the score space, but it's a decision boundary in the log odds space, okay?

00:08:41 So if you rewrite the probability, then you will find that it's actually a logistic function.

00:08:48 So in that sense, it's called a logistic regression, okay?

00:08:52 So if you rewrite the log odds, you are regressing the log odds, and the probabilities are a logistic function of a linear function, okay?

00:09:01 Of course, you can generalize it to nonlinear cases and so on, okay?

00:09:04 If we look at a logistic function just to have intuition, right?

00:09:11 So if you have sigma of X, that's a blue curve, right?

00:09:14 So when X equal to 0, you should reach 1 over 2, right?

00:09:19 So 0.5, like this point, right?

00:09:21 When X is 0, you should reach 0.5.

00:09:24 So if you replace X by 2X, then you have this cyan curve, which still pass 0.5, but it's getting like steeper transitions, right?

00:09:39 This S shape is having steeper transitions.

00:09:42 If you replace this X by X minus 2, then this is like what you learned in the signals and systems, right?

00:09:52 It's like what?

00:09:55 It's just a, you know, affine transformation on time, right?

00:09:59 So this is what?

00:10:02 This is minus 2.

00:10:05 So this is moving to the left or to the right.

00:10:09 Sigma of X minus 2 relative to sigma of X is moving to the right.

00:10:13 So it's advanced, right?

00:10:16 It's time advanced, right?

00:10:17 If you think about X as time.

00:10:19 Then if you have the combination, which is affine transformation of X, which is linear transformation,

00:10:25 then in this case, you have this red one.

00:10:28 So it's going to be shifted to left or right by how much?

00:10:32 0.5X plus 1.

00:10:41 So 0.5X compared to 2X, it's expanded, right?

00:10:52 So the signal is expanded, right?

00:10:54 So that's why compared to the blue, it's like stretch to the left and the right.

00:10:59 It's expanded because 0.5X plus 2.

00:11:01 And then it's expanded and then also shifted to the left, right?

00:11:08 By two units, right?

00:11:09 Depending on whether you shift or scale first, right?

00:11:12 But, you know, if you follow one convention, you're moving left by two units, okay?

00:11:17 It's 0.5 times X plus 2, right?

00:11:21 So these are the decision boundary.

00:11:24 In this way, you can see that depending on your learned W, right, you have the different logistic function, okay?

00:11:33 Okay, now let's rewrite what we had here before but put a subscript, I, as the index of the training example, okay?

00:11:44 And we also have a semicolon here to indicate what's behind the semicolon is a parameter, okay?

00:11:55 So it's a parameter.

00:11:56 So in this case, we are asking what's the conditional probability of example I being in class 1 or 0,

00:12:05 given the features XI and, of course, the parameter W, okay?

00:12:13 So based on our defined decision boundary, we can find that it's just a sigma function of this linear function, okay?

00:12:22 It's logistic in this linear function, all right?

00:12:26 Okay, so then, and then in this case, now we have to ask ourselves, right?

00:12:36 So if you have defined a new classifier, right?

00:12:39 In the past, I understand that you used this linear classifier in the score space,

00:12:45 and your goal was to intuitively maximize the margin, right?

00:12:52 And we derived, well, I didn't fully derive it, but I hope you have tried it, right?

00:12:57 I gave the hints.

00:12:58 We derived that the margin was proportional to 1 over the norm of W, right?

00:13:05 And then we can equivalently minimize the norm squared, L2 norm squared of W, right?

00:13:13 But in this case, if you want to find the best W,

00:13:17 given a set of examples I from 1 to N,

00:13:21 what would be your loss function?

00:13:26 What is an appropriate loss function?

00:13:41 What is an appropriate loss function here?

00:13:43 You are trying to adjust W for the given N training examples of paired XI, YI,

00:13:59 where I goes from 1 to N, so that some loss is optimized.

00:14:07 What is that loss in this case?

00:14:08 Right?

00:14:13 See, we are looking at the probabilities now, okay?

00:14:18 Yes.

00:14:22 What is an appropriate loss function here?

00:14:26 You must know.

00:14:26 Don't be shy.

00:14:27 What I wrote there?

00:14:34 Actually, it's one step before that, actually.

00:14:39 What I wrote here was already following some assumptions.

00:14:44 Okay?

00:14:45 So what is the probability that you are trying to optimize here?

00:14:49 The probability of all the Y, Y1 through YI through YN,

00:15:06 given all the XI, I from 1 to N, semicolon W.

00:15:12 Is that right?

00:15:12 The probability of observing all the Y, given all the X and W, right?

00:15:21 That is the principle.

00:15:23 Is that right?

00:15:24 Now, if I assume these training examples are IID,

00:15:31 identically and independently distributed,

00:15:34 then I can factorize that conditional probability

00:15:38 into the product of all these P's, right?

00:15:46 Am I right?

00:15:47 Don't know what I touched, but it looks very...

00:15:58 Do I agree?

00:15:59 So then the probability of the vector Y,

00:16:04 given the vector X and W,

00:16:06 where by vector I mean Y equal to Y1 through Y2

00:16:10 through YI through YN,

00:16:12 and X vector is X1 through XN, right?

00:16:16 So that conditional probability can be now factorized

00:16:20 as a product of all these P of YI given XI and W, right?

00:16:25 Then you have the...

00:16:28 Then you can, of course, maximize the product of the P, right?

00:16:32 Or you can maximize the log of the product of the P, right?

00:16:38 Which is equivalent.

00:16:39 And then if you maximize the log of the product,

00:16:42 then you are maximizing the sum of the log probability, right?

00:16:47 So you have to start with the principle.

00:16:49 So I know I put the final step there,

00:16:53 but that's not where things started, right?

00:16:56 Okay.

00:16:57 If we agree on that,

00:16:59 then we can focus on what's behind the summation

00:17:04 because for every example I,

00:17:06 what's behind is the same, right?

00:17:09 Okay.

00:17:11 Then how do we write this log of P, right?

00:17:14 So this is something just like the...

00:17:20 What we did in the...

00:17:24 Support vector machine, remember?

00:17:27 We had two kinds of constraints, right?

00:17:30 Whether you're blue or red points,

00:17:32 depending on which side of the decision boundary you are, right?

00:17:35 Your score may be bigger than or less than that threshold, right?

00:17:40 But we united them together into one constraint using yi, right?

00:17:45 So here it's the same.

00:17:46 So we are using yi here.

00:17:48 So if yi is equal to one,

00:17:50 it's log of this probability, right?

00:17:54 If yi is zero,

00:17:56 then only...

00:17:57 If yi is one,

00:17:58 then only the first term is non-zero,

00:18:02 and the second term is going to be zero,

00:18:03 and the first term is nothing but the probability of yi equal to one,

00:18:09 given xi and w, right here.

00:18:12 If y equal to zero,

00:18:14 then only the second term is non-zero,

00:18:17 and what we have here is exactly log of the probability of yi equal to zero,

00:18:24 given xi and w.

00:18:25 So that's what we have here.

00:18:27 Okay?

00:18:27 Okay?

00:18:28 So that's what we have here.

00:18:31 Okay?

00:18:31 So this is...

00:18:33 So what's the relationship between what I write right here

00:18:38 and binary cross-entropy?

00:18:45 You must be using my binary cross-entropy somewhere, right?

00:18:49 So what's the relationship between what I wrote here

00:18:52 and the binary cross-entropy?

00:18:56 Can someone try?

00:19:01 How many of you have used binary cross-entropy?

00:19:05 You must have used it.

00:19:08 No?

00:19:11 Binary cross-entropy?

00:19:13 If you use classification,

00:19:15 if you do any classification,

00:19:17 you must use it, right?

00:19:19 Okay, so I recommend you to look into that

00:19:21 and see what's the relationship with this one

00:19:24 and gain a better understanding.

00:19:26 Okay?

00:19:27 So again,

00:19:28 this is just a background.

00:19:30 Okay?

00:19:30 So this is a baseline model, right,

00:19:34 that I start with.

00:19:35 So my question is,

00:19:36 how do I diagnose?

00:19:37 Okay?

00:19:39 All right.

00:19:39 So I have a model like this, right?

00:19:42 So...

00:19:42 And I apply this model to,

00:19:45 as you know,

00:19:46 one of the early application

00:19:48 of machine learning.

00:19:51 Okay?

00:19:51 Not deep learning.

00:19:52 Machine learning.

00:19:53 That is spam detection.

00:19:55 Okay?

00:19:55 Nowadays,

00:19:56 most of the email,

00:19:58 probably actually all of the email

00:19:59 providers also give you

00:20:02 this spam detection, right,

00:20:05 automatically.

00:20:06 Sometimes it does make mistakes, right?

00:20:09 By the way,

00:20:11 which one annoys you more?

00:20:14 False positives or false negatives?

00:20:16 If I have a spam detector,

00:20:21 it's a classifier,

00:20:22 machine learning classifier.

00:20:24 Which one annoys you more?

00:20:26 Because that matters

00:20:27 in how I train my classifier, right?

00:20:30 Are you annoyed more

00:20:31 by the false positives?

00:20:35 Yes.

00:20:36 Or false negatives?

00:20:39 You are more annoyed

00:20:40 by the false positives.

00:20:41 You are more worried

00:20:43 about missing important emails

00:20:45 than get bothered

00:20:48 by probably five more

00:20:50 just spam emails

00:20:54 that just got escaped, right?

00:20:57 Probably.

00:20:59 Okay, so then we may have

00:21:00 to think about

00:21:01 what is the last function then, okay?

00:21:03 So let's say that

00:21:04 this is a very simple case

00:21:07 in the early days, right?

00:21:08 Right?

00:21:09 That you have

00:21:10 a small set of 100 words

00:21:12 to use as features, okay?

00:21:16 And in this case,

00:21:18 we use Bayesian logistic regression, okay?

00:21:23 And we are using gradient descent

00:21:27 to optimize this loss function, right?

00:21:30 And we found a 20% test error,

00:21:34 which is unacceptably high, okay?

00:21:36 Again, so in our case,

00:21:37 we are maximizing the minus loss,

00:21:41 which is minimizing the loss, right?

00:21:43 This is what you have seen before.

00:21:45 And also,

00:21:46 we are regularizing

00:21:48 this model complexity, right?

00:21:49 Here, this model parameter

00:21:51 is theta, right?

00:21:52 Okay, and I have

00:21:53 M training examples, okay?

00:21:56 So,

00:21:57 and also,

00:22:00 just,

00:22:01 you may wonder,

00:22:02 well,

00:22:03 why did I have to use

00:22:05 gradient descent?

00:22:06 There is a unique global optimum

00:22:08 for the maximum likelihood prediction

00:22:10 under the logistic regression,

00:22:12 but

00:22:12 this is the same like,

00:22:14 you know,

00:22:16 mean squared error

00:22:17 with LASO,

00:22:19 L1 penalty, right?

00:22:20 There is a unique global optimum,

00:22:23 but

00:22:23 in that case,

00:22:25 there is no,

00:22:26 you know,

00:22:26 gradient method

00:22:28 that

00:22:28 optimizer that works

00:22:30 because of the smoothness issue.

00:22:32 In this case,

00:22:34 you know,

00:22:35 there is also

00:22:36 no closed form solution,

00:22:38 but luckily,

00:22:38 you can still use

00:22:39 a gradient descent

00:22:40 to solve the problem,

00:22:41 okay?

00:22:43 Okay,

00:22:43 so we sometimes

00:22:44 use the W before,

00:22:46 right here,

00:22:46 we are using theta here,

00:22:48 so please don't be confused,

00:22:49 okay?

00:22:50 All right,

00:22:51 so what do you do next,

00:22:52 right?

00:22:52 So,

00:22:53 that is possible,

00:22:54 right?

00:22:54 Even nowadays,

00:22:56 right,

00:22:57 when we are using

00:22:57 a lot of fancier method,

00:22:59 whether it's some kind

00:23:00 of deep learning,

00:23:01 right?

00:23:01 The only difference

00:23:02 is now your data

00:23:03 is bigger,

00:23:05 and the cost

00:23:06 of evaluating

00:23:08 the model performance

00:23:10 as well as

00:23:13 the cost

00:23:14 to improve

00:23:15 the model performance

00:23:16 are also increasing,

00:23:18 actually.

00:23:19 So that just makes

00:23:20 diagnosis

00:23:20 even more important,

00:23:22 right?

00:23:22 So let's use this example

00:23:24 to see how we diagnose,

00:23:25 right?

00:23:27 Okay,

00:23:27 so let's see.

00:23:30 What do you do?

00:23:30 How do you diagnose,

00:23:31 right?

00:23:32 So let's ask

00:23:33 a similar question,

00:23:34 right?

00:23:35 So if you,

00:23:36 you know,

00:23:37 found some problems,

00:23:39 right,

00:23:39 some random noise

00:23:40 or some issues

00:23:42 of your car,

00:23:43 right?

00:23:44 Right?

00:23:44 What do you do,

00:23:46 right?

00:23:46 So,

00:23:47 there might be

00:23:49 different scenario,

00:23:51 but usually you send

00:23:52 it to a checkup,

00:23:53 right,

00:23:53 a diagnosis.

00:23:54 What you don't do,

00:23:57 what you never do

00:23:58 is try to replace

00:24:00 every part

00:24:00 one by one

00:24:02 and eventually see

00:24:03 whether it works or not.

00:24:05 You never do that,

00:24:06 right?

00:24:06 Because the cost

00:24:07 is

00:24:08 unreasonable,

00:24:11 but more importantly,

00:24:12 the practice

00:24:12 is not reasonable,

00:24:13 right?

00:24:14 But the funny thing

00:24:15 is that if you talk

00:24:16 about our research,

00:24:17 especially,

00:24:18 you know,

00:24:19 diagnosing,

00:24:20 sorry,

00:24:20 I shouldn't use

00:24:21 that word,

00:24:22 improving our

00:24:22 machine learning

00:24:23 model's performance,

00:24:26 what we don't realize

00:24:27 is that we are

00:24:28 much more often

00:24:29 doing something like that.

00:24:31 We start with this model,

00:24:33 we see,

00:24:33 oh,

00:24:33 the performance

00:24:34 is not good,

00:24:35 then we try

00:24:35 everything we can.

00:24:37 We try everything

00:24:38 we can

00:24:38 and then hope

00:24:39 that the performance

00:24:40 will increase,

00:24:41 right?

00:24:41 So,

00:24:42 like in this case,

00:24:43 right?

00:24:44 So you can try

00:24:45 using getting

00:24:46 more training examples,

00:24:48 right?

00:24:49 You can use

00:24:50 the same training

00:24:52 example,

00:24:52 but try bootstrapping,

00:24:54 right?

00:24:55 You can use

00:24:56 a smaller set of features,

00:24:57 you can use

00:24:58 a larger set of features,

00:24:59 right?

00:24:59 Or you just change

00:25:00 the features,

00:25:01 right?

00:25:02 For instance,

00:25:02 you feel like

00:25:03 maybe the words

00:25:05 from the email header

00:25:07 is more

00:25:09 implicative,

00:25:10 right?

00:25:11 Or the ones

00:25:12 in the email bodies

00:25:13 can be more

00:25:13 implicative,

00:25:14 right?

00:25:16 And then you think

00:25:17 that,

00:25:17 well,

00:25:17 I'm solving

00:25:18 an optimization problem,

00:25:19 maybe I can run

00:25:20 the gradient descent

00:25:21 for more iterations

00:25:22 so that I can get

00:25:24 a better quality

00:25:25 optimum,

00:25:26 right?

00:25:27 And then you realize

00:25:28 that, oh,

00:25:28 I learned all these

00:25:30 great theories

00:25:31 about the iterative

00:25:32 optimizers,

00:25:33 right?

00:25:33 We learned that

00:25:34 the steepest descent

00:25:35 was simple

00:25:36 but slow,

00:25:37 so maybe I can try

00:25:39 Newton's method.

00:25:41 Sorry,

00:25:42 it's simple

00:25:43 but fast,

00:25:44 right?

00:25:45 Well,

00:25:46 no, no,

00:25:47 I take it back.

00:25:47 It's simple

00:25:48 and every step

00:25:50 you just take

00:25:50 small amount

00:25:51 of calculations

00:25:52 but it's going

00:25:53 to take a lot

00:25:54 of steps

00:25:54 because convergence

00:25:56 rate is slow,

00:25:57 right?

00:25:58 And then you figure

00:25:59 that, well,

00:26:00 maybe I can try

00:26:01 Newton's method,

00:26:01 right?

00:26:02 Each step,

00:26:03 iteration,

00:26:03 I spend more time

00:26:05 calculating,

00:26:06 including the second

00:26:07 order,

00:26:07 right?

00:26:08 Information,

00:26:09 but maybe I can

00:26:10 take less steps,

00:26:11 right?

00:26:12 And what else?

00:26:13 I can try

00:26:14 the different

00:26:14 lambda,

00:26:15 right?

00:26:15 I can try

00:26:16 different lambda

00:26:17 or I just use

00:26:18 the support vector

00:26:19 machine instead

00:26:20 of this Bayesian

00:26:23 logistic regression,

00:26:24 okay?

00:26:27 These are all

00:26:28 possible

00:26:28 but the question

00:26:29 is,

00:26:30 just like the case

00:26:31 of the car,

00:26:31 right?

00:26:32 Would you just

00:26:32 try all of them

00:26:34 or just try them

00:26:36 one by one

00:26:37 or what's the order

00:26:38 you're going to take?

00:26:40 Any

00:26:40 suggestions?

00:26:43 These are all

00:26:44 things we can try.

00:26:48 How are you

00:26:48 going to decide?

00:26:59 Apparently,

00:27:00 we can do all

00:27:00 of them.

00:27:00 It's just going

00:27:01 to take much

00:27:02 longer time,

00:27:02 right?

00:27:05 But if you cannot

00:27:06 afford doing all

00:27:07 of them,

00:27:08 or more importantly,

00:27:10 you want to be

00:27:11 rationally satisfied

00:27:12 instead of trying

00:27:13 everything possible,

00:27:15 what would you do?

00:27:25 Just like what

00:27:26 you do with your

00:27:28 cars.

00:27:29 You send it to

00:27:30 a body shop

00:27:33 for a diagnosis

00:27:35 first.

00:27:36 In other words,

00:27:37 you have to figure out

00:27:38 what's the problem

00:27:39 first.

00:27:39 I know you say

00:27:40 that.

00:27:41 Oh, no, no, no.

00:27:41 I know the problem.

00:27:42 The problem is

00:27:43 that the test error

00:27:43 is high.

00:27:44 Yes,

00:27:45 that is the final

00:27:47 ultimate symptom.

00:27:49 But what was

00:27:50 the problem?

00:27:51 Okay,

00:27:51 what's the disease

00:27:52 that caused

00:27:53 the symptom?

00:27:54 That's the question

00:27:54 I'm trying to ask.

00:27:56 You need to figure out

00:27:57 what's the problem.

00:27:58 Okay?

00:28:00 Okay,

00:28:01 so the better

00:28:01 approach is,

00:28:02 as I said,

00:28:03 you try to run

00:28:04 diagnosis to figure out

00:28:05 what the problem is

00:28:06 and then fix

00:28:07 that problem.

00:28:09 Exactly.

00:28:10 Okay?

00:28:11 So,

00:28:12 in this case,

00:28:13 let's say

00:28:14 we start with a problem

00:28:15 that is very,

00:28:16 you know,

00:28:17 superficial,

00:28:18 just the phenomena

00:28:19 that the test error

00:28:21 is 20%,

00:28:22 unacceptably high.

00:28:23 Right?

00:28:24 So,

00:28:25 then you have to start

00:28:26 with some hypothesis.

00:28:27 Right?

00:28:28 So,

00:28:28 one hypothesis

00:28:29 we have learned before

00:28:30 was the overfitting

00:28:33 versus

00:28:33 underfitting.

00:28:35 Right?

00:28:36 We learned before

00:28:37 one perspective

00:28:38 that is

00:28:39 bias-variance trade-off.

00:28:41 Right?

00:28:41 I think you can use that

00:28:42 as one of your ways

00:28:43 to diagnose.

00:28:44 Remember,

00:28:45 there are so many

00:28:45 different ways

00:28:46 of diagnosing

00:28:47 and this is just

00:28:48 one way to diagnose.

00:28:50 Then each diagnosis

00:28:51 is actually following

00:28:52 some theoretical

00:28:53 foundations.

00:28:54 Right?

00:28:54 Here,

00:28:56 you might wonder,

00:28:57 well,

00:28:58 based on what I know

00:28:59 about the

00:28:59 bias-variance trade-off,

00:29:01 in my case,

00:29:03 am I

00:29:03 experiencing

00:29:04 the high-variance issue

00:29:06 that's overfitting

00:29:07 or it's probably

00:29:09 high-biased issue.

00:29:11 Right?

00:29:12 Right?

00:29:13 So,

00:29:13 that's one way.

00:29:14 Right?

00:29:14 So,

00:29:15 then we can

00:29:15 follow this direction

00:29:17 and do diagnosis.

00:29:19 Right?

00:29:19 The diagnosis is

00:29:20 look at the variance

00:29:22 and the bias.

00:29:24 Okay?

00:29:24 So,

00:29:25 the diagnostic,

00:29:26 in the case of the variance,

00:29:27 you're going to compare

00:29:28 this training error

00:29:31 with the validation error.

00:29:33 Right?

00:29:34 In the case of the bias,

00:29:35 you can also look at

00:29:36 your training error

00:29:37 and whether

00:29:39 it itself

00:29:41 is bad or not.

00:29:43 Right?

00:29:43 In the case of the variance,

00:29:44 you can compare the error

00:29:46 for the training set

00:29:47 and the validation set.

00:29:49 Right?

00:29:49 See how this

00:29:50 model behaves differently,

00:29:53 how the predictions

00:29:54 behave differently

00:29:55 for different data sets.

00:29:57 That's what the variance

00:29:57 is about.

00:29:58 Right?

00:29:59 Bias is about

00:30:00 how off the prediction

00:30:01 is compared

00:30:02 to the actual.

00:30:03 Right?

00:30:03 So,

00:30:04 you can look at

00:30:04 the value

00:30:06 of the training error

00:30:07 or the validation error

00:30:07 itself.

00:30:08 Right?

00:30:09 Rather than comparing

00:30:09 the two.

00:30:10 Right?

00:30:13 Okay.

00:30:13 So,

00:30:14 we actually have already

00:30:15 compared,

00:30:16 we have already

00:30:17 discussed this before.

00:30:18 Remember,

00:30:19 in this case,

00:30:20 we have an underlying

00:30:22 grain function

00:30:23 that is not known

00:30:24 but observed

00:30:25 with the noisy

00:30:27 observations,

00:30:29 which are the blue

00:30:30 circles.

00:30:31 Right?

00:30:31 So,

00:30:31 if you use

00:30:32 an overly complex

00:30:35 model,

00:30:36 like 9th order

00:30:37 polynomial,

00:30:38 you can fit

00:30:39 all the blue circles

00:30:41 perfectly,

00:30:42 but it's

00:30:44 of high variance.

00:30:46 Right?

00:30:46 Right?

00:30:48 So,

00:30:48 if imagine

00:30:49 that these blue circles,

00:30:50 any of them

00:30:51 just move a tiny bit,

00:30:53 then your circles would,

00:30:54 I'm sorry,

00:30:54 then your red function

00:30:55 will just change

00:30:57 drastically.

00:30:58 Right?

00:30:58 So,

00:30:58 that's overfitting.

00:30:59 Okay?

00:31:00 So,

00:31:00 now let's look at

00:31:02 an illustration here.

00:31:04 Okay?

00:31:05 So,

00:31:05 this is like

00:31:06 a dart board.

00:31:09 Right?

00:31:10 This is like a dart board.

00:31:11 Right?

00:31:12 So,

00:31:13 what you are trying to do,

00:31:14 right,

00:31:15 you are throwing

00:31:16 the dart

00:31:17 to the dart board,

00:31:19 and let's say,

00:31:20 you know,

00:31:21 this is,

00:31:21 the center,

00:31:22 right,

00:31:23 is perfect.

00:31:25 Right?

00:31:25 It's a perfect

00:31:26 machine learning model,

00:31:28 like this green function.

00:31:30 You're getting

00:31:31 the ground truth.

00:31:32 Okay?

00:31:32 Let's say that's,

00:31:33 that's what it corresponds to.

00:31:35 Okay?

00:31:37 And then,

00:31:38 let's imagine that

00:31:40 as your data changes,

00:31:42 your training data changes,

00:31:43 every time

00:31:44 your trained machine learning model

00:31:47 is gonna give a performance.

00:31:49 Right?

00:31:49 It's like,

00:31:50 you know,

00:31:50 this,

00:31:51 every time you throw the dart

00:31:52 to the board,

00:31:53 right?

00:31:53 So,

00:31:54 it's gonna land somewhere.

00:31:56 Right?

00:31:57 It's gonna land somewhere.

00:31:59 And,

00:32:00 the landing points,

00:32:02 right,

00:32:04 are shown,

00:32:05 if we just compare

00:32:06 among the landing points,

00:32:08 it's like comparing

00:32:09 the model predictions.

00:32:11 Right?

00:32:11 So,

00:32:12 the ideal case

00:32:13 is that

00:32:14 these landing points

00:32:15 are close to each other.

00:32:17 Meaning that

00:32:17 the model predictions

00:32:19 are close to each other.

00:32:20 Low variance.

00:32:21 Right?

00:32:22 But,

00:32:22 I mean,

00:32:22 in the meantime,

00:32:24 they are all close

00:32:25 to the origin.

00:32:26 Right?

00:32:27 Right?

00:32:28 Low bias.

00:32:29 Right?

00:32:29 So,

00:32:29 in other words,

00:32:30 the model predictions

00:32:32 differ from each other

00:32:33 very lowly,

00:32:34 and their error

00:32:36 is very low.

00:32:37 The distance to the origin.

00:32:39 Okay?

00:32:40 So,

00:32:40 this is the idea,

00:32:41 a low bias

00:32:42 and a low variance.

00:32:43 Right?

00:32:44 But,

00:32:44 unfortunately,

00:32:45 we don't

00:32:46 often

00:32:47 see that.

00:32:47 What often happens

00:32:48 is,

00:32:49 like in this case.

00:32:51 Right?

00:32:52 In this case,

00:32:53 if you look at

00:32:54 these model predictions,

00:32:56 right,

00:32:57 they are actually,

00:32:57 they are actually,

00:33:00 you know,

00:33:00 not,

00:33:01 well,

00:33:02 in this case,

00:33:03 at least we can see

00:33:05 that they are far away

00:33:06 from each other.

00:33:06 They are scattered

00:33:07 from each other.

00:33:09 Right?

00:33:09 The model predictions

00:33:09 are very different

00:33:10 from each other

00:33:11 as the data changes.

00:33:13 That means

00:33:13 it's high variance,

00:33:15 isn't it?

00:33:16 But in the meantime,

00:33:17 they are not

00:33:17 too far away

00:33:18 from the origin.

00:33:19 Right?

00:33:19 They are far away

00:33:20 from each other

00:33:21 but not far away

00:33:22 from the ground truth.

00:33:23 And their average

00:33:25 might be even

00:33:25 unbiased,

00:33:26 actually.

00:33:27 So,

00:33:27 in that case,

00:33:28 it's low bias

00:33:29 but high variance.

00:33:30 Right?

00:33:31 Another scenario

00:33:32 is that

00:33:33 these landing points

00:33:34 are all close

00:33:35 to each other,

00:33:36 meaning it's low variance,

00:33:37 but they are all far away

00:33:40 from,

00:33:41 you know,

00:33:43 the ground truth label.

00:33:45 So,

00:33:45 in that case,

00:33:46 it's high bias.

00:33:47 Right?

00:33:48 But of course,

00:33:49 sometimes what's breaking

00:33:50 you is that,

00:33:52 like in this case.

00:33:54 Right?

00:33:54 So,

00:33:55 your model predictions

00:33:56 are far away

00:33:57 from each other,

00:33:58 indicating a high variance.

00:34:00 In the meantime,

00:34:01 they are all far away

00:34:01 from the ground truth,

00:34:03 indicating a high bias

00:34:04 as well.

00:34:05 Right?

00:34:06 So,

00:34:06 this is one kind

00:34:08 of diagnosis

00:34:09 you can,

00:34:10 you can,

00:34:12 one kind of diagnosis

00:34:14 you can perform

00:34:14 that is looking

00:34:16 at whether your problem

00:34:17 is overfitting

00:34:19 or underfitting.

00:34:20 Right?

00:34:20 So,

00:34:21 you know what to do

00:34:22 when there's overfitting.

00:34:24 Right?

00:34:24 You know what to do

00:34:25 if there's overfitting.

00:34:26 What do we do

00:34:27 if there's overfitting?

00:34:29 If it's overfitting,

00:34:30 that means model

00:34:31 is too complex.

00:34:32 So,

00:34:32 we can do what?

00:34:33 We can reduce

00:34:34 the model complexity

00:34:35 by regularization

00:34:36 we can also use

00:34:38 ensemble methods

00:34:39 to reduce the variance.

00:34:41 Right?

00:34:42 So,

00:34:43 right?

00:34:43 So,

00:34:45 that's what we have learned.

00:34:46 But now,

00:34:47 let's make this diagnosis

00:34:49 a little bit more concrete

00:34:51 because this is an illustration.

00:34:53 This is an illustration,

00:34:55 right?

00:34:55 Which is very intuitive.

00:34:56 Right?

00:34:57 But let's make it more concrete.

00:34:59 Right?

00:35:00 Let's,

00:35:00 let's,

00:35:01 you know,

00:35:02 let's make it more concrete.

00:35:03 Okay?

00:35:03 So,

00:35:04 let me first repeat that.

00:35:08 Now,

00:35:08 we're looking at the diagnostics

00:35:10 of machine learning algorithms.

00:35:12 Okay?

00:35:12 So,

00:35:13 don't ask

00:35:13 why isn't my machine learning working

00:35:17 or,

00:35:18 you know,

00:35:18 why is this test error

00:35:20 too that high?

00:35:21 The right question

00:35:22 is,

00:35:23 you know,

00:35:24 what could be the problem

00:35:26 and how could I diagnose?

00:35:27 And if I determined the cause,

00:35:31 how would I change the model?

00:35:32 Right?

00:35:33 Okay.

00:35:34 So,

00:35:35 the one right,

00:35:36 you know,

00:35:37 the correct mentality

00:35:39 is let's use diagnostics

00:35:41 to figure out

00:35:41 what's the problem

00:35:42 and then fix them

00:35:43 correspondingly.

00:35:45 One kind of diagnostics

00:35:47 is,

00:35:47 as I said,

00:35:48 is to figure out

00:35:48 whether it's underfitting,

00:35:50 overfitting,

00:35:51 or maybe just doing fine.

00:35:52 Okay?

00:35:52 This is just one diagnostic.

00:35:54 There are other diagnostics

00:35:55 as well.

00:35:56 Okay?

00:35:56 So,

00:35:57 in this case,

00:35:58 we are trying to figure out

00:35:59 whether it's underfitting,

00:36:00 overfitting,

00:36:01 or probably both.

00:36:02 Right?

00:36:03 Then we can compare,

00:36:04 for instance,

00:36:05 the training error

00:36:05 and the validation error.

00:36:08 Right?

00:36:08 Right?

00:36:09 So,

00:36:09 if both errors

00:36:11 are high,

00:36:12 that means

00:36:13 probably a high bias

00:36:16 on the fitting issue.

00:36:17 Right?

00:36:18 If the training error

00:36:20 is low,

00:36:20 but the validation error

00:36:21 is high,

00:36:22 right,

00:36:24 you might be experiencing

00:36:26 the high variance issue,

00:36:28 which is overfitting.

00:36:29 Okay?

00:36:30 By the way,

00:36:30 you have to be careful.

00:36:31 In reality,

00:36:33 is your training set

00:36:34 and the validation set

00:36:35 from the same distribution

00:36:36 or not?

00:36:37 Right?

00:36:37 Because I have seen

00:36:38 sometimes people

00:36:39 actually use a much more

00:36:41 challenging validation set

00:36:42 than they found

00:36:42 that the performance

00:36:43 of validation set

00:36:44 is so bad.

00:36:45 Then,

00:36:46 you know,

00:36:47 maybe that really prevents you

00:36:48 from the diagnostics.

00:36:50 because here,

00:36:52 this diagnostic

00:36:52 is assuming

00:36:54 that training

00:36:55 and the validation

00:36:56 data are from

00:36:58 the same distribution.

00:37:00 Okay?

00:37:01 Right?

00:37:01 Otherwise,

00:37:02 the validation set

00:37:03 is going to be

00:37:04 in a target domain

00:37:06 that's very different

00:37:07 from the origin domain.

00:37:08 Then,

00:37:09 there is a transfer error,

00:37:10 of course.

00:37:11 Right?

00:37:11 So,

00:37:12 then we cannot just draw

00:37:14 that conclusion

00:37:14 that easily.

00:37:15 Right?

00:37:16 And,

00:37:17 a good case,

00:37:18 of course,

00:37:18 is training error

00:37:19 is low

00:37:19 and the validation error

00:37:20 is just a little bit higher.

00:37:22 That's a good fit.

00:37:24 Right?

00:37:24 So,

00:37:25 that's a good thing.

00:37:26 So,

00:37:26 now let's,

00:37:27 let's try to see that.

00:37:30 Right?

00:37:32 So,

00:37:32 let's see.

00:37:33 Right?

00:37:33 Let's say there is,

00:37:35 let's look at

00:37:37 this error level.

00:37:41 Right?

00:37:42 What I'm changing here

00:37:44 is the size

00:37:45 of the training set.

00:37:46 Okay?

00:37:47 And,

00:37:48 let's say that

00:37:48 there is an ideal,

00:37:50 desired performance

00:37:51 like here.

00:37:52 This is like the

00:37:54 center of the dartboard

00:37:56 you want to land in.

00:37:57 Okay?

00:37:57 That's the,

00:37:58 that's the ideal case.

00:37:59 Okay?

00:38:00 So,

00:38:02 let's see in this case,

00:38:03 right?

00:38:04 As the training set size

00:38:06 increases,

00:38:07 the test,

00:38:08 you know,

00:38:08 the training error

00:38:09 and the test error

00:38:11 are both getting closer

00:38:12 to the desired performance.

00:38:14 Okay?

00:38:14 Please don't get confused.

00:38:16 I'm not saying

00:38:17 that training error

00:38:17 is increasing.

00:38:18 Okay?

00:38:19 So,

00:38:19 this is like the,

00:38:20 approaching this desired

00:38:22 performance from both directions.

00:38:24 Okay?

00:38:25 So,

00:38:26 in other words,

00:38:27 as the training set size

00:38:28 increases,

00:38:30 right,

00:38:30 so both are getting closer

00:38:32 to the desired performance.

00:38:33 Right?

00:38:34 But what you see here

00:38:35 is the following,

00:38:38 that

00:38:39 the training error

00:38:40 is,

00:38:41 you know,

00:38:41 much smaller

00:38:42 compared to the test error.

00:38:43 right?

00:38:44 Or,

00:38:46 in our case,

00:38:46 I would rather call it

00:38:49 validation or development error.

00:38:51 Right?

00:38:51 Because I usually

00:38:52 do the test set

00:38:55 evaluation

00:38:56 in the end.

00:38:57 We don't,

00:38:58 you know,

00:38:58 we just deploy

00:38:59 the trained model once.

00:39:01 So,

00:39:01 usually,

00:39:01 I look at the validation error

00:39:03 instead.

00:39:05 Okay?

00:39:05 All right.

00:39:07 So,

00:39:07 in this case,

00:39:08 the training error

00:39:10 is low

00:39:10 and the validation error

00:39:11 is high.

00:39:12 So,

00:39:12 we call it

00:39:13 a typical

00:39:14 high variance issue.

00:39:15 Right?

00:39:16 So,

00:39:17 in this case,

00:39:18 we see that

00:39:21 if we keep increasing

00:39:22 the training set size,

00:39:23 it may help

00:39:24 because,

00:39:25 after all,

00:39:26 test error

00:39:27 is decreasing.

00:39:28 Right?

00:39:29 Just slower

00:39:29 than training error does.

00:39:31 So,

00:39:31 if you keep increasing

00:39:32 the training set size,

00:39:33 it can still

00:39:35 help,

00:39:37 but there is still

00:39:38 this gap

00:39:39 may get saturated.

00:39:40 So,

00:39:41 you need to do something.

00:39:42 Okay?

00:39:42 We'll talk about

00:39:43 something later.

00:39:45 Another case

00:39:46 is like this.

00:39:47 Right?

00:39:47 So,

00:39:47 in this case,

00:39:48 there's not

00:39:50 too much difference,

00:39:51 actually,

00:39:52 as the training set size

00:39:53 increases

00:39:53 between the training

00:39:54 and the validation set error.

00:39:56 That is good,

00:39:57 but,

00:39:58 unfortunately,

00:39:59 both are

00:40:00 unacceptably high

00:40:02 from the

00:40:03 desired performance.

00:40:05 So,

00:40:06 if that's the case,

00:40:07 it's a typical

00:40:08 high bias.

00:40:10 Right?

00:40:10 Because the variance

00:40:11 is low,

00:40:12 but the bias is high.

00:40:13 So,

00:40:13 these are

00:40:13 the diagnoses

00:40:15 you can look at.

00:40:16 Right?

00:40:16 So,

00:40:17 in practice,

00:40:18 sometimes,

00:40:18 people actually

00:40:19 don't change

00:40:20 the training set size

00:40:21 a lot sometimes,

00:40:22 but still,

00:40:23 you can

00:40:24 even do the diagnosis

00:40:27 during the training.

00:40:28 Right?

00:40:29 You can look at

00:40:29 the errors

00:40:31 over epochs,

00:40:32 even,

00:40:32 during training.

00:40:33 So,

00:40:35 now,

00:40:37 once you figure out

00:40:39 the problem,

00:40:42 now you can choose

00:40:44 the fixes accordingly.

00:40:45 Okay?

00:40:46 So,

00:40:47 as I said,

00:40:48 if you want to fix

00:40:50 the high variance,

00:40:51 you may want to get

00:40:52 more training examples.

00:40:54 So,

00:40:54 but remember,

00:40:55 this is only to fix

00:40:56 the high variance.

00:40:56 Right?

00:40:57 And,

00:40:59 smaller set of features

00:41:00 is what?

00:41:02 Is reducing

00:41:03 the model complexity.

00:41:05 Right?

00:41:05 So,

00:41:05 it also fixes

00:41:06 high variance.

00:41:07 Right?

00:41:08 If you use a larger

00:41:09 set of the features,

00:41:10 right,

00:41:11 to some extent,

00:41:11 it's increasing

00:41:13 the model complexity,

00:41:14 you may fix

00:41:17 a high bias issue.

00:41:18 Or,

00:41:19 you are using

00:41:19 the email header features,

00:41:20 using more

00:41:21 informative features.

00:41:24 Right?

00:41:24 In this case,

00:41:24 you can also fix

00:41:25 a high bias issue.

00:41:26 Right?

00:41:27 So,

00:41:28 as you can see,

00:41:29 this is,

00:41:30 you know,

00:41:32 what you will do

00:41:33 after you figure out

00:41:34 what's the problem.

00:41:35 Right?

00:41:35 You wouldn't try

00:41:35 all of them,

00:41:36 would you?

00:41:37 Right?

00:41:37 Okay,

00:41:38 then you ask,

00:41:39 well,

00:41:40 what about the remainder?

00:41:41 Well,

00:41:42 the remainder

00:41:42 has not,

00:41:44 are fixes

00:41:45 that have nothing

00:41:47 to do with

00:41:47 the first diagnostic

00:41:48 we talked about.

00:41:49 Because the first diagnostic

00:41:50 is only about

00:41:51 overfitting

00:41:53 or underfitting.

00:41:54 Okay,

00:41:54 we haven't talked

00:41:55 about other

00:41:55 diagnostics yet.

00:41:57 What other

00:41:58 diagnostic?

00:42:00 Don't forget

00:42:01 that we are,

00:42:02 we are solving

00:42:02 an optimization problem.

00:42:05 Right?

00:42:07 So,

00:42:09 one question

00:42:10 you have to ask

00:42:11 is,

00:42:13 if I'm optimizing

00:42:16 the right function,

00:42:19 is my optimization

00:42:20 algorithm converging?

00:42:23 Okay,

00:42:23 let's say

00:42:24 I'm optimizing

00:42:25 the right function,

00:42:26 but is it converging?

00:42:28 Right?

00:42:28 Did I report

00:42:29 a converging solution?

00:42:31 Right?

00:42:31 So,

00:42:32 in other words,

00:42:32 it's optimizers

00:42:33 convergence issue,

00:42:35 or maybe I'm not even

00:42:36 optimizing the right function.

00:42:38 Right?

00:42:39 So,

00:42:40 in other words,

00:42:41 you have to figure out,

00:42:43 previously the diagnostic

00:42:44 is about your model

00:42:45 performance,

00:42:46 whether it's

00:42:46 high variance

00:42:47 or high bias.

00:42:49 Then,

00:42:50 this diagnostic

00:42:50 is about

00:42:51 optimization,

00:42:53 formulation,

00:42:54 and algorithm.

00:42:56 Right?

00:42:57 So,

00:42:57 let's try to figure out

00:42:58 both.

00:42:59 Right?

00:42:59 Whether your formulation

00:43:00 is correct.

00:43:00 The choice

00:43:01 of the loss function

00:43:02 is agreeing

00:43:04 with what you want.

00:43:05 Right?

00:43:06 And,

00:43:07 if that's the case,

00:43:08 whether the optimizer

00:43:09 is doing the right thing

00:43:10 to get the optimal parameters.

00:43:12 Okay?

00:43:14 Okay,

00:43:14 so,

00:43:15 first of all,

00:43:16 let's ask what we care about

00:43:17 for this spam detection.

00:43:20 Right?

00:43:20 So,

00:43:21 as I was asking you earlier,

00:43:22 right,

00:43:23 do you care about

00:43:24 the false positives

00:43:25 or false negatives

00:43:26 more?

00:43:28 Right?

00:43:28 But,

00:43:29 in general,

00:43:30 we can say that

00:43:31 there is probably

00:43:32 an assessment,

00:43:34 right,

00:43:34 like an indicator,

00:43:36 right,

00:43:36 which is a weighted sum

00:43:38 of all these indicator functions.

00:43:40 Right?

00:43:42 Right?

00:43:42 So,

00:43:43 this is

00:43:44 when the predicted

00:43:47 is agreeing

00:43:48 with the actual

00:43:49 mail,

00:43:51 email label.

00:43:52 Right?

00:43:52 We can also

00:43:53 try to,

00:43:54 we can try to maximize

00:43:55 this one,

00:43:56 right,

00:43:56 or we can try to minimize

00:43:57 when this

00:43:58 equality changes

00:44:00 to inequality.

00:44:01 Right?

00:44:02 But,

00:44:03 depending on

00:44:04 the Y

00:44:05 being the spam

00:44:06 or non-spam,

00:44:08 we may want to give

00:44:09 different weights

00:44:10 to that correct decision.

00:44:12 Right?

00:44:13 Right?

00:44:13 So,

00:44:13 in other words,

00:44:14 you may care about

00:44:15 this one,

00:44:17 this measurement

00:44:19 that is more

00:44:21 reflecting

00:44:22 the actual performance

00:44:24 of this spam detection.

00:44:26 Okay?

00:44:26 By the way,

00:44:27 this is not always,

00:44:28 this is not always

00:44:30 the loss function

00:44:31 you use.

00:44:32 Okay?

00:44:32 Because you see

00:44:33 in the different

00:44:33 machine learning models,

00:44:36 they have different

00:44:36 principles

00:44:38 and different loss functions.

00:44:40 Whether it's

00:44:40 maximum margin,

00:44:42 I'm talking about

00:44:42 the classifiers for now,

00:44:44 or in this case,

00:44:45 the maximum likelihood,

00:44:48 or the,

00:44:49 you know,

00:44:49 maximum

00:44:50 posterior,

00:44:52 right?

00:44:52 So,

00:44:53 there are

00:44:53 different principles.

00:44:54 Right?

00:44:55 And those principles

00:44:56 determine the loss function,

00:44:58 but they

00:44:59 don't have to be

00:45:01 actually the

00:45:01 performance evaluator.

00:45:03 So,

00:45:04 maybe in this case,

00:45:05 you care about

00:45:05 the performance

00:45:06 that can be

00:45:07 valid in this way.

00:45:08 Okay?

00:45:09 All right.

00:45:10 So,

00:45:10 now,

00:45:11 let's

00:45:12 look at

00:45:14 the following.

00:45:16 Okay?

00:45:17 Let's try to figure out

00:45:18 whether it's

00:45:19 a formulation issue

00:45:20 or optimizer issue.

00:45:22 Okay?

00:45:23 Okay.

00:45:26 For,

00:45:27 we first can figure out

00:45:28 whether it's

00:45:29 a formulation issue,

00:45:30 right?

00:45:30 For instance,

00:45:31 you wondered

00:45:32 whether I should

00:45:32 use support vector

00:45:33 machine,

00:45:34 right?

00:45:35 Versus

00:45:36 evasion logistic

00:45:37 regression,

00:45:37 right?

00:45:37 Let's assume

00:45:38 that for each

00:45:39 model,

00:45:40 the formulation

00:45:41 is already

00:45:42 correct in the

00:45:42 sense that

00:45:43 you have taken

00:45:44 care of this

00:45:44 lambda or c,

00:45:45 these hyperparameters.

00:45:47 Okay?

00:45:47 So,

00:45:47 for each

00:45:48 formulation,

00:45:49 the corresponding

00:45:49 hyperparameters

00:45:50 is already

00:45:51 taken care of.

00:45:51 Let's assume

00:45:52 that.

00:45:53 Okay?

00:45:54 Once you

00:45:55 have that,

00:45:56 right?

00:45:56 Because you see

00:45:57 that these

00:45:58 different formulations

00:45:59 are different,

00:46:00 right?

00:46:01 Vaspation logistic

00:46:02 regression is

00:46:02 maximizing this

00:46:03 j,

00:46:04 right?

00:46:04 And then

00:46:05 support vector

00:46:05 machine is

00:46:06 minimizing

00:46:06 this

00:46:07 one over

00:46:09 the

00:46:11 squared margin,

00:46:13 right?

00:46:14 Plus

00:46:15 the slack

00:46:16 variables

00:46:17 for the

00:46:17 soft

00:46:18 right?

00:46:22 So,

00:46:22 now,

00:46:25 okay,

00:46:25 so then

00:46:26 let's see.

00:46:28 First,

00:46:29 let's say

00:46:30 the support vector

00:46:31 machine gives you

00:46:32 a parameter

00:46:32 theta

00:46:32 SVM,

00:46:34 and

00:46:34 Bayesian logistic

00:46:35 regression gives you

00:46:36 the parameter

00:46:37 theta

00:46:37 BRR,

00:46:38 okay?

00:46:39 You can ask

00:46:40 yourself,

00:46:41 which one performs

00:46:42 better

00:46:43 in the eyes

00:46:45 of the

00:46:47 weighted accuracy

00:46:47 that you care

00:46:48 about,

00:46:49 okay?

00:46:49 You can plug in

00:46:50 these two models

00:46:51 and ask

00:46:52 which one has

00:46:53 better performance,

00:46:54 right?

00:46:55 Okay,

00:46:56 let's say that

00:46:57 you found actually

00:46:58 support vector

00:46:58 machine even

00:46:59 does better

00:47:00 in the sense

00:47:02 of this

00:47:02 performance

00:47:03 evaluation

00:47:03 than

00:47:04 Bayesian

00:47:05 logistic

00:47:05 regression,

00:47:06 okay?

00:47:08 Well,

00:47:08 then you ask

00:47:09 the following,

00:47:10 is my Bayesian

00:47:13 logistic

00:47:13 regression

00:47:13 performing

00:47:16 worse

00:47:17 than support

00:47:17 vector machine

00:47:18 because

00:47:19 it's

00:47:20 Bayesian

00:47:21 logistic

00:47:21 regression

00:47:22 or because

00:47:23 I actually

00:47:24 didn't optimize

00:47:25 this Bayesian

00:47:26 logistic regression

00:47:27 problem,

00:47:29 okay?

00:47:30 Right?

00:47:30 I didn't find

00:47:31 the optimum.

00:47:32 So this is,

00:47:33 right?

00:47:33 So to answer

00:47:34 this question,

00:47:36 right,

00:47:36 you can compare

00:47:37 both support

00:47:39 vector machine

00:47:40 and the Bayesian

00:47:40 logistic regression

00:47:42 in the same

00:47:44 objective function

00:47:45 as the Bayesian

00:47:46 logistic regression

00:47:47 J,

00:47:48 okay?

00:47:50 Remember,

00:47:50 Bayesian logistic

00:47:51 regression is

00:47:51 trying to maximize

00:47:52 that J,

00:47:53 right?

00:47:54 That's the battlefield

00:47:55 of Bayesian

00:47:56 logistic regression.

00:47:57 If your

00:47:58 support vector

00:47:59 machine's

00:48:00 answer

00:48:01 is even

00:48:02 better than

00:48:03 the Bayesian

00:48:03 logistic regression

00:48:04 in its own

00:48:05 battlefield,

00:48:07 the logist,

00:48:08 sorry,

00:48:08 the last function

00:48:09 that BRR

00:48:10 is trying

00:48:10 to maximize,

00:48:12 that means

00:48:12 what?

00:48:15 The optimization

00:48:16 algorithm,

00:48:18 or optimizer,

00:48:19 did not give us

00:48:20 an optimal solution,

00:48:22 or far from it,

00:48:23 right?

00:48:23 Because we were

00:48:24 trying to maximize

00:48:25 J,

00:48:26 but,

00:48:27 you know,

00:48:27 actually,

00:48:28 when our optimizer

00:48:30 trying to maximize

00:48:31 for J,

00:48:32 actually,

00:48:32 it's worse

00:48:33 than support vector

00:48:34 machine,

00:48:35 which is trying

00:48:35 to optimize

00:48:35 for something

00:48:36 else,

00:48:37 but it turned

00:48:38 out to be

00:48:38 even better,

00:48:40 also in,

00:48:41 you know,

00:48:41 if also

00:48:42 evaluated in J,

00:48:43 is that right?

00:48:44 Right?

00:48:45 So,

00:48:45 it's,

00:48:45 you know,

00:48:46 it's like,

00:48:48 you know,

00:48:50 you pick

00:48:50 the high jump

00:48:52 champion,

00:48:53 right?

00:48:54 And then

00:48:54 compete with you

00:48:55 on the 100-meter

00:48:57 track,

00:48:57 and then actually,

00:48:58 you know,

00:48:59 he or she

00:48:59 outperforms you,

00:49:00 then you may

00:49:01 wonder

00:49:02 whether your race

00:49:04 of the 100-meter

00:49:05 was competitive

00:49:06 enough,

00:49:07 right?

00:49:07 So,

00:49:08 you need a better

00:49:08 optimizer,

00:49:09 so that's one

00:49:10 issue,

00:49:10 right?

00:49:11 So,

00:49:12 in the case one,

00:49:13 as I said,

00:49:13 that support vector

00:49:14 machine has better

00:49:15 performance

00:49:16 in our performance

00:49:18 assessment,

00:49:19 and it even beat

00:49:21 the BRR in its own

00:49:22 track,

00:49:23 in its own loss

00:49:24 function,

00:49:25 right?

00:49:25 So,

00:49:26 that means

00:49:26 our optimization

00:49:28 for the Bayesian

00:49:29 logistical regression

00:49:30 failed to maximize

00:49:31 J,

00:49:31 far from it,

00:49:32 right?

00:49:33 So,

00:49:33 the problem is

00:49:34 now with the

00:49:35 convergence of

00:49:35 the optimizer,

00:49:37 right?

00:49:37 Right?

00:49:38 So,

00:49:38 that's one

00:49:38 possibility,

00:49:41 right?

00:49:41 The other possibility

00:49:42 is that,

00:49:43 yes,

00:49:43 the support vector

00:49:44 machine is better

00:49:45 than BRR

00:49:45 in the performance

00:49:46 we care,

00:49:48 but its

00:49:50 solution is not

00:49:53 necessarily better

00:49:53 than BRR

00:49:54 in the track

00:49:55 of BRR.

00:49:56 In other words,

00:49:57 if we evaluated

00:49:58 these two,

00:49:59 both in J,

00:50:00 this loss

00:50:01 function of

00:50:02 BRR,

00:50:04 BRR is,

00:50:05 you know,

00:50:05 still better.

00:50:06 Support vector

00:50:07 machines model

00:50:07 are not

00:50:08 naturally better,

00:50:09 okay?

00:50:10 So,

00:50:11 this means

00:50:11 that probably,

00:50:13 probably,

00:50:15 right?

00:50:15 I'm not saying

00:50:16 there's guaranteed

00:50:17 no issue

00:50:17 with optimizer,

00:50:19 but so far

00:50:20 there's,

00:50:21 you know,

00:50:21 nothing suspicious

00:50:22 about optimizer.

00:50:24 There's more

00:50:25 issues similarly

00:50:26 with the loss

00:50:26 function similarly,

00:50:28 right?

00:50:28 Because if I,

00:50:30 you know,

00:50:30 because support vector

00:50:31 machine apparently

00:50:32 uses a different

00:50:33 loss function,

00:50:34 right?

00:50:35 And it's not

00:50:36 better than the BRR

00:50:37 in J,

00:50:38 this optimal solution,

00:50:40 but the support vector

00:50:42 machine chosen

00:50:42 by its own

00:50:43 margin maximization

00:50:45 performed better,

00:50:46 right?

00:50:48 So this means

00:50:49 that probably,

00:50:50 in this case,

00:50:51 support vector

00:50:52 machine is a better

00:50:53 formulation,

00:50:55 optimization,

00:50:55 formulation

00:50:56 than Bayesian

00:50:57 logistic regression,

00:50:58 isn't it?

00:50:59 Right?

00:50:59 So this means

00:51:00 that maybe

00:51:01 the J of theta

00:51:02 with this Bayesian

00:51:03 logistic regression,

00:51:04 right,

00:51:04 is probably the wrong

00:51:05 function to maximize,

00:51:07 and what you care

00:51:08 about is A of theta,

00:51:10 then the loss function

00:51:11 of support vector

00:51:12 machine may agree

00:51:13 with your A of theta

00:51:14 closer,

00:51:15 okay?

00:51:16 Then you may wonder,

00:51:17 well,

00:51:17 if I care about

00:51:18 A of theta,

00:51:19 why don't I directly

00:51:20 optimize A of theta

00:51:22 in training my model?

00:51:24 So that's a complicated

00:51:26 question, right?

00:51:27 First of all,

00:51:28 not all of such

00:51:29 A of theta

00:51:30 can be easily

00:51:32 analytically expressed

00:51:34 and then solved

00:51:36 by gradient algorithms,

00:51:37 right?

00:51:38 But it's half and half.

00:51:40 One example of

00:51:41 is area under

00:51:42 the precision recall curve,

00:51:44 right?

00:51:44 If you try to

00:51:45 maximize the area

00:51:47 under the precision recall curve,

00:51:49 you rarely see that

00:51:50 directly goes into

00:51:51 the loss function

00:51:52 of training your

00:51:53 classifier, right?

00:51:54 But now people

00:51:55 are developing

00:51:56 better and better

00:51:57 algorithms so that

00:51:58 you can have

00:51:59 surrogate functions

00:52:00 who are easy

00:52:01 to optimize,

00:52:02 but whose optimality

00:52:04 more or less

00:52:05 aligns with the

00:52:07 optimum area

00:52:08 under the precision recall curve.

00:52:10 Okay,

00:52:10 that's a possibility,

00:52:11 okay?

00:52:12 But it still

00:52:13 remains a challenge.

00:52:14 The second one

00:52:15 is that different model

00:52:15 have their different

00:52:16 assumptions, right?

00:52:18 So they not necessarily

00:52:19 all agree with

00:52:20 this performance measure

00:52:22 you care about,

00:52:22 A of theta, right?

00:52:23 So sometimes

00:52:25 as a trade-off,

00:52:25 some people just

00:52:26 use A of theta

00:52:27 as the validation

00:52:30 assessment.

00:52:31 So when you

00:52:32 choose a hyperparameters,

00:52:34 you evaluate A of theta

00:52:36 on the validation set,

00:52:38 and then that's

00:52:39 something that people do,

00:52:40 right?

00:52:41 But there are

00:52:42 all kinds of possibilities.

00:52:43 But so far,

00:52:44 let's come back

00:52:45 to the second diagnostic.

00:52:46 It's about

00:52:47 optimization, right?

00:52:48 Whether we are

00:52:49 looking at the

00:52:51 correct optimization

00:52:52 formulation or

00:52:53 objective,

00:52:55 or are we using

00:52:56 the correct optimizer

00:52:57 or optimization

00:52:58 algorithm, right?

00:53:00 So if it's

00:53:01 an algorithm issue,

00:53:02 then I probably

00:53:03 should make

00:53:04 it more likely

00:53:05 to converge,

00:53:06 like run it

00:53:07 more iterations

00:53:08 or try Newton's method,

00:53:10 right?

00:53:11 In the case of

00:53:11 deep learning,

00:53:12 you may use

00:53:13 different methods,

00:53:14 change

00:53:15 stochastic descent

00:53:17 to

00:53:18 atom

00:53:19 or even

00:53:20 something more

00:53:21 sophisticated,

00:53:22 right?

00:53:23 Because,

00:53:24 why?

00:53:24 Because atom

00:53:25 uses more information

00:53:26 like momentum,

00:53:27 right?

00:53:28 Than stochastic

00:53:28 descent.

00:53:31 You can also

00:53:31 try more

00:53:32 sophisticated algorithm,

00:53:33 like in the

00:53:34 low-dimensional case,

00:53:35 you can try

00:53:36 Newton's method,

00:53:36 but in the deep

00:53:37 learning,

00:53:37 as I said,

00:53:38 you may consider

00:53:38 like atom

00:53:40 and others

00:53:41 that use

00:53:41 second-order information

00:53:43 or momentum

00:53:43 or more information

00:53:44 than gradients,

00:53:46 right?

00:53:47 If it's

00:53:48 an objective issue,

00:53:49 maybe your lambda

00:53:50 was an issue,

00:53:51 okay?

00:53:51 I previously said

00:53:52 I hope your lambda

00:53:53 was fine,

00:53:54 but probably your lambda

00:53:55 is an issue,

00:53:56 okay?

00:53:57 But if your lambda

00:53:59 is not the issue,

00:54:00 then maybe you

00:54:01 should change

00:54:02 the loss function

00:54:03 and the model

00:54:04 totally,

00:54:05 right?

00:54:05 So in this case,

00:54:06 probably changing it

00:54:08 to a support vector

00:54:09 machine,

00:54:09 which is actually

00:54:10 agreeing with your

00:54:11 performance metric

00:54:12 even more.

00:54:13 Any question?

00:54:16 So now that you see

00:54:18 we have two kinds

00:54:20 of diagnostics,

00:54:21 right?

00:54:21 One is about

00:54:22 the model behavior,

00:54:23 right?

00:54:24 Right?

00:54:25 From the perspective

00:54:26 of bias and variance.

00:54:28 One is from

00:54:29 the optimization,

00:54:30 right?

00:54:30 From the,

00:54:31 you know,

00:54:31 perspective

00:54:32 algorithm

00:54:34 and objective

00:54:35 or formulation,

00:54:36 right?

00:54:36 So depending

00:54:38 on the outcome

00:54:39 outcome of each

00:54:40 diagnostic,

00:54:42 we can choose

00:54:43 the corresponding

00:54:45 fixes,

00:54:46 right?

00:54:46 So in other words,

00:54:47 what you do is

00:54:48 starting from the far

00:54:50 right,

00:54:50 there is something

00:54:51 called diagnostic.

00:54:52 Based on the

00:54:53 diagnostic,

00:54:54 you figure out

00:54:54 that I want to fix

00:54:55 a high variance

00:54:56 or a high bias

00:54:57 or optimization

00:54:59 algorithms

00:54:59 or optimization

00:55:01 objective

00:55:01 and then you

00:55:02 decide to use

00:55:03 one of them.

00:55:04 Okay?

00:55:04 So that's

00:55:05 the direction.

00:55:07 It's not the other

00:55:08 way around

00:55:09 that you try

00:55:09 all the possible

00:55:10 fixes and write

00:55:11 all of them

00:55:12 down and then

00:55:13 try all of them,

00:55:14 right?

00:55:15 Okay,

00:55:18 just some

00:55:19 summary of the

00:55:20 quick fixes,

00:55:21 right?

00:55:21 For the high variance,

00:55:22 we can try more

00:55:23 examples,

00:55:24 but if you cannot

00:55:25 afford it,

00:55:26 try a bagging,

00:55:27 right?

00:55:28 And then

00:55:29 try less features,

00:55:31 right?

00:55:32 Try regularization,

00:55:33 right?

00:55:34 If you try to fix

00:55:35 a high bias,

00:55:36 you may want to

00:55:37 try a larger set

00:55:38 of features

00:55:38 with more information,

00:55:40 right?

00:55:40 That really is the

00:55:41 case.

00:55:42 I have seen so

00:55:43 many papers

00:55:44 focusing on

00:55:45 the deep

00:55:46 learning

00:55:47 architecture

00:55:47 so much.

00:55:49 They introduced

00:55:49 very sophisticated

00:55:50 architecture,

00:55:52 but there are

00:55:53 two scenarios,

00:55:54 okay?

00:55:55 Two possibilities.

00:55:57 One possibility

00:55:57 is that their

00:55:58 performance did

00:55:59 improve compared

00:56:00 to the state of

00:56:01 art,

00:56:02 but not really

00:56:03 because of the

00:56:03 architecture,

00:56:04 but because of

00:56:05 their features

00:56:06 were much more

00:56:07 indicative

00:56:09 and informative.

00:56:11 That's one

00:56:12 possibility.

00:56:12 possibility.

00:56:13 The second

00:56:14 possibility is

00:56:15 that they

00:56:16 tried a lot

00:56:17 of things,

00:56:17 but they just

00:56:18 cannot beat

00:56:19 the state of

00:56:19 art.

00:56:20 Maybe somewhere

00:56:21 here they're

00:56:22 doing slightly

00:56:22 better,

00:56:23 but there,

00:56:24 you know,

00:56:24 it's like

00:56:24 roughly the

00:56:26 same.

00:56:27 Then people

00:56:27 ask why bother

00:56:28 trying so many

00:56:29 things,

00:56:29 right?

00:56:31 And very

00:56:32 often they

00:56:33 just forget

00:56:33 about an

00:56:34 easy,

00:56:36 well,

00:56:36 it's conceptually

00:56:37 easy,

00:56:37 just practically

00:56:38 difficult thing.

00:56:39 That is,

00:56:41 come up

00:56:41 with smart

00:56:42 features.

00:56:43 I know

00:56:43 everybody is

00:56:43 talking about

00:56:44 end-to-end

00:56:44 as if you

00:56:45 can stack

00:56:46 the original

00:56:48 raw data

00:56:49 into the

00:56:50 input end

00:56:51 and then

00:56:51 hope the

00:56:52 output end

00:56:52 will give

00:56:53 you the

00:56:54 perfect solution

00:56:54 when you train

00:56:56 the model

00:56:56 enough.

00:56:57 That does not

00:56:58 happen that

00:56:58 way.

00:56:59 Actually,

00:56:59 you still have

00:57:00 to spend a

00:57:00 lot of time

00:57:01 on features,

00:57:02 right?

00:57:03 But that

00:57:04 is painful

00:57:04 for people

00:57:05 who are

00:57:06 not familiar

00:57:07 with the

00:57:07 domain

00:57:07 knowledge,

00:57:09 right?

00:57:10 If you

00:57:10 just treat

00:57:11 the protein

00:57:11 sequence as

00:57:12 an alphabet

00:57:13 of,

00:57:14 I'm sorry,

00:57:15 as a text

00:57:15 written in

00:57:16 alphabet of

00:57:16 20 letters,

00:57:17 that's super

00:57:17 easy.

00:57:18 But if I

00:57:18 ask you

00:57:19 to do,

00:57:20 well,

00:57:20 what about

00:57:21 the position

00:57:21 specific

00:57:22 sequence,

00:57:24 I'm sorry,

00:57:24 position specific

00:57:26 scoring matrix

00:57:27 or profile,

00:57:28 then you need

00:57:29 to do the

00:57:29 multiple sequence

00:57:30 alignment and

00:57:31 construct those.

00:57:32 And there are

00:57:32 other kind of

00:57:33 things,

00:57:33 right?

00:57:34 Then you may

00:57:34 figure that out

00:57:35 that's too

00:57:36 much painful,

00:57:37 right?

00:57:38 But we

00:57:39 really need

00:57:40 to pay

00:57:41 attention to

00:57:42 this issue

00:57:43 because very

00:57:44 often what I

00:57:44 see is that

00:57:46 our models

00:57:47 are both

00:57:48 high variance

00:57:50 and high bias.

00:57:51 And the high

00:57:52 variance is

00:57:52 easier to treat

00:57:53 actually.

00:57:54 The high bias,

00:57:55 you actually need

00:57:56 more information

00:57:56 from the domain.

00:57:58 That's an

00:57:58 issue,

00:57:59 okay?

00:58:01 And then

00:58:02 sometimes it's

00:58:03 about optimizer,

00:58:04 right?

00:58:04 the objective

00:58:05 function and

00:58:06 optimization

00:58:07 algorithms,

00:58:08 okay?

00:58:10 And this

00:58:11 was where I

00:58:12 downloaded before

00:58:13 but I think the

00:58:13 link might have

00:58:14 been changed

00:58:15 and maybe the

00:58:16 slides were even

00:58:16 taken away.

00:58:18 So let's see

00:58:19 whether that

00:58:20 still works.

00:58:21 You may not be

00:58:22 able to directly

00:58:22 click it because

00:58:23 sometimes this

00:58:24 hyphen,

00:58:25 when converted

00:58:27 into the

00:58:28 HTTP URL link,

00:58:31 it gets into a

00:58:33 different format

00:58:35 or something.

00:58:35 So it becomes

00:58:37 a different

00:58:37 ASCII character.

00:58:39 So if you click

00:58:40 it directly,

00:58:40 you may not get

00:58:41 it, okay?

00:58:42 But usually if you

00:58:43 change that weird

00:58:45 ASCII character

00:58:46 into a hyphen,

00:58:47 you should be able

00:58:47 to get it, okay?

00:58:48 So this is about

00:58:54 the diagnosis,

00:58:56 okay?

00:58:56 Another thing I

00:58:57 want to say is

00:58:58 two kind of analysis

00:58:59 of your models,

00:59:00 okay?

00:59:00 Let's say you have

00:59:02 your model and you

00:59:03 want to understand

00:59:04 your model behavior

00:59:05 and you want to

00:59:07 improve your model

00:59:08 behavior.

00:59:09 So these are two

00:59:12 things that we

00:59:13 try to figure out.

00:59:14 So what you can

00:59:15 do is two

00:59:16 directions, right?

00:59:18 You can start

00:59:19 with your model

00:59:21 and look at

00:59:22 different components

00:59:24 and replace

00:59:26 each component

00:59:27 with the most

00:59:29 ideal scenario

00:59:30 and see whether

00:59:32 that helps

00:59:34 a lot or not,

00:59:36 right?

00:59:36 Then you figure

00:59:36 out, oh, maybe

00:59:37 I need to pay

00:59:38 more attention

00:59:39 to this component.

00:59:40 So that's called

00:59:41 the error analysis,

00:59:42 okay?

00:59:43 The error analysis

00:59:44 is about, you know,

00:59:46 what if I change

00:59:47 each component

00:59:48 to much better

00:59:50 scenario?

00:59:50 Does that improve

00:59:51 the performance

00:59:52 a lot or not, okay?

00:59:53 If that is the case,

00:59:54 then I'm going to

00:59:55 spend more time

00:59:55 on this component,

00:59:56 okay?

00:59:58 Another kind of

00:59:58 analysis is what

01:00:00 you often do

01:00:00 called ablation study.

01:00:02 So in the case

01:00:02 of the ablation study,

01:00:04 right, you start

01:00:06 with, again,

01:00:07 with your baseline

01:00:08 model, right,

01:00:10 with different

01:00:10 components,

01:00:11 and you are saying

01:00:12 that, well,

01:00:13 let me take away

01:00:15 the good contributions

01:00:16 I made to each

01:00:17 component and see

01:00:18 how much worse

01:00:20 the performance

01:00:21 becomes, okay?

01:00:22 In that way,

01:00:23 I can also

01:00:24 delineate, right,

01:00:26 or isolate the

01:00:27 contributions

01:00:28 from each individual

01:00:30 components.

01:00:31 So both are important,

01:00:32 right?

01:00:32 One is about

01:00:33 how to move forward.

01:00:35 One is how to

01:00:37 explain that

01:00:37 what you introduced

01:00:38 actually worked,

01:00:40 okay?

01:00:40 But they are just

01:00:41 opposite directions

01:00:42 in some sense,

01:00:43 okay?

01:00:44 So now let's see.

01:00:45 You have a machine

01:00:46 learning model

01:00:47 with a pipeline,

01:00:48 right?

01:00:48 So let's say

01:00:49 this is a face

01:00:51 recognition, right?

01:00:52 It's a much easier

01:00:53 problem nowadays.

01:00:55 There's a camera image.

01:00:56 You have to

01:00:57 preprocess it,

01:00:58 like removing background,

01:00:59 and then you have

01:01:00 to detect the face,

01:01:02 and then you need

01:01:03 to do segmentation,

01:01:04 figuring out

01:01:04 which pixels

01:01:05 correspond for eyes,

01:01:06 nose, mouth,

01:01:07 and then you may

01:01:09 end up having,

01:01:10 using this as features

01:01:11 to a machine

01:01:12 learning model

01:01:13 and then predict

01:01:14 the label, okay?

01:01:15 So this is a very

01:01:16 easy case.

01:01:18 Now, you see that

01:01:19 this is actually

01:01:20 a very sophisticated

01:01:21 machine learning

01:01:22 pipeline already,

01:01:24 and that's often

01:01:25 the case, right?

01:01:26 So let's see.

01:01:28 The error analysis

01:01:30 basically asks

01:01:31 how much error

01:01:32 is attributable

01:01:34 to each of the components,

01:01:37 okay?

01:01:37 So it's basically asking,

01:01:39 you know,

01:01:39 how much error

01:01:40 is attributable

01:01:41 to each component?

01:01:42 So in this case,

01:01:44 as I said,

01:01:45 for each component,

01:01:47 you can plug

01:01:48 in the ground truth

01:01:49 because the most

01:01:51 ideal component

01:01:53 here is to give you

01:01:55 the exact ground truth,

01:01:57 right?

01:01:57 For instance,

01:01:58 it's giving me

01:01:59 the face detection

01:02:00 perfectly, right?

01:02:02 In reality,

01:02:03 your preprocessing

01:02:05 may not give you

01:02:07 the ideal input

01:02:08 or your face detection

01:02:09 may not give you

01:02:10 the perfect face detection,

01:02:11 okay?

01:02:12 But let's say

01:02:13 that I replace

01:02:14 each component

01:02:16 with the ground truth,

01:02:18 okay?

01:02:19 If that's the best

01:02:20 you can do,

01:02:21 how does the accuracy

01:02:21 change this, okay?

01:02:23 For instance,

01:02:24 in this case,

01:02:25 the overall system

01:02:27 is 85%,

01:02:28 if I do the preprocessing

01:02:31 perfectly,

01:02:33 like expert,

01:02:34 just manually remove

01:02:35 the background

01:02:36 for instance,

01:02:37 you see the accuracy

01:02:38 only increased

01:02:38 by 0.1%,

01:02:41 okay?

01:02:41 That means,

01:02:42 all right,

01:02:43 stop there.

01:02:43 You don't need to spend

01:02:44 more time on background

01:02:46 preprocessing anymore,

01:02:48 okay?

01:02:49 But if it's a face detection,

01:02:50 let's say,

01:02:51 you know,

01:02:51 I always detect a face

01:02:53 successfully

01:02:54 without any error

01:02:56 and then pass

01:02:57 to the downstream components,

01:02:59 then you see

01:03:00 the accuracy increase

01:03:01 to 91%.

01:03:02 Great.

01:03:02 That's something

01:03:03 I can improve,

01:03:04 right?

01:03:05 And then let's say

01:03:07 starting from

01:03:08 whatever the

01:03:09 imperfect face detection,

01:03:11 right?

01:03:14 Oh, sorry.

01:03:14 I think in this case

01:03:16 it's probably,

01:03:17 let's see,

01:03:18 is it cumulative?

01:03:20 The numbers look increasing,

01:03:21 but I think he,

01:03:25 I think we are plugging

01:03:26 in this individually,

01:03:28 but do you think

01:03:30 that in this case,

01:03:32 in this case,

01:03:33 eyes segmentation,

01:03:35 we definitely plug in

01:03:36 the ground truth

01:03:37 for the eyes segmentation,

01:03:38 but do you think

01:03:39 we keep the ground truth

01:03:41 for the face detection

01:03:42 from the early stage

01:03:43 or not?

01:03:44 Based on the numbers,

01:03:48 it probably is the case.

01:03:50 I cannot tell for sure,

01:03:51 okay?

01:03:52 But it makes sense

01:03:53 to say that

01:03:54 let me incrementally

01:03:55 add things one by one,

01:03:56 but usually I do have

01:03:59 these individual perturbations

01:04:01 like the marginal perturbations.

01:04:02 I do perturb each one

01:04:04 separately

01:04:06 without incrementally,

01:04:07 okay?

01:04:08 You could have another lineage

01:04:09 of incremental,

01:04:12 like the ablation study,

01:04:13 you could, you know,

01:04:14 incrementally reduce

01:04:15 more and more components,

01:04:16 but usually you will also

01:04:18 leave some other combinations,

01:04:20 right?

01:04:21 Like I just remove

01:04:22 any one of these components,

01:04:25 right?

01:04:25 And then starting

01:04:25 from one of them,

01:04:26 I incrementally remove

01:04:28 more and more.

01:04:29 So in this case,

01:04:30 we see that eyes segmentation,

01:04:32 nose segmentation,

01:04:34 and mouth segmentation,

01:04:35 let's assume that

01:04:36 these are just

01:04:37 plugging in the

01:04:38 corresponding ground truth

01:04:40 alone without changing

01:04:41 the other parts,

01:04:42 okay?

01:04:43 So it looks like

01:04:44 if that's the case,

01:04:45 the most room

01:04:46 for improvement

01:04:47 is in face detection

01:04:49 and eyes segmentation.

01:04:52 If that's the conclusion,

01:04:54 then I think they are

01:04:55 doing this incrementally,

01:04:57 right?

01:04:57 If that's the conclusion.

01:04:58 Because they figured out,

01:05:00 oh, after adding

01:05:01 the face detection

01:05:02 perfectly,

01:05:03 I jumped,

01:05:04 increased 6%,

01:05:05 and then another jump,

01:05:07 4% is,

01:05:08 on top of the

01:05:09 perfect face detection,

01:05:10 I add the

01:05:11 perfect eyes segmentation.

01:05:13 So this one is incremental.

01:05:15 Okay?

01:05:15 Is this clear?

01:05:16 So that means what?

01:05:18 You want to spend

01:05:18 more time

01:05:19 on face detection

01:05:21 and eyes segmentation

01:05:23 and improve those components,

01:05:25 right?

01:05:26 You can save your efforts

01:05:27 and maximize your outcome.

01:05:30 So this is what

01:05:31 error analysis does

01:05:32 if you want to

01:05:33 improve your method.

01:05:34 Ablation analysis

01:05:37 is trying to explain

01:05:43 why your current model

01:05:44 is behaving like that.

01:05:47 Okay?

01:05:47 So error analysis

01:05:48 is explaining

01:05:49 the difference

01:05:49 between the current performance

01:05:51 and the perfect performance,

01:05:53 right?

01:05:53 What if I replace

01:05:54 the individual components

01:05:56 by the perfect ground truth?

01:05:58 Ablation analysis

01:05:59 is trying to explain

01:05:59 the difference

01:06:00 between your current performance

01:06:03 and some baseline

01:06:04 when you remove

01:06:06 some good properties

01:06:09 of that component,

01:06:11 right?

01:06:13 So,

01:06:14 for instance,

01:06:16 if you build

01:06:16 anti-spam,

01:06:17 right,

01:06:18 on top of this

01:06:19 logistical regression,

01:06:21 you also have

01:06:22 these clever contributions,

01:06:23 right?

01:06:23 You say that,

01:06:25 well,

01:06:25 these are my major contributions,

01:06:26 right?

01:06:27 In the end,

01:06:27 when I have

01:06:28 all these contributions

01:06:29 together,

01:06:30 I did improve

01:06:31 compared to Steve-Art,

01:06:33 by 10%,

01:06:34 right?

01:06:35 But then the question

01:06:36 is,

01:06:37 you know,

01:06:38 how much

01:06:39 did each of these components

01:06:41 actually help,

01:06:42 right?

01:06:43 So,

01:06:43 in other words,

01:06:44 you want to remove

01:06:45 some feature

01:06:46 in quotation,

01:06:47 I don't mean

01:06:47 the input feature,

01:06:48 I mean the properties

01:06:49 of the algorithm

01:06:50 and see the impact

01:06:52 to the performance,

01:06:53 right?

01:06:54 So,

01:06:54 in this case,

01:06:55 it's probably

01:06:56 incrementally

01:06:58 removing

01:06:59 the components,

01:07:00 right?

01:07:01 Right?

01:07:01 The reason I think

01:07:04 incremental

01:07:05 adding or removing

01:07:07 is tricky

01:07:08 is because

01:07:09 the order may not

01:07:10 be that clear.

01:07:11 So,

01:07:11 sometimes,

01:07:12 when we have

01:07:13 a few contributions

01:07:15 or components,

01:07:15 we usually have to combine

01:07:17 the incremental

01:07:18 and the individual,

01:07:19 you know,

01:07:20 any one or any two

01:07:21 sometimes even.

01:07:23 So,

01:07:23 in this case,

01:07:24 let's say the accuracy

01:07:25 is already 99.9%.

01:07:27 if we remove

01:07:29 our contribution

01:07:30 to the spelling correction,

01:07:32 right?

01:07:32 So,

01:07:33 the spelling correction.

01:07:34 If we don't do

01:07:35 spelling correction,

01:07:36 the accuracy

01:07:37 is 99.0.

01:07:39 If we don't include

01:07:41 the features

01:07:42 from the sender host,

01:07:43 the accuracy drops

01:07:44 to 98.9,

01:07:45 but,

01:07:46 you know,

01:07:46 you keep dropping

01:07:47 the major contributions

01:07:49 that you introduced

01:07:51 and that separates you

01:07:53 from the baseline

01:07:54 or state of art,

01:07:54 then you see that

01:07:56 eventually drops

01:07:57 to 94.0,

01:07:58 okay?

01:07:59 This is also

01:08:00 a good idea

01:08:01 sometimes

01:08:01 when your

01:08:02 implementation

01:08:04 is different

01:08:06 from the baseline,

01:08:07 right?

01:08:08 Sometimes,

01:08:08 you had to use

01:08:09 your own implementation.

01:08:10 You cannot change

01:08:11 the baseline repo

01:08:12 for your own contribution,

01:08:14 right?

01:08:14 So,

01:08:15 in that case,

01:08:16 you definitely want

01:08:17 to do the ablation study

01:08:18 all the way to see here

01:08:19 whether the performance

01:08:20 agree with the published

01:08:21 baseline performance

01:08:22 because it's possible

01:08:23 that your implementation

01:08:24 has some faulty issue

01:08:26 and you realize

01:08:27 that I'm always

01:08:28 like 80% here

01:08:29 compared to the baseline,

01:08:31 right?

01:08:31 Which is 82%.

01:08:33 What was wrong?

01:08:34 Then you keep

01:08:35 ablating your features

01:08:36 and realize that,

01:08:37 oh,

01:08:38 if I remove

01:08:38 all my contributions

01:08:39 and my accuracy

01:08:40 is only 70%,

01:08:41 right?

01:08:42 Then why is my

01:08:43 implementation

01:08:44 of the baseline

01:08:44 so different

01:08:45 from the reported baseline?

01:08:47 There must be

01:08:47 some implementation

01:08:48 issue as well.

01:08:49 So,

01:08:49 that's another reason

01:08:50 I like this kind

01:08:51 of ablation study,

01:08:52 right?

01:08:52 Because you think

01:08:53 conceptually,

01:08:54 if I,

01:08:55 you know,

01:08:55 make these contributions

01:08:56 they should help,

01:08:58 right?

01:08:58 But if you remove

01:08:59 all of them

01:09:00 you cannot even

01:09:00 replicate the baseline

01:09:01 then that means

01:09:02 some implementation

01:09:03 issues are there,

01:09:05 okay?

01:09:07 And in this case

01:09:09 we can conclude

01:09:10 where the biggest

01:09:11 job was

01:09:12 here,

01:09:15 98.9% to 95%,

01:09:17 right?

01:09:18 So it looks like

01:09:19 the email

01:09:19 text parser features

01:09:20 are most important

01:09:22 similarly,

01:09:22 right?

01:09:23 So,

01:09:24 and I think

01:09:25 now it's almost

01:09:26 the standard

01:09:26 that you need

01:09:28 to include

01:09:28 ablation study

01:09:29 in your paper

01:09:32 to show

01:09:32 that you understand

01:09:33 or you have

01:09:35 a proof

01:09:36 that your major

01:09:37 contributions

01:09:38 actually had

01:09:39 an impact

01:09:40 of your performance

01:09:42 improvement

01:09:43 against the state of the art,

01:09:45 okay?

01:09:46 So these are

01:09:47 the two kinds

01:09:47 of analysis.

01:09:48 Again,

01:09:48 error analysis

01:09:49 and ablation study,

01:09:50 okay?

01:09:51 Again,

01:09:51 during the development

01:09:52 process you may want

01:09:54 to do some error analysis

01:09:55 trying to see

01:09:56 how you can improve

01:09:58 but once you figure it out

01:09:59 you can do the ablation study,

01:10:01 right?

01:10:01 Usually in our research

01:10:03 what we do

01:10:04 is the reverse

01:10:05 ablation study.

01:10:06 We start with a baseline

01:10:07 and we think

01:10:08 that we have several ideas

01:10:09 and then we

01:10:10 build upon this idea,

01:10:13 build upon this

01:10:14 baseline

01:10:15 and add these ideas

01:10:16 one by one.