Lecture 15: Machine Learning in Practice - Diagnostics and Fixes
Recording. Download the text transcript.
Automatic transcript; technical terms and equations may contain recognition errors.
00:00:00 Okay, so let me continue. And as I said, we have talked about machine learning in concepts. We have talked about machine learning in theories. And one of the most important theory to me is a bias variance tradeoff, right? Which says that we know machine learning is not about fitting observed data points. It's about making predictions for unobserved data points.
00:00:28 So in that sense, we could look at the expected error, right? So the expected error on the test data. But where is that expectation coming from? The expectation is based on the fact that our training data, the observed data, could be sampled from a distribution, right? So we draw samples from that distribution.
00:00:52 So if we calculate the expected value of the generalization error, okay? The error is mean squared error. If you're doing regression, it's the generalization error because the error was evaluated on the test set, right?
00:01:09 And it's expected generalization error because that was expected or averaged over the distribution of the training set, right?
00:01:18 So we derived that. Again, I recommend you to look at the derivation and try to do that by yourself. It should be straightforward.
00:01:31 That you will see that there are three terms. One is squared bias and one is the variance and one is uncertainty in the measurement that we cannot control.
00:01:42 So if you want to reduce the generalization error, we can try to reduce the sum of the first two terms, right? But in the meantime, it's just too idealistic and actually impossible to reduce both simultaneously.
00:02:03 I asked you to look at the theory behind it. Have you taken a look at it?
00:02:07 Have you?
00:02:08 Yeah, the Kramer-Rey inequality, right? Which is relating the variance, right, to the information, right?
00:02:20 So I strongly recommend you to look into that. That actually tells you why there is a trade-off, right?
00:02:26 So then, in reality, we can make our models significantly reduce the variance part while sacrificing the bias a little bit, right?
00:02:39 How do we do that? Well, if you make the model a little bit regularized in its complexity, right?
00:02:47 Then the bias, you know, the complexity decreases, the bias increases, but hopefully the variance decreases much faster, right?
00:02:54 And we also introduced how do you quantify the model complexity, right?
00:03:00 Which is not only determined by the model, but actually also dependent on determined by the data, right?
00:03:07 So we used the what?
00:03:09 The...
00:03:09 What method did we use to quantify?
00:03:15 The shrinkage method, the LP norm, right? So that can be used as a penalty in your loss function, so that you can simultaneously do the minimization of the loss to the observed data and reducing the model complexity simultaneously, right?
00:03:42 So those are the theoretical concepts that are very important.
00:03:47 And we're actually going to connect to them again, right?
00:03:51 But today we're going to focus on, you know, practical machine learning.
00:03:55 Again, just like what I said in the case of the optimization,
00:04:00 actually you want to have a theoretical guideline when you are developing or even choosing optimization algorithms, right?
00:04:10 You actually have seen that when the LP norm is L2, it's actually a very simple optimization problem if the loss is mean squared error.
00:04:18 You actually have an analytical expression of the mean squared error plus L2 norm, right?
00:04:25 If it's L1 norm, well, it's a little bit more complicated.
00:04:31 It's still convex, but there's no smooth gradient anymore, right?
00:04:35 And then in that case, there are still algorithms, right?
00:04:40 Just not gradient-based algorithms.
00:04:42 And then if the P is between 0 and 1, then it's even more difficult.
00:04:47 It's non-convex, right?
00:04:49 But if P is less than or equal to 1 in general, you have the benefit of having the sparse solution.
00:04:54 So you actually have to decide, right, a lot of things depending on your goals, right?
00:05:01 How you choose the loss function and how do you do the regularization, okay?
00:05:06 So we'll connect with those a little bit, but today, you know, we would like to talk about machine learning in practice, okay?
00:05:14 This is largely based on the great slides, advice for applying machine learning from Android, okay?
00:05:24 I think it was used to shared somewhere.
00:05:27 I'll probably change the link in somewhere, but I thought this is very useful, so I'd like to talk about it a little bit, okay?
00:05:36 So I'm going to start with, you know, whether it's machine learning or deep learning in future, I think these are very useful.
00:05:44 So I'm going to start with a classification machine learning model, right, that I haven't talked about, but hear me out so that, you know, you can get it, right?
00:05:58 So previously, you remember that we use, for instance, a linear classifier, right?
00:06:06 W transpose times X.
00:06:08 If X also include a feature 1, then that corresponding W is W naught, right?
00:06:16 So then there is a linear decision boundary, right?
00:06:21 And then we have a linear classifier, right?
00:06:25 Okay, so that's what we have seen before, okay?
00:06:29 So in the case of the logistic regression compared to the linear regression, sorry, not linear regression, linear classifier, okay, sorry.
00:06:37 So in the case of the logistic regression, we still have a linear decision boundary, but that linear decision boundary is not about the score of each sample, okay?
00:06:53 So previously, we have a score of each sample, W transpose times X, okay?
00:06:58 But here, I still have W transpose times X, but it's not about the score, okay?
00:07:04 It's about the log odds, okay?
00:07:09 So this is a log of the odds.
00:07:11 What is an odd?
00:07:12 Odd is the probability of Y equal to 1, given X, divided by probability of Y equal to 0, given X.
00:07:18 So log odds, okay?
00:07:21 So the log odds is approximated by a linear decision boundary, equal to 0, okay?
00:07:28 And then if you know that the, you know, numerator and denominator add up to 1, right?
00:07:36 Because that's one of the three principles or three laws of probability, right?
00:07:42 Normalization.
00:07:44 Then you can calculate P of Y equal to 1, given X, the conditional probability that a sample X belonged to class 1, right?
00:07:54 And then you will find that it's a logistic function of this form, okay?
00:08:00 So sometimes people also write it as sigma of this function, of this linear function, right?
00:08:08 Right?
00:08:09 So as a reminder, the logistic function is like sigma of X equal to 1 over 1 plus E2 minus X, okay?
00:08:17 So you replace the X by this linear function in the logistic.
00:08:22 So that's the name logistic regression is coming from, okay?
00:08:26 So you still have a linear function, but it's no longer the decision boundary in the score space, but it's a decision boundary in the log odds space, okay?
00:08:41 So if you rewrite the probability, then you will find that it's actually a logistic function.
00:08:48 So in that sense, it's called a logistic regression, okay?
00:08:52 So if you rewrite the log odds, you are regressing the log odds, and the probabilities are a logistic function of a linear function, okay?
00:09:01 Of course, you can generalize it to nonlinear cases and so on, okay?
00:09:04 If we look at a logistic function just to have intuition, right?
00:09:11 So if you have sigma of X, that's a blue curve, right?
00:09:14 So when X equal to 0, you should reach 1 over 2, right?
00:09:19 So 0.5, like this point, right?
00:09:21 When X is 0, you should reach 0.5.
00:09:24 So if you replace X by 2X, then you have this cyan curve, which still pass 0.5, but it's getting like steeper transitions, right?
00:09:39 This S shape is having steeper transitions.
00:09:42 If you replace this X by X minus 2, then this is like what you learned in the signals and systems, right?
00:09:52 It's like what?
00:09:55 It's just a, you know, affine transformation on time, right?
00:09:59 So this is what?
00:10:02 This is minus 2.
00:10:05 So this is moving to the left or to the right.
00:10:09 Sigma of X minus 2 relative to sigma of X is moving to the right.
00:10:13 So it's advanced, right?
00:10:16 It's time advanced, right?
00:10:17 If you think about X as time.
00:10:19 Then if you have the combination, which is affine transformation of X, which is linear transformation,
00:10:25 then in this case, you have this red one.
00:10:28 So it's going to be shifted to left or right by how much?
00:10:32 0.5X plus 1.
00:10:41 So 0.5X compared to 2X, it's expanded, right?
00:10:52 So the signal is expanded, right?
00:10:54 So that's why compared to the blue, it's like stretch to the left and the right.
00:10:59 It's expanded because 0.5X plus 2.
00:11:01 And then it's expanded and then also shifted to the left, right?
00:11:08 By two units, right?
00:11:09 Depending on whether you shift or scale first, right?
00:11:12 But, you know, if you follow one convention, you're moving left by two units, okay?
00:11:17 It's 0.5 times X plus 2, right?
00:11:21 So these are the decision boundary.
00:11:24 In this way, you can see that depending on your learned W, right, you have the different logistic function, okay?
00:11:33 Okay, now let's rewrite what we had here before but put a subscript, I, as the index of the training example, okay?
00:11:44 And we also have a semicolon here to indicate what's behind the semicolon is a parameter, okay?
00:11:55 So it's a parameter.
00:11:56 So in this case, we are asking what's the conditional probability of example I being in class 1 or 0,
00:12:05 given the features XI and, of course, the parameter W, okay?
00:12:13 So based on our defined decision boundary, we can find that it's just a sigma function of this linear function, okay?
00:12:22 It's logistic in this linear function, all right?
00:12:26 Okay, so then, and then in this case, now we have to ask ourselves, right?
00:12:36 So if you have defined a new classifier, right?
00:12:39 In the past, I understand that you used this linear classifier in the score space,
00:12:45 and your goal was to intuitively maximize the margin, right?
00:12:52 And we derived, well, I didn't fully derive it, but I hope you have tried it, right?
00:12:57 I gave the hints.
00:12:58 We derived that the margin was proportional to 1 over the norm of W, right?
00:13:05 And then we can equivalently minimize the norm squared, L2 norm squared of W, right?
00:13:13 But in this case, if you want to find the best W,
00:13:17 given a set of examples I from 1 to N,
00:13:21 what would be your loss function?
00:13:26 What is an appropriate loss function?
00:13:41 What is an appropriate loss function here?
00:13:43 You are trying to adjust W for the given N training examples of paired XI, YI,
00:13:59 where I goes from 1 to N, so that some loss is optimized.
00:14:07 What is that loss in this case?
00:14:08 Right?
00:14:13 See, we are looking at the probabilities now, okay?
00:14:18 Yes.
00:14:22 What is an appropriate loss function here?
00:14:26 You must know.
00:14:26 Don't be shy.
00:14:27 What I wrote there?
00:14:34 Actually, it's one step before that, actually.
00:14:39 What I wrote here was already following some assumptions.
00:14:44 Okay?
00:14:45 So what is the probability that you are trying to optimize here?
00:14:49 The probability of all the Y, Y1 through YI through YN,
00:15:06 given all the XI, I from 1 to N, semicolon W.
00:15:12 Is that right?
00:15:12 The probability of observing all the Y, given all the X and W, right?
00:15:21 That is the principle.
00:15:23 Is that right?
00:15:24 Now, if I assume these training examples are IID,
00:15:31 identically and independently distributed,
00:15:34 then I can factorize that conditional probability
00:15:38 into the product of all these P's, right?
00:15:46 Am I right?
00:15:47 Don't know what I touched, but it looks very...
00:15:58 Do I agree?
00:15:59 So then the probability of the vector Y,
00:16:04 given the vector X and W,
00:16:06 where by vector I mean Y equal to Y1 through Y2
00:16:10 through YI through YN,
00:16:12 and X vector is X1 through XN, right?
00:16:16 So that conditional probability can be now factorized
00:16:20 as a product of all these P of YI given XI and W, right?
00:16:25 Then you have the...
00:16:28 Then you can, of course, maximize the product of the P, right?
00:16:32 Or you can maximize the log of the product of the P, right?
00:16:38 Which is equivalent.
00:16:39 And then if you maximize the log of the product,
00:16:42 then you are maximizing the sum of the log probability, right?
00:16:47 So you have to start with the principle.
00:16:49 So I know I put the final step there,
00:16:53 but that's not where things started, right?
00:16:56 Okay.
00:16:57 If we agree on that,
00:16:59 then we can focus on what's behind the summation
00:17:04 because for every example I,
00:17:06 what's behind is the same, right?
00:17:09 Okay.
00:17:11 Then how do we write this log of P, right?
00:17:14 So this is something just like the...
00:17:20 What we did in the...
00:17:24 Support vector machine, remember?
00:17:27 We had two kinds of constraints, right?
00:17:30 Whether you're blue or red points,
00:17:32 depending on which side of the decision boundary you are, right?
00:17:35 Your score may be bigger than or less than that threshold, right?
00:17:40 But we united them together into one constraint using yi, right?
00:17:45 So here it's the same.
00:17:46 So we are using yi here.
00:17:48 So if yi is equal to one,
00:17:50 it's log of this probability, right?
00:17:54 If yi is zero,
00:17:56 then only...
00:17:57 If yi is one,
00:17:58 then only the first term is non-zero,
00:18:02 and the second term is going to be zero,
00:18:03 and the first term is nothing but the probability of yi equal to one,
00:18:09 given xi and w, right here.
00:18:12 If y equal to zero,
00:18:14 then only the second term is non-zero,
00:18:17 and what we have here is exactly log of the probability of yi equal to zero,
00:18:24 given xi and w.
00:18:25 So that's what we have here.
00:18:27 Okay?
00:18:27 Okay?
00:18:28 So that's what we have here.
00:18:31 Okay?
00:18:31 So this is...
00:18:33 So what's the relationship between what I write right here
00:18:38 and binary cross-entropy?
00:18:45 You must be using my binary cross-entropy somewhere, right?
00:18:49 So what's the relationship between what I wrote here
00:18:52 and the binary cross-entropy?
00:18:56 Can someone try?
00:19:01 How many of you have used binary cross-entropy?
00:19:05 You must have used it.
00:19:08 No?
00:19:11 Binary cross-entropy?
00:19:13 If you use classification,
00:19:15 if you do any classification,
00:19:17 you must use it, right?
00:19:19 Okay, so I recommend you to look into that
00:19:21 and see what's the relationship with this one
00:19:24 and gain a better understanding.
00:19:26 Okay?
00:19:27 So again,
00:19:28 this is just a background.
00:19:30 Okay?
00:19:30 So this is a baseline model, right,
00:19:34 that I start with.
00:19:35 So my question is,
00:19:36 how do I diagnose?
00:19:37 Okay?
00:19:39 All right.
00:19:39 So I have a model like this, right?
00:19:42 So...
00:19:42 And I apply this model to,
00:19:45 as you know,
00:19:46 one of the early application
00:19:48 of machine learning.
00:19:51 Okay?
00:19:51 Not deep learning.
00:19:52 Machine learning.
00:19:53 That is spam detection.
00:19:55 Okay?
00:19:55 Nowadays,
00:19:56 most of the email,
00:19:58 probably actually all of the email
00:19:59 providers also give you
00:20:02 this spam detection, right,
00:20:05 automatically.
00:20:06 Sometimes it does make mistakes, right?
00:20:09 By the way,
00:20:11 which one annoys you more?
00:20:14 False positives or false negatives?
00:20:16 If I have a spam detector,
00:20:21 it's a classifier,
00:20:22 machine learning classifier.
00:20:24 Which one annoys you more?
00:20:26 Because that matters
00:20:27 in how I train my classifier, right?
00:20:30 Are you annoyed more
00:20:31 by the false positives?
00:20:35 Yes.
00:20:36 Or false negatives?
00:20:39 You are more annoyed
00:20:40 by the false positives.
00:20:41 You are more worried
00:20:43 about missing important emails
00:20:45 than get bothered
00:20:48 by probably five more
00:20:50 just spam emails
00:20:54 that just got escaped, right?
00:20:57 Probably.
00:20:59 Okay, so then we may have
00:21:00 to think about
00:21:01 what is the last function then, okay?
00:21:03 So let's say that
00:21:04 this is a very simple case
00:21:07 in the early days, right?
00:21:08 Right?
00:21:09 That you have
00:21:10 a small set of 100 words
00:21:12 to use as features, okay?
00:21:16 And in this case,
00:21:18 we use Bayesian logistic regression, okay?
00:21:23 And we are using gradient descent
00:21:27 to optimize this loss function, right?
00:21:30 And we found a 20% test error,
00:21:34 which is unacceptably high, okay?
00:21:36 Again, so in our case,
00:21:37 we are maximizing the minus loss,
00:21:41 which is minimizing the loss, right?
00:21:43 This is what you have seen before.
00:21:45 And also,
00:21:46 we are regularizing
00:21:48 this model complexity, right?
00:21:49 Here, this model parameter
00:21:51 is theta, right?
00:21:52 Okay, and I have
00:21:53 M training examples, okay?
00:21:56 So,
00:21:57 and also,
00:22:00 just,
00:22:01 you may wonder,
00:22:02 well,
00:22:03 why did I have to use
00:22:05 gradient descent?
00:22:06 There is a unique global optimum
00:22:08 for the maximum likelihood prediction
00:22:10 under the logistic regression,
00:22:12 but
00:22:12 this is the same like,
00:22:14 you know,
00:22:16 mean squared error
00:22:17 with LASO,
00:22:19 L1 penalty, right?
00:22:20 There is a unique global optimum,
00:22:23 but
00:22:23 in that case,
00:22:25 there is no,
00:22:26 you know,
00:22:26 gradient method
00:22:28 that
00:22:28 optimizer that works
00:22:30 because of the smoothness issue.
00:22:32 In this case,
00:22:34 you know,
00:22:35 there is also
00:22:36 no closed form solution,
00:22:38 but luckily,
00:22:38 you can still use
00:22:39 a gradient descent
00:22:40 to solve the problem,
00:22:41 okay?
00:22:43 Okay,
00:22:43 so we sometimes
00:22:44 use the W before,
00:22:46 right here,
00:22:46 we are using theta here,
00:22:48 so please don't be confused,
00:22:49 okay?
00:22:50 All right,
00:22:51 so what do you do next,
00:22:52 right?
00:22:52 So,
00:22:53 that is possible,
00:22:54 right?
00:22:54 Even nowadays,
00:22:56 right,
00:22:57 when we are using
00:22:57 a lot of fancier method,
00:22:59 whether it's some kind
00:23:00 of deep learning,
00:23:01 right?
00:23:01 The only difference
00:23:02 is now your data
00:23:03 is bigger,
00:23:05 and the cost
00:23:06 of evaluating
00:23:08 the model performance
00:23:10 as well as
00:23:13 the cost
00:23:14 to improve
00:23:15 the model performance
00:23:16 are also increasing,
00:23:18 actually.
00:23:19 So that just makes
00:23:20 diagnosis
00:23:20 even more important,
00:23:22 right?
00:23:22 So let's use this example
00:23:24 to see how we diagnose,
00:23:25 right?
00:23:27 Okay,
00:23:27 so let's see.
00:23:30 What do you do?
00:23:30 How do you diagnose,
00:23:31 right?
00:23:32 So let's ask
00:23:33 a similar question,
00:23:34 right?
00:23:35 So if you,
00:23:36 you know,
00:23:37 found some problems,
00:23:39 right,
00:23:39 some random noise
00:23:40 or some issues
00:23:42 of your car,
00:23:43 right?
00:23:44 Right?
00:23:44 What do you do,
00:23:46 right?
00:23:46 So,
00:23:47 there might be
00:23:49 different scenario,
00:23:51 but usually you send
00:23:52 it to a checkup,
00:23:53 right,
00:23:53 a diagnosis.
00:23:54 What you don't do,
00:23:57 what you never do
00:23:58 is try to replace
00:24:00 every part
00:24:00 one by one
00:24:02 and eventually see
00:24:03 whether it works or not.
00:24:05 You never do that,
00:24:06 right?
00:24:06 Because the cost
00:24:07 is
00:24:08 unreasonable,
00:24:11 but more importantly,
00:24:12 the practice
00:24:12 is not reasonable,
00:24:13 right?
00:24:14 But the funny thing
00:24:15 is that if you talk
00:24:16 about our research,
00:24:17 especially,
00:24:18 you know,
00:24:19 diagnosing,
00:24:20 sorry,
00:24:20 I shouldn't use
00:24:21 that word,
00:24:22 improving our
00:24:22 machine learning
00:24:23 model's performance,
00:24:26 what we don't realize
00:24:27 is that we are
00:24:28 much more often
00:24:29 doing something like that.
00:24:31 We start with this model,
00:24:33 we see,
00:24:33 oh,
00:24:33 the performance
00:24:34 is not good,
00:24:35 then we try
00:24:35 everything we can.
00:24:37 We try everything
00:24:38 we can
00:24:38 and then hope
00:24:39 that the performance
00:24:40 will increase,
00:24:41 right?
00:24:41 So,
00:24:42 like in this case,
00:24:43 right?
00:24:44 So you can try
00:24:45 using getting
00:24:46 more training examples,
00:24:48 right?
00:24:49 You can use
00:24:50 the same training
00:24:52 example,
00:24:52 but try bootstrapping,
00:24:54 right?
00:24:55 You can use
00:24:56 a smaller set of features,
00:24:57 you can use
00:24:58 a larger set of features,
00:24:59 right?
00:24:59 Or you just change
00:25:00 the features,
00:25:01 right?
00:25:02 For instance,
00:25:02 you feel like
00:25:03 maybe the words
00:25:05 from the email header
00:25:07 is more
00:25:09 implicative,
00:25:10 right?
00:25:11 Or the ones
00:25:12 in the email bodies
00:25:13 can be more
00:25:13 implicative,
00:25:14 right?
00:25:16 And then you think
00:25:17 that,
00:25:17 well,
00:25:17 I'm solving
00:25:18 an optimization problem,
00:25:19 maybe I can run
00:25:20 the gradient descent
00:25:21 for more iterations
00:25:22 so that I can get
00:25:24 a better quality
00:25:25 optimum,
00:25:26 right?
00:25:27 And then you realize
00:25:28 that, oh,
00:25:28 I learned all these
00:25:30 great theories
00:25:31 about the iterative
00:25:32 optimizers,
00:25:33 right?
00:25:33 We learned that
00:25:34 the steepest descent
00:25:35 was simple
00:25:36 but slow,
00:25:37 so maybe I can try
00:25:39 Newton's method.
00:25:41 Sorry,
00:25:42 it's simple
00:25:43 but fast,
00:25:44 right?
00:25:45 Well,
00:25:46 no, no,
00:25:47 I take it back.
00:25:47 It's simple
00:25:48 and every step
00:25:50 you just take
00:25:50 small amount
00:25:51 of calculations
00:25:52 but it's going
00:25:53 to take a lot
00:25:54 of steps
00:25:54 because convergence
00:25:56 rate is slow,
00:25:57 right?
00:25:58 And then you figure
00:25:59 that, well,
00:26:00 maybe I can try
00:26:01 Newton's method,
00:26:01 right?
00:26:02 Each step,
00:26:03 iteration,
00:26:03 I spend more time
00:26:05 calculating,
00:26:06 including the second
00:26:07 order,
00:26:07 right?
00:26:08 Information,
00:26:09 but maybe I can
00:26:10 take less steps,
00:26:11 right?
00:26:12 And what else?
00:26:13 I can try
00:26:14 the different
00:26:14 lambda,
00:26:15 right?
00:26:15 I can try
00:26:16 different lambda
00:26:17 or I just use
00:26:18 the support vector
00:26:19 machine instead
00:26:20 of this Bayesian
00:26:23 logistic regression,
00:26:24 okay?
00:26:27 These are all
00:26:28 possible
00:26:28 but the question
00:26:29 is,
00:26:30 just like the case
00:26:31 of the car,
00:26:31 right?
00:26:32 Would you just
00:26:32 try all of them
00:26:34 or just try them
00:26:36 one by one
00:26:37 or what's the order
00:26:38 you're going to take?
00:26:40 Any
00:26:40 suggestions?
00:26:43 These are all
00:26:44 things we can try.
00:26:48 How are you
00:26:48 going to decide?
00:26:59 Apparently,
00:27:00 we can do all
00:27:00 of them.
00:27:00 It's just going
00:27:01 to take much
00:27:02 longer time,
00:27:02 right?
00:27:05 But if you cannot
00:27:06 afford doing all
00:27:07 of them,
00:27:08 or more importantly,
00:27:10 you want to be
00:27:11 rationally satisfied
00:27:12 instead of trying
00:27:13 everything possible,
00:27:15 what would you do?
00:27:25 Just like what
00:27:26 you do with your
00:27:28 cars.
00:27:29 You send it to
00:27:30 a body shop
00:27:33 for a diagnosis
00:27:35 first.
00:27:36 In other words,
00:27:37 you have to figure out
00:27:38 what's the problem
00:27:39 first.
00:27:39 I know you say
00:27:40 that.
00:27:41 Oh, no, no, no.
00:27:41 I know the problem.
00:27:42 The problem is
00:27:43 that the test error
00:27:43 is high.
00:27:44 Yes,
00:27:45 that is the final
00:27:47 ultimate symptom.
00:27:49 But what was
00:27:50 the problem?
00:27:51 Okay,
00:27:51 what's the disease
00:27:52 that caused
00:27:53 the symptom?
00:27:54 That's the question
00:27:54 I'm trying to ask.
00:27:56 You need to figure out
00:27:57 what's the problem.
00:27:58 Okay?
00:28:00 Okay,
00:28:01 so the better
00:28:01 approach is,
00:28:02 as I said,
00:28:03 you try to run
00:28:04 diagnosis to figure out
00:28:05 what the problem is
00:28:06 and then fix
00:28:07 that problem.
00:28:09 Exactly.
00:28:10 Okay?
00:28:11 So,
00:28:12 in this case,
00:28:13 let's say
00:28:14 we start with a problem
00:28:15 that is very,
00:28:16 you know,
00:28:17 superficial,
00:28:18 just the phenomena
00:28:19 that the test error
00:28:21 is 20%,
00:28:22 unacceptably high.
00:28:23 Right?
00:28:24 So,
00:28:25 then you have to start
00:28:26 with some hypothesis.
00:28:27 Right?
00:28:28 So,
00:28:28 one hypothesis
00:28:29 we have learned before
00:28:30 was the overfitting
00:28:33 versus
00:28:33 underfitting.
00:28:35 Right?
00:28:36 We learned before
00:28:37 one perspective
00:28:38 that is
00:28:39 bias-variance trade-off.
00:28:41 Right?
00:28:41 I think you can use that
00:28:42 as one of your ways
00:28:43 to diagnose.
00:28:44 Remember,
00:28:45 there are so many
00:28:45 different ways
00:28:46 of diagnosing
00:28:47 and this is just
00:28:48 one way to diagnose.
00:28:50 Then each diagnosis
00:28:51 is actually following
00:28:52 some theoretical
00:28:53 foundations.
00:28:54 Right?
00:28:54 Here,
00:28:56 you might wonder,
00:28:57 well,
00:28:58 based on what I know
00:28:59 about the
00:28:59 bias-variance trade-off,
00:29:01 in my case,
00:29:03 am I
00:29:03 experiencing
00:29:04 the high-variance issue
00:29:06 that's overfitting
00:29:07 or it's probably
00:29:09 high-biased issue.
00:29:11 Right?
00:29:12 Right?
00:29:13 So,
00:29:13 that's one way.
00:29:14 Right?
00:29:14 So,
00:29:15 then we can
00:29:15 follow this direction
00:29:17 and do diagnosis.
00:29:19 Right?
00:29:19 The diagnosis is
00:29:20 look at the variance
00:29:22 and the bias.
00:29:24 Okay?
00:29:24 So,
00:29:25 the diagnostic,
00:29:26 in the case of the variance,
00:29:27 you're going to compare
00:29:28 this training error
00:29:31 with the validation error.
00:29:33 Right?
00:29:34 In the case of the bias,
00:29:35 you can also look at
00:29:36 your training error
00:29:37 and whether
00:29:39 it itself
00:29:41 is bad or not.
00:29:43 Right?
00:29:43 In the case of the variance,
00:29:44 you can compare the error
00:29:46 for the training set
00:29:47 and the validation set.
00:29:49 Right?
00:29:49 See how this
00:29:50 model behaves differently,
00:29:53 how the predictions
00:29:54 behave differently
00:29:55 for different data sets.
00:29:57 That's what the variance
00:29:57 is about.
00:29:58 Right?
00:29:59 Bias is about
00:30:00 how off the prediction
00:30:01 is compared
00:30:02 to the actual.
00:30:03 Right?
00:30:03 So,
00:30:04 you can look at
00:30:04 the value
00:30:06 of the training error
00:30:07 or the validation error
00:30:07 itself.
00:30:08 Right?
00:30:09 Rather than comparing
00:30:09 the two.
00:30:10 Right?
00:30:13 Okay.
00:30:13 So,
00:30:14 we actually have already
00:30:15 compared,
00:30:16 we have already
00:30:17 discussed this before.
00:30:18 Remember,
00:30:19 in this case,
00:30:20 we have an underlying
00:30:22 grain function
00:30:23 that is not known
00:30:24 but observed
00:30:25 with the noisy
00:30:27 observations,
00:30:29 which are the blue
00:30:30 circles.
00:30:31 Right?
00:30:31 So,
00:30:31 if you use
00:30:32 an overly complex
00:30:35 model,
00:30:36 like 9th order
00:30:37 polynomial,
00:30:38 you can fit
00:30:39 all the blue circles
00:30:41 perfectly,
00:30:42 but it's
00:30:44 of high variance.
00:30:46 Right?
00:30:46 Right?
00:30:48 So,
00:30:48 if imagine
00:30:49 that these blue circles,
00:30:50 any of them
00:30:51 just move a tiny bit,
00:30:53 then your circles would,
00:30:54 I'm sorry,
00:30:54 then your red function
00:30:55 will just change
00:30:57 drastically.
00:30:58 Right?
00:30:58 So,
00:30:58 that's overfitting.
00:30:59 Okay?
00:31:00 So,
00:31:00 now let's look at
00:31:02 an illustration here.
00:31:04 Okay?
00:31:05 So,
00:31:05 this is like
00:31:06 a dart board.
00:31:09 Right?
00:31:10 This is like a dart board.
00:31:11 Right?
00:31:12 So,
00:31:13 what you are trying to do,
00:31:14 right,
00:31:15 you are throwing
00:31:16 the dart
00:31:17 to the dart board,
00:31:19 and let's say,
00:31:20 you know,
00:31:21 this is,
00:31:21 the center,
00:31:22 right,
00:31:23 is perfect.
00:31:25 Right?
00:31:25 It's a perfect
00:31:26 machine learning model,
00:31:28 like this green function.
00:31:30 You're getting
00:31:31 the ground truth.
00:31:32 Okay?
00:31:32 Let's say that's,
00:31:33 that's what it corresponds to.
00:31:35 Okay?
00:31:37 And then,
00:31:38 let's imagine that
00:31:40 as your data changes,
00:31:42 your training data changes,
00:31:43 every time
00:31:44 your trained machine learning model
00:31:47 is gonna give a performance.
00:31:49 Right?
00:31:49 It's like,
00:31:50 you know,
00:31:50 this,
00:31:51 every time you throw the dart
00:31:52 to the board,
00:31:53 right?
00:31:53 So,
00:31:54 it's gonna land somewhere.
00:31:56 Right?
00:31:57 It's gonna land somewhere.
00:31:59 And,
00:32:00 the landing points,
00:32:02 right,
00:32:04 are shown,
00:32:05 if we just compare
00:32:06 among the landing points,
00:32:08 it's like comparing
00:32:09 the model predictions.
00:32:11 Right?
00:32:11 So,
00:32:12 the ideal case
00:32:13 is that
00:32:14 these landing points
00:32:15 are close to each other.
00:32:17 Meaning that
00:32:17 the model predictions
00:32:19 are close to each other.
00:32:20 Low variance.
00:32:21 Right?
00:32:22 But,
00:32:22 I mean,
00:32:22 in the meantime,
00:32:24 they are all close
00:32:25 to the origin.
00:32:26 Right?
00:32:27 Right?
00:32:28 Low bias.
00:32:29 Right?
00:32:29 So,
00:32:29 in other words,
00:32:30 the model predictions
00:32:32 differ from each other
00:32:33 very lowly,
00:32:34 and their error
00:32:36 is very low.
00:32:37 The distance to the origin.
00:32:39 Okay?
00:32:40 So,
00:32:40 this is the idea,
00:32:41 a low bias
00:32:42 and a low variance.
00:32:43 Right?
00:32:44 But,
00:32:44 unfortunately,
00:32:45 we don't
00:32:46 often
00:32:47 see that.
00:32:47 What often happens
00:32:48 is,
00:32:49 like in this case.
00:32:51 Right?
00:32:52 In this case,
00:32:53 if you look at
00:32:54 these model predictions,
00:32:56 right,
00:32:57 they are actually,
00:32:57 they are actually,
00:33:00 you know,
00:33:00 not,
00:33:01 well,
00:33:02 in this case,
00:33:03 at least we can see
00:33:05 that they are far away
00:33:06 from each other.
00:33:06 They are scattered
00:33:07 from each other.
00:33:09 Right?
00:33:09 The model predictions
00:33:09 are very different
00:33:10 from each other
00:33:11 as the data changes.
00:33:13 That means
00:33:13 it's high variance,
00:33:15 isn't it?
00:33:16 But in the meantime,
00:33:17 they are not
00:33:17 too far away
00:33:18 from the origin.
00:33:19 Right?
00:33:19 They are far away
00:33:20 from each other
00:33:21 but not far away
00:33:22 from the ground truth.
00:33:23 And their average
00:33:25 might be even
00:33:25 unbiased,
00:33:26 actually.
00:33:27 So,
00:33:27 in that case,
00:33:28 it's low bias
00:33:29 but high variance.
00:33:30 Right?
00:33:31 Another scenario
00:33:32 is that
00:33:33 these landing points
00:33:34 are all close
00:33:35 to each other,
00:33:36 meaning it's low variance,
00:33:37 but they are all far away
00:33:40 from,
00:33:41 you know,
00:33:43 the ground truth label.
00:33:45 So,
00:33:45 in that case,
00:33:46 it's high bias.
00:33:47 Right?
00:33:48 But of course,
00:33:49 sometimes what's breaking
00:33:50 you is that,
00:33:52 like in this case.
00:33:54 Right?
00:33:54 So,
00:33:55 your model predictions
00:33:56 are far away
00:33:57 from each other,
00:33:58 indicating a high variance.
00:34:00 In the meantime,
00:34:01 they are all far away
00:34:01 from the ground truth,
00:34:03 indicating a high bias
00:34:04 as well.
00:34:05 Right?
00:34:06 So,
00:34:06 this is one kind
00:34:08 of diagnosis
00:34:09 you can,
00:34:10 you can,
00:34:12 one kind of diagnosis
00:34:14 you can perform
00:34:14 that is looking
00:34:16 at whether your problem
00:34:17 is overfitting
00:34:19 or underfitting.
00:34:20 Right?
00:34:20 So,
00:34:21 you know what to do
00:34:22 when there's overfitting.
00:34:24 Right?
00:34:24 You know what to do
00:34:25 if there's overfitting.
00:34:26 What do we do
00:34:27 if there's overfitting?
00:34:29 If it's overfitting,
00:34:30 that means model
00:34:31 is too complex.
00:34:32 So,
00:34:32 we can do what?
00:34:33 We can reduce
00:34:34 the model complexity
00:34:35 by regularization
00:34:36 we can also use
00:34:38 ensemble methods
00:34:39 to reduce the variance.
00:34:41 Right?
00:34:42 So,
00:34:43 right?
00:34:43 So,
00:34:45 that's what we have learned.
00:34:46 But now,
00:34:47 let's make this diagnosis
00:34:49 a little bit more concrete
00:34:51 because this is an illustration.
00:34:53 This is an illustration,
00:34:55 right?
00:34:55 Which is very intuitive.
00:34:56 Right?
00:34:57 But let's make it more concrete.
00:34:59 Right?
00:35:00 Let's,
00:35:00 let's,
00:35:01 you know,
00:35:02 let's make it more concrete.
00:35:03 Okay?
00:35:03 So,
00:35:04 let me first repeat that.
00:35:08 Now,
00:35:08 we're looking at the diagnostics
00:35:10 of machine learning algorithms.
00:35:12 Okay?
00:35:12 So,
00:35:13 don't ask
00:35:13 why isn't my machine learning working
00:35:17 or,
00:35:18 you know,
00:35:18 why is this test error
00:35:20 too that high?
00:35:21 The right question
00:35:22 is,
00:35:23 you know,
00:35:24 what could be the problem
00:35:26 and how could I diagnose?
00:35:27 And if I determined the cause,
00:35:31 how would I change the model?
00:35:32 Right?
00:35:33 Okay.
00:35:34 So,
00:35:35 the one right,
00:35:36 you know,
00:35:37 the correct mentality
00:35:39 is let's use diagnostics
00:35:41 to figure out
00:35:41 what's the problem
00:35:42 and then fix them
00:35:43 correspondingly.
00:35:45 One kind of diagnostics
00:35:47 is,
00:35:47 as I said,
00:35:48 is to figure out
00:35:48 whether it's underfitting,
00:35:50 overfitting,
00:35:51 or maybe just doing fine.
00:35:52 Okay?
00:35:52 This is just one diagnostic.
00:35:54 There are other diagnostics
00:35:55 as well.
00:35:56 Okay?
00:35:56 So,
00:35:57 in this case,
00:35:58 we are trying to figure out
00:35:59 whether it's underfitting,
00:36:00 overfitting,
00:36:01 or probably both.
00:36:02 Right?
00:36:03 Then we can compare,
00:36:04 for instance,
00:36:05 the training error
00:36:05 and the validation error.
00:36:08 Right?
00:36:08 Right?
00:36:09 So,
00:36:09 if both errors
00:36:11 are high,
00:36:12 that means
00:36:13 probably a high bias
00:36:16 on the fitting issue.
00:36:17 Right?
00:36:18 If the training error
00:36:20 is low,
00:36:20 but the validation error
00:36:21 is high,
00:36:22 right,
00:36:24 you might be experiencing
00:36:26 the high variance issue,
00:36:28 which is overfitting.
00:36:29 Okay?
00:36:30 By the way,
00:36:30 you have to be careful.
00:36:31 In reality,
00:36:33 is your training set
00:36:34 and the validation set
00:36:35 from the same distribution
00:36:36 or not?
00:36:37 Right?
00:36:37 Because I have seen
00:36:38 sometimes people
00:36:39 actually use a much more
00:36:41 challenging validation set
00:36:42 than they found
00:36:42 that the performance
00:36:43 of validation set
00:36:44 is so bad.
00:36:45 Then,
00:36:46 you know,
00:36:47 maybe that really prevents you
00:36:48 from the diagnostics.
00:36:50 because here,
00:36:52 this diagnostic
00:36:52 is assuming
00:36:54 that training
00:36:55 and the validation
00:36:56 data are from
00:36:58 the same distribution.
00:37:00 Okay?
00:37:01 Right?
00:37:01 Otherwise,
00:37:02 the validation set
00:37:03 is going to be
00:37:04 in a target domain
00:37:06 that's very different
00:37:07 from the origin domain.
00:37:08 Then,
00:37:09 there is a transfer error,
00:37:10 of course.
00:37:11 Right?
00:37:11 So,
00:37:12 then we cannot just draw
00:37:14 that conclusion
00:37:14 that easily.
00:37:15 Right?
00:37:16 And,
00:37:17 a good case,
00:37:18 of course,
00:37:18 is training error
00:37:19 is low
00:37:19 and the validation error
00:37:20 is just a little bit higher.
00:37:22 That's a good fit.
00:37:24 Right?
00:37:24 So,
00:37:25 that's a good thing.
00:37:26 So,
00:37:26 now let's,
00:37:27 let's try to see that.
00:37:30 Right?
00:37:32 So,
00:37:32 let's see.
00:37:33 Right?
00:37:33 Let's say there is,
00:37:35 let's look at
00:37:37 this error level.
00:37:41 Right?
00:37:42 What I'm changing here
00:37:44 is the size
00:37:45 of the training set.
00:37:46 Okay?
00:37:47 And,
00:37:48 let's say that
00:37:48 there is an ideal,
00:37:50 desired performance
00:37:51 like here.
00:37:52 This is like the
00:37:54 center of the dartboard
00:37:56 you want to land in.
00:37:57 Okay?
00:37:57 That's the,
00:37:58 that's the ideal case.
00:37:59 Okay?
00:38:00 So,
00:38:02 let's see in this case,
00:38:03 right?
00:38:04 As the training set size
00:38:06 increases,
00:38:07 the test,
00:38:08 you know,
00:38:08 the training error
00:38:09 and the test error
00:38:11 are both getting closer
00:38:12 to the desired performance.
00:38:14 Okay?
00:38:14 Please don't get confused.
00:38:16 I'm not saying
00:38:17 that training error
00:38:17 is increasing.
00:38:18 Okay?
00:38:19 So,
00:38:19 this is like the,
00:38:20 approaching this desired
00:38:22 performance from both directions.
00:38:24 Okay?
00:38:25 So,
00:38:26 in other words,
00:38:27 as the training set size
00:38:28 increases,
00:38:30 right,
00:38:30 so both are getting closer
00:38:32 to the desired performance.
00:38:33 Right?
00:38:34 But what you see here
00:38:35 is the following,
00:38:38 that
00:38:39 the training error
00:38:40 is,
00:38:41 you know,
00:38:41 much smaller
00:38:42 compared to the test error.
00:38:43 right?
00:38:44 Or,
00:38:46 in our case,
00:38:46 I would rather call it
00:38:49 validation or development error.
00:38:51 Right?
00:38:51 Because I usually
00:38:52 do the test set
00:38:55 evaluation
00:38:56 in the end.
00:38:57 We don't,
00:38:58 you know,
00:38:58 we just deploy
00:38:59 the trained model once.
00:39:01 So,
00:39:01 usually,
00:39:01 I look at the validation error
00:39:03 instead.
00:39:05 Okay?
00:39:05 All right.
00:39:07 So,
00:39:07 in this case,
00:39:08 the training error
00:39:10 is low
00:39:10 and the validation error
00:39:11 is high.
00:39:12 So,
00:39:12 we call it
00:39:13 a typical
00:39:14 high variance issue.
00:39:15 Right?
00:39:16 So,
00:39:17 in this case,
00:39:18 we see that
00:39:21 if we keep increasing
00:39:22 the training set size,
00:39:23 it may help
00:39:24 because,
00:39:25 after all,
00:39:26 test error
00:39:27 is decreasing.
00:39:28 Right?
00:39:29 Just slower
00:39:29 than training error does.
00:39:31 So,
00:39:31 if you keep increasing
00:39:32 the training set size,
00:39:33 it can still
00:39:35 help,
00:39:37 but there is still
00:39:38 this gap
00:39:39 may get saturated.
00:39:40 So,
00:39:41 you need to do something.
00:39:42 Okay?
00:39:42 We'll talk about
00:39:43 something later.
00:39:45 Another case
00:39:46 is like this.
00:39:47 Right?
00:39:47 So,
00:39:47 in this case,
00:39:48 there's not
00:39:50 too much difference,
00:39:51 actually,
00:39:52 as the training set size
00:39:53 increases
00:39:53 between the training
00:39:54 and the validation set error.
00:39:56 That is good,
00:39:57 but,
00:39:58 unfortunately,
00:39:59 both are
00:40:00 unacceptably high
00:40:02 from the
00:40:03 desired performance.
00:40:05 So,
00:40:06 if that's the case,
00:40:07 it's a typical
00:40:08 high bias.
00:40:10 Right?
00:40:10 Because the variance
00:40:11 is low,
00:40:12 but the bias is high.
00:40:13 So,
00:40:13 these are
00:40:13 the diagnoses
00:40:15 you can look at.
00:40:16 Right?
00:40:16 So,
00:40:17 in practice,
00:40:18 sometimes,
00:40:18 people actually
00:40:19 don't change
00:40:20 the training set size
00:40:21 a lot sometimes,
00:40:22 but still,
00:40:23 you can
00:40:24 even do the diagnosis
00:40:27 during the training.
00:40:28 Right?
00:40:29 You can look at
00:40:29 the errors
00:40:31 over epochs,
00:40:32 even,
00:40:32 during training.
00:40:33 So,
00:40:35 now,
00:40:37 once you figure out
00:40:39 the problem,
00:40:42 now you can choose
00:40:44 the fixes accordingly.
00:40:45 Okay?
00:40:46 So,
00:40:47 as I said,
00:40:48 if you want to fix
00:40:50 the high variance,
00:40:51 you may want to get
00:40:52 more training examples.
00:40:54 So,
00:40:54 but remember,
00:40:55 this is only to fix
00:40:56 the high variance.
00:40:56 Right?
00:40:57 And,
00:40:59 smaller set of features
00:41:00 is what?
00:41:02 Is reducing
00:41:03 the model complexity.
00:41:05 Right?
00:41:05 So,
00:41:05 it also fixes
00:41:06 high variance.
00:41:07 Right?
00:41:08 If you use a larger
00:41:09 set of the features,
00:41:10 right,
00:41:11 to some extent,
00:41:11 it's increasing
00:41:13 the model complexity,
00:41:14 you may fix
00:41:17 a high bias issue.
00:41:18 Or,
00:41:19 you are using
00:41:19 the email header features,
00:41:20 using more
00:41:21 informative features.
00:41:24 Right?
00:41:24 In this case,
00:41:24 you can also fix
00:41:25 a high bias issue.
00:41:26 Right?
00:41:27 So,
00:41:28 as you can see,
00:41:29 this is,
00:41:30 you know,
00:41:32 what you will do
00:41:33 after you figure out
00:41:34 what's the problem.
00:41:35 Right?
00:41:35 You wouldn't try
00:41:35 all of them,
00:41:36 would you?
00:41:37 Right?
00:41:37 Okay,
00:41:38 then you ask,
00:41:39 well,
00:41:40 what about the remainder?
00:41:41 Well,
00:41:42 the remainder
00:41:42 has not,
00:41:44 are fixes
00:41:45 that have nothing
00:41:47 to do with
00:41:47 the first diagnostic
00:41:48 we talked about.
00:41:49 Because the first diagnostic
00:41:50 is only about
00:41:51 overfitting
00:41:53 or underfitting.
00:41:54 Okay,
00:41:54 we haven't talked
00:41:55 about other
00:41:55 diagnostics yet.
00:41:57 What other
00:41:58 diagnostic?
00:42:00 Don't forget
00:42:01 that we are,
00:42:02 we are solving
00:42:02 an optimization problem.
00:42:05 Right?
00:42:07 So,
00:42:09 one question
00:42:10 you have to ask
00:42:11 is,
00:42:13 if I'm optimizing
00:42:16 the right function,
00:42:19 is my optimization
00:42:20 algorithm converging?
00:42:23 Okay,
00:42:23 let's say
00:42:24 I'm optimizing
00:42:25 the right function,
00:42:26 but is it converging?
00:42:28 Right?
00:42:28 Did I report
00:42:29 a converging solution?
00:42:31 Right?
00:42:31 So,
00:42:32 in other words,
00:42:32 it's optimizers
00:42:33 convergence issue,
00:42:35 or maybe I'm not even
00:42:36 optimizing the right function.
00:42:38 Right?
00:42:39 So,
00:42:40 in other words,
00:42:41 you have to figure out,
00:42:43 previously the diagnostic
00:42:44 is about your model
00:42:45 performance,
00:42:46 whether it's
00:42:46 high variance
00:42:47 or high bias.
00:42:49 Then,
00:42:50 this diagnostic
00:42:50 is about
00:42:51 optimization,
00:42:53 formulation,
00:42:54 and algorithm.
00:42:56 Right?
00:42:57 So,
00:42:57 let's try to figure out
00:42:58 both.
00:42:59 Right?
00:42:59 Whether your formulation
00:43:00 is correct.
00:43:00 The choice
00:43:01 of the loss function
00:43:02 is agreeing
00:43:04 with what you want.
00:43:05 Right?
00:43:06 And,
00:43:07 if that's the case,
00:43:08 whether the optimizer
00:43:09 is doing the right thing
00:43:10 to get the optimal parameters.
00:43:12 Okay?
00:43:14 Okay,
00:43:14 so,
00:43:15 first of all,
00:43:16 let's ask what we care about
00:43:17 for this spam detection.
00:43:20 Right?
00:43:20 So,
00:43:21 as I was asking you earlier,
00:43:22 right,
00:43:23 do you care about
00:43:24 the false positives
00:43:25 or false negatives
00:43:26 more?
00:43:28 Right?
00:43:28 But,
00:43:29 in general,
00:43:30 we can say that
00:43:31 there is probably
00:43:32 an assessment,
00:43:34 right,
00:43:34 like an indicator,
00:43:36 right,
00:43:36 which is a weighted sum
00:43:38 of all these indicator functions.
00:43:40 Right?
00:43:42 Right?
00:43:42 So,
00:43:43 this is
00:43:44 when the predicted
00:43:47 is agreeing
00:43:48 with the actual
00:43:49 mail,
00:43:51 email label.
00:43:52 Right?
00:43:52 We can also
00:43:53 try to,
00:43:54 we can try to maximize
00:43:55 this one,
00:43:56 right,
00:43:56 or we can try to minimize
00:43:57 when this
00:43:58 equality changes
00:44:00 to inequality.
00:44:01 Right?
00:44:02 But,
00:44:03 depending on
00:44:04 the Y
00:44:05 being the spam
00:44:06 or non-spam,
00:44:08 we may want to give
00:44:09 different weights
00:44:10 to that correct decision.
00:44:12 Right?
00:44:13 Right?
00:44:13 So,
00:44:13 in other words,
00:44:14 you may care about
00:44:15 this one,
00:44:17 this measurement
00:44:19 that is more
00:44:21 reflecting
00:44:22 the actual performance
00:44:24 of this spam detection.
00:44:26 Okay?
00:44:26 By the way,
00:44:27 this is not always,
00:44:28 this is not always
00:44:30 the loss function
00:44:31 you use.
00:44:32 Okay?
00:44:32 Because you see
00:44:33 in the different
00:44:33 machine learning models,
00:44:36 they have different
00:44:36 principles
00:44:38 and different loss functions.
00:44:40 Whether it's
00:44:40 maximum margin,
00:44:42 I'm talking about
00:44:42 the classifiers for now,
00:44:44 or in this case,
00:44:45 the maximum likelihood,
00:44:48 or the,
00:44:49 you know,
00:44:49 maximum
00:44:50 posterior,
00:44:52 right?
00:44:52 So,
00:44:53 there are
00:44:53 different principles.
00:44:54 Right?
00:44:55 And those principles
00:44:56 determine the loss function,
00:44:58 but they
00:44:59 don't have to be
00:45:01 actually the
00:45:01 performance evaluator.
00:45:03 So,
00:45:04 maybe in this case,
00:45:05 you care about
00:45:05 the performance
00:45:06 that can be
00:45:07 valid in this way.
00:45:08 Okay?
00:45:09 All right.
00:45:10 So,
00:45:10 now,
00:45:11 let's
00:45:12 look at
00:45:14 the following.
00:45:16 Okay?
00:45:17 Let's try to figure out
00:45:18 whether it's
00:45:19 a formulation issue
00:45:20 or optimizer issue.
00:45:22 Okay?
00:45:23 Okay.
00:45:26 For,
00:45:27 we first can figure out
00:45:28 whether it's
00:45:29 a formulation issue,
00:45:30 right?
00:45:30 For instance,
00:45:31 you wondered
00:45:32 whether I should
00:45:32 use support vector
00:45:33 machine,
00:45:34 right?
00:45:35 Versus
00:45:36 evasion logistic
00:45:37 regression,
00:45:37 right?
00:45:37 Let's assume
00:45:38 that for each
00:45:39 model,
00:45:40 the formulation
00:45:41 is already
00:45:42 correct in the
00:45:42 sense that
00:45:43 you have taken
00:45:44 care of this
00:45:44 lambda or c,
00:45:45 these hyperparameters.
00:45:47 Okay?
00:45:47 So,
00:45:47 for each
00:45:48 formulation,
00:45:49 the corresponding
00:45:49 hyperparameters
00:45:50 is already
00:45:51 taken care of.
00:45:51 Let's assume
00:45:52 that.
00:45:53 Okay?
00:45:54 Once you
00:45:55 have that,
00:45:56 right?
00:45:56 Because you see
00:45:57 that these
00:45:58 different formulations
00:45:59 are different,
00:46:00 right?
00:46:01 Vaspation logistic
00:46:02 regression is
00:46:02 maximizing this
00:46:03 j,
00:46:04 right?
00:46:04 And then
00:46:05 support vector
00:46:05 machine is
00:46:06 minimizing
00:46:06 this
00:46:07 one over
00:46:09 the
00:46:11 squared margin,
00:46:13 right?
00:46:14 Plus
00:46:15 the slack
00:46:16 variables
00:46:17 for the
00:46:17 soft
00:46:18 right?
00:46:22 So,
00:46:22 now,
00:46:25 okay,
00:46:25 so then
00:46:26 let's see.
00:46:28 First,
00:46:29 let's say
00:46:30 the support vector
00:46:31 machine gives you
00:46:32 a parameter
00:46:32 theta
00:46:32 SVM,
00:46:34 and
00:46:34 Bayesian logistic
00:46:35 regression gives you
00:46:36 the parameter
00:46:37 theta
00:46:37 BRR,
00:46:38 okay?
00:46:39 You can ask
00:46:40 yourself,
00:46:41 which one performs
00:46:42 better
00:46:43 in the eyes
00:46:45 of the
00:46:47 weighted accuracy
00:46:47 that you care
00:46:48 about,
00:46:49 okay?
00:46:49 You can plug in
00:46:50 these two models
00:46:51 and ask
00:46:52 which one has
00:46:53 better performance,
00:46:54 right?
00:46:55 Okay,
00:46:56 let's say that
00:46:57 you found actually
00:46:58 support vector
00:46:58 machine even
00:46:59 does better
00:47:00 in the sense
00:47:02 of this
00:47:02 performance
00:47:03 evaluation
00:47:03 than
00:47:04 Bayesian
00:47:05 logistic
00:47:05 regression,
00:47:06 okay?
00:47:08 Well,
00:47:08 then you ask
00:47:09 the following,
00:47:10 is my Bayesian
00:47:13 logistic
00:47:13 regression
00:47:13 performing
00:47:16 worse
00:47:17 than support
00:47:17 vector machine
00:47:18 because
00:47:19 it's
00:47:20 Bayesian
00:47:21 logistic
00:47:21 regression
00:47:22 or because
00:47:23 I actually
00:47:24 didn't optimize
00:47:25 this Bayesian
00:47:26 logistic regression
00:47:27 problem,
00:47:29 okay?
00:47:30 Right?
00:47:30 I didn't find
00:47:31 the optimum.
00:47:32 So this is,
00:47:33 right?
00:47:33 So to answer
00:47:34 this question,
00:47:36 right,
00:47:36 you can compare
00:47:37 both support
00:47:39 vector machine
00:47:40 and the Bayesian
00:47:40 logistic regression
00:47:42 in the same
00:47:44 objective function
00:47:45 as the Bayesian
00:47:46 logistic regression
00:47:47 J,
00:47:48 okay?
00:47:50 Remember,
00:47:50 Bayesian logistic
00:47:51 regression is
00:47:51 trying to maximize
00:47:52 that J,
00:47:53 right?
00:47:54 That's the battlefield
00:47:55 of Bayesian
00:47:56 logistic regression.
00:47:57 If your
00:47:58 support vector
00:47:59 machine's
00:48:00 answer
00:48:01 is even
00:48:02 better than
00:48:03 the Bayesian
00:48:03 logistic regression
00:48:04 in its own
00:48:05 battlefield,
00:48:07 the logist,
00:48:08 sorry,
00:48:08 the last function
00:48:09 that BRR
00:48:10 is trying
00:48:10 to maximize,
00:48:12 that means
00:48:12 what?
00:48:15 The optimization
00:48:16 algorithm,
00:48:18 or optimizer,
00:48:19 did not give us
00:48:20 an optimal solution,
00:48:22 or far from it,
00:48:23 right?
00:48:23 Because we were
00:48:24 trying to maximize
00:48:25 J,
00:48:26 but,
00:48:27 you know,
00:48:27 actually,
00:48:28 when our optimizer
00:48:30 trying to maximize
00:48:31 for J,
00:48:32 actually,
00:48:32 it's worse
00:48:33 than support vector
00:48:34 machine,
00:48:35 which is trying
00:48:35 to optimize
00:48:35 for something
00:48:36 else,
00:48:37 but it turned
00:48:38 out to be
00:48:38 even better,
00:48:40 also in,
00:48:41 you know,
00:48:41 if also
00:48:42 evaluated in J,
00:48:43 is that right?
00:48:44 Right?
00:48:45 So,
00:48:45 it's,
00:48:45 you know,
00:48:46 it's like,
00:48:48 you know,
00:48:50 you pick
00:48:50 the high jump
00:48:52 champion,
00:48:53 right?
00:48:54 And then
00:48:54 compete with you
00:48:55 on the 100-meter
00:48:57 track,
00:48:57 and then actually,
00:48:58 you know,
00:48:59 he or she
00:48:59 outperforms you,
00:49:00 then you may
00:49:01 wonder
00:49:02 whether your race
00:49:04 of the 100-meter
00:49:05 was competitive
00:49:06 enough,
00:49:07 right?
00:49:07 So,
00:49:08 you need a better
00:49:08 optimizer,
00:49:09 so that's one
00:49:10 issue,
00:49:10 right?
00:49:11 So,
00:49:12 in the case one,
00:49:13 as I said,
00:49:13 that support vector
00:49:14 machine has better
00:49:15 performance
00:49:16 in our performance
00:49:18 assessment,
00:49:19 and it even beat
00:49:21 the BRR in its own
00:49:22 track,
00:49:23 in its own loss
00:49:24 function,
00:49:25 right?
00:49:25 So,
00:49:26 that means
00:49:26 our optimization
00:49:28 for the Bayesian
00:49:29 logistical regression
00:49:30 failed to maximize
00:49:31 J,
00:49:31 far from it,
00:49:32 right?
00:49:33 So,
00:49:33 the problem is
00:49:34 now with the
00:49:35 convergence of
00:49:35 the optimizer,
00:49:37 right?
00:49:37 Right?
00:49:38 So,
00:49:38 that's one
00:49:38 possibility,
00:49:41 right?
00:49:41 The other possibility
00:49:42 is that,
00:49:43 yes,
00:49:43 the support vector
00:49:44 machine is better
00:49:45 than BRR
00:49:45 in the performance
00:49:46 we care,
00:49:48 but its
00:49:50 solution is not
00:49:53 necessarily better
00:49:53 than BRR
00:49:54 in the track
00:49:55 of BRR.
00:49:56 In other words,
00:49:57 if we evaluated
00:49:58 these two,
00:49:59 both in J,
00:50:00 this loss
00:50:01 function of
00:50:02 BRR,
00:50:04 BRR is,
00:50:05 you know,
00:50:05 still better.
00:50:06 Support vector
00:50:07 machines model
00:50:07 are not
00:50:08 naturally better,
00:50:09 okay?
00:50:10 So,
00:50:11 this means
00:50:11 that probably,
00:50:13 probably,
00:50:15 right?
00:50:15 I'm not saying
00:50:16 there's guaranteed
00:50:17 no issue
00:50:17 with optimizer,
00:50:19 but so far
00:50:20 there's,
00:50:21 you know,
00:50:21 nothing suspicious
00:50:22 about optimizer.
00:50:24 There's more
00:50:25 issues similarly
00:50:26 with the loss
00:50:26 function similarly,
00:50:28 right?
00:50:28 Because if I,
00:50:30 you know,
00:50:30 because support vector
00:50:31 machine apparently
00:50:32 uses a different
00:50:33 loss function,
00:50:34 right?
00:50:35 And it's not
00:50:36 better than the BRR
00:50:37 in J,
00:50:38 this optimal solution,
00:50:40 but the support vector
00:50:42 machine chosen
00:50:42 by its own
00:50:43 margin maximization
00:50:45 performed better,
00:50:46 right?
00:50:48 So this means
00:50:49 that probably,
00:50:50 in this case,
00:50:51 support vector
00:50:52 machine is a better
00:50:53 formulation,
00:50:55 optimization,
00:50:55 formulation
00:50:56 than Bayesian
00:50:57 logistic regression,
00:50:58 isn't it?
00:50:59 Right?
00:50:59 So this means
00:51:00 that maybe
00:51:01 the J of theta
00:51:02 with this Bayesian
00:51:03 logistic regression,
00:51:04 right,
00:51:04 is probably the wrong
00:51:05 function to maximize,
00:51:07 and what you care
00:51:08 about is A of theta,
00:51:10 then the loss function
00:51:11 of support vector
00:51:12 machine may agree
00:51:13 with your A of theta
00:51:14 closer,
00:51:15 okay?
00:51:16 Then you may wonder,
00:51:17 well,
00:51:17 if I care about
00:51:18 A of theta,
00:51:19 why don't I directly
00:51:20 optimize A of theta
00:51:22 in training my model?
00:51:24 So that's a complicated
00:51:26 question, right?
00:51:27 First of all,
00:51:28 not all of such
00:51:29 A of theta
00:51:30 can be easily
00:51:32 analytically expressed
00:51:34 and then solved
00:51:36 by gradient algorithms,
00:51:37 right?
00:51:38 But it's half and half.
00:51:40 One example of
00:51:41 is area under
00:51:42 the precision recall curve,
00:51:44 right?
00:51:44 If you try to
00:51:45 maximize the area
00:51:47 under the precision recall curve,
00:51:49 you rarely see that
00:51:50 directly goes into
00:51:51 the loss function
00:51:52 of training your
00:51:53 classifier, right?
00:51:54 But now people
00:51:55 are developing
00:51:56 better and better
00:51:57 algorithms so that
00:51:58 you can have
00:51:59 surrogate functions
00:52:00 who are easy
00:52:01 to optimize,
00:52:02 but whose optimality
00:52:04 more or less
00:52:05 aligns with the
00:52:07 optimum area
00:52:08 under the precision recall curve.
00:52:10 Okay,
00:52:10 that's a possibility,
00:52:11 okay?
00:52:12 But it still
00:52:13 remains a challenge.
00:52:14 The second one
00:52:15 is that different model
00:52:15 have their different
00:52:16 assumptions, right?
00:52:18 So they not necessarily
00:52:19 all agree with
00:52:20 this performance measure
00:52:22 you care about,
00:52:22 A of theta, right?
00:52:23 So sometimes
00:52:25 as a trade-off,
00:52:25 some people just
00:52:26 use A of theta
00:52:27 as the validation
00:52:30 assessment.
00:52:31 So when you
00:52:32 choose a hyperparameters,
00:52:34 you evaluate A of theta
00:52:36 on the validation set,
00:52:38 and then that's
00:52:39 something that people do,
00:52:40 right?
00:52:41 But there are
00:52:42 all kinds of possibilities.
00:52:43 But so far,
00:52:44 let's come back
00:52:45 to the second diagnostic.
00:52:46 It's about
00:52:47 optimization, right?
00:52:48 Whether we are
00:52:49 looking at the
00:52:51 correct optimization
00:52:52 formulation or
00:52:53 objective,
00:52:55 or are we using
00:52:56 the correct optimizer
00:52:57 or optimization
00:52:58 algorithm, right?
00:53:00 So if it's
00:53:01 an algorithm issue,
00:53:02 then I probably
00:53:03 should make
00:53:04 it more likely
00:53:05 to converge,
00:53:06 like run it
00:53:07 more iterations
00:53:08 or try Newton's method,
00:53:10 right?
00:53:11 In the case of
00:53:11 deep learning,
00:53:12 you may use
00:53:13 different methods,
00:53:14 change
00:53:15 stochastic descent
00:53:17 to
00:53:18 atom
00:53:19 or even
00:53:20 something more
00:53:21 sophisticated,
00:53:22 right?
00:53:23 Because,
00:53:24 why?
00:53:24 Because atom
00:53:25 uses more information
00:53:26 like momentum,
00:53:27 right?
00:53:28 Than stochastic
00:53:28 descent.
00:53:31 You can also
00:53:31 try more
00:53:32 sophisticated algorithm,
00:53:33 like in the
00:53:34 low-dimensional case,
00:53:35 you can try
00:53:36 Newton's method,
00:53:36 but in the deep
00:53:37 learning,
00:53:37 as I said,
00:53:38 you may consider
00:53:38 like atom
00:53:40 and others
00:53:41 that use
00:53:41 second-order information
00:53:43 or momentum
00:53:43 or more information
00:53:44 than gradients,
00:53:46 right?
00:53:47 If it's
00:53:48 an objective issue,
00:53:49 maybe your lambda
00:53:50 was an issue,
00:53:51 okay?
00:53:51 I previously said
00:53:52 I hope your lambda
00:53:53 was fine,
00:53:54 but probably your lambda
00:53:55 is an issue,
00:53:56 okay?
00:53:57 But if your lambda
00:53:59 is not the issue,
00:54:00 then maybe you
00:54:01 should change
00:54:02 the loss function
00:54:03 and the model
00:54:04 totally,
00:54:05 right?
00:54:05 So in this case,
00:54:06 probably changing it
00:54:08 to a support vector
00:54:09 machine,
00:54:09 which is actually
00:54:10 agreeing with your
00:54:11 performance metric
00:54:12 even more.
00:54:13 Any question?
00:54:16 So now that you see
00:54:18 we have two kinds
00:54:20 of diagnostics,
00:54:21 right?
00:54:21 One is about
00:54:22 the model behavior,
00:54:23 right?
00:54:24 Right?
00:54:25 From the perspective
00:54:26 of bias and variance.
00:54:28 One is from
00:54:29 the optimization,
00:54:30 right?
00:54:30 From the,
00:54:31 you know,
00:54:31 perspective
00:54:32 algorithm
00:54:34 and objective
00:54:35 or formulation,
00:54:36 right?
00:54:36 So depending
00:54:38 on the outcome
00:54:39 outcome of each
00:54:40 diagnostic,
00:54:42 we can choose
00:54:43 the corresponding
00:54:45 fixes,
00:54:46 right?
00:54:46 So in other words,
00:54:47 what you do is
00:54:48 starting from the far
00:54:50 right,
00:54:50 there is something
00:54:51 called diagnostic.
00:54:52 Based on the
00:54:53 diagnostic,
00:54:54 you figure out
00:54:54 that I want to fix
00:54:55 a high variance
00:54:56 or a high bias
00:54:57 or optimization
00:54:59 algorithms
00:54:59 or optimization
00:55:01 objective
00:55:01 and then you
00:55:02 decide to use
00:55:03 one of them.
00:55:04 Okay?
00:55:04 So that's
00:55:05 the direction.
00:55:07 It's not the other
00:55:08 way around
00:55:09 that you try
00:55:09 all the possible
00:55:10 fixes and write
00:55:11 all of them
00:55:12 down and then
00:55:13 try all of them,
00:55:14 right?
00:55:15 Okay,
00:55:18 just some
00:55:19 summary of the
00:55:20 quick fixes,
00:55:21 right?
00:55:21 For the high variance,
00:55:22 we can try more
00:55:23 examples,
00:55:24 but if you cannot
00:55:25 afford it,
00:55:26 try a bagging,
00:55:27 right?
00:55:28 And then
00:55:29 try less features,
00:55:31 right?
00:55:32 Try regularization,
00:55:33 right?
00:55:34 If you try to fix
00:55:35 a high bias,
00:55:36 you may want to
00:55:37 try a larger set
00:55:38 of features
00:55:38 with more information,
00:55:40 right?
00:55:40 That really is the
00:55:41 case.
00:55:42 I have seen so
00:55:43 many papers
00:55:44 focusing on
00:55:45 the deep
00:55:46 learning
00:55:47 architecture
00:55:47 so much.
00:55:49 They introduced
00:55:49 very sophisticated
00:55:50 architecture,
00:55:52 but there are
00:55:53 two scenarios,
00:55:54 okay?
00:55:55 Two possibilities.
00:55:57 One possibility
00:55:57 is that their
00:55:58 performance did
00:55:59 improve compared
00:56:00 to the state of
00:56:01 art,
00:56:02 but not really
00:56:03 because of the
00:56:03 architecture,
00:56:04 but because of
00:56:05 their features
00:56:06 were much more
00:56:07 indicative
00:56:09 and informative.
00:56:11 That's one
00:56:12 possibility.
00:56:12 possibility.
00:56:13 The second
00:56:14 possibility is
00:56:15 that they
00:56:16 tried a lot
00:56:17 of things,
00:56:17 but they just
00:56:18 cannot beat
00:56:19 the state of
00:56:19 art.
00:56:20 Maybe somewhere
00:56:21 here they're
00:56:22 doing slightly
00:56:22 better,
00:56:23 but there,
00:56:24 you know,
00:56:24 it's like
00:56:24 roughly the
00:56:26 same.
00:56:27 Then people
00:56:27 ask why bother
00:56:28 trying so many
00:56:29 things,
00:56:29 right?
00:56:31 And very
00:56:32 often they
00:56:33 just forget
00:56:33 about an
00:56:34 easy,
00:56:36 well,
00:56:36 it's conceptually
00:56:37 easy,
00:56:37 just practically
00:56:38 difficult thing.
00:56:39 That is,
00:56:41 come up
00:56:41 with smart
00:56:42 features.
00:56:43 I know
00:56:43 everybody is
00:56:43 talking about
00:56:44 end-to-end
00:56:44 as if you
00:56:45 can stack
00:56:46 the original
00:56:48 raw data
00:56:49 into the
00:56:50 input end
00:56:51 and then
00:56:51 hope the
00:56:52 output end
00:56:52 will give
00:56:53 you the
00:56:54 perfect solution
00:56:54 when you train
00:56:56 the model
00:56:56 enough.
00:56:57 That does not
00:56:58 happen that
00:56:58 way.
00:56:59 Actually,
00:56:59 you still have
00:57:00 to spend a
00:57:00 lot of time
00:57:01 on features,
00:57:02 right?
00:57:03 But that
00:57:04 is painful
00:57:04 for people
00:57:05 who are
00:57:06 not familiar
00:57:07 with the
00:57:07 domain
00:57:07 knowledge,
00:57:09 right?
00:57:10 If you
00:57:10 just treat
00:57:11 the protein
00:57:11 sequence as
00:57:12 an alphabet
00:57:13 of,
00:57:14 I'm sorry,
00:57:15 as a text
00:57:15 written in
00:57:16 alphabet of
00:57:16 20 letters,
00:57:17 that's super
00:57:17 easy.
00:57:18 But if I
00:57:18 ask you
00:57:19 to do,
00:57:20 well,
00:57:20 what about
00:57:21 the position
00:57:21 specific
00:57:22 sequence,
00:57:24 I'm sorry,
00:57:24 position specific
00:57:26 scoring matrix
00:57:27 or profile,
00:57:28 then you need
00:57:29 to do the
00:57:29 multiple sequence
00:57:30 alignment and
00:57:31 construct those.
00:57:32 And there are
00:57:32 other kind of
00:57:33 things,
00:57:33 right?
00:57:34 Then you may
00:57:34 figure that out
00:57:35 that's too
00:57:36 much painful,
00:57:37 right?
00:57:38 But we
00:57:39 really need
00:57:40 to pay
00:57:41 attention to
00:57:42 this issue
00:57:43 because very
00:57:44 often what I
00:57:44 see is that
00:57:46 our models
00:57:47 are both
00:57:48 high variance
00:57:50 and high bias.
00:57:51 And the high
00:57:52 variance is
00:57:52 easier to treat
00:57:53 actually.
00:57:54 The high bias,
00:57:55 you actually need
00:57:56 more information
00:57:56 from the domain.
00:57:58 That's an
00:57:58 issue,
00:57:59 okay?
00:58:01 And then
00:58:02 sometimes it's
00:58:03 about optimizer,
00:58:04 right?
00:58:04 the objective
00:58:05 function and
00:58:06 optimization
00:58:07 algorithms,
00:58:08 okay?
00:58:10 And this
00:58:11 was where I
00:58:12 downloaded before
00:58:13 but I think the
00:58:13 link might have
00:58:14 been changed
00:58:15 and maybe the
00:58:16 slides were even
00:58:16 taken away.
00:58:18 So let's see
00:58:19 whether that
00:58:20 still works.
00:58:21 You may not be
00:58:22 able to directly
00:58:22 click it because
00:58:23 sometimes this
00:58:24 hyphen,
00:58:25 when converted
00:58:27 into the
00:58:28 HTTP URL link,
00:58:31 it gets into a
00:58:33 different format
00:58:35 or something.
00:58:35 So it becomes
00:58:37 a different
00:58:37 ASCII character.
00:58:39 So if you click
00:58:40 it directly,
00:58:40 you may not get
00:58:41 it, okay?
00:58:42 But usually if you
00:58:43 change that weird
00:58:45 ASCII character
00:58:46 into a hyphen,
00:58:47 you should be able
00:58:47 to get it, okay?
00:58:48 So this is about
00:58:54 the diagnosis,
00:58:56 okay?
00:58:56 Another thing I
00:58:57 want to say is
00:58:58 two kind of analysis
00:58:59 of your models,
00:59:00 okay?
00:59:00 Let's say you have
00:59:02 your model and you
00:59:03 want to understand
00:59:04 your model behavior
00:59:05 and you want to
00:59:07 improve your model
00:59:08 behavior.
00:59:09 So these are two
00:59:12 things that we
00:59:13 try to figure out.
00:59:14 So what you can
00:59:15 do is two
00:59:16 directions, right?
00:59:18 You can start
00:59:19 with your model
00:59:21 and look at
00:59:22 different components
00:59:24 and replace
00:59:26 each component
00:59:27 with the most
00:59:29 ideal scenario
00:59:30 and see whether
00:59:32 that helps
00:59:34 a lot or not,
00:59:36 right?
00:59:36 Then you figure
00:59:36 out, oh, maybe
00:59:37 I need to pay
00:59:38 more attention
00:59:39 to this component.
00:59:40 So that's called
00:59:41 the error analysis,
00:59:42 okay?
00:59:43 The error analysis
00:59:44 is about, you know,
00:59:46 what if I change
00:59:47 each component
00:59:48 to much better
00:59:50 scenario?
00:59:50 Does that improve
00:59:51 the performance
00:59:52 a lot or not, okay?
00:59:53 If that is the case,
00:59:54 then I'm going to
00:59:55 spend more time
00:59:55 on this component,
00:59:56 okay?
00:59:58 Another kind of
00:59:58 analysis is what
01:00:00 you often do
01:00:00 called ablation study.
01:00:02 So in the case
01:00:02 of the ablation study,
01:00:04 right, you start
01:00:06 with, again,
01:00:07 with your baseline
01:00:08 model, right,
01:00:10 with different
01:00:10 components,
01:00:11 and you are saying
01:00:12 that, well,
01:00:13 let me take away
01:00:15 the good contributions
01:00:16 I made to each
01:00:17 component and see
01:00:18 how much worse
01:00:20 the performance
01:00:21 becomes, okay?
01:00:22 In that way,
01:00:23 I can also
01:00:24 delineate, right,
01:00:26 or isolate the
01:00:27 contributions
01:00:28 from each individual
01:00:30 components.
01:00:31 So both are important,
01:00:32 right?
01:00:32 One is about
01:00:33 how to move forward.
01:00:35 One is how to
01:00:37 explain that
01:00:37 what you introduced
01:00:38 actually worked,
01:00:40 okay?
01:00:40 But they are just
01:00:41 opposite directions
01:00:42 in some sense,
01:00:43 okay?
01:00:44 So now let's see.
01:00:45 You have a machine
01:00:46 learning model
01:00:47 with a pipeline,
01:00:48 right?
01:00:48 So let's say
01:00:49 this is a face
01:00:51 recognition, right?
01:00:52 It's a much easier
01:00:53 problem nowadays.
01:00:55 There's a camera image.
01:00:56 You have to
01:00:57 preprocess it,
01:00:58 like removing background,
01:00:59 and then you have
01:01:00 to detect the face,
01:01:02 and then you need
01:01:03 to do segmentation,
01:01:04 figuring out
01:01:04 which pixels
01:01:05 correspond for eyes,
01:01:06 nose, mouth,
01:01:07 and then you may
01:01:09 end up having,
01:01:10 using this as features
01:01:11 to a machine
01:01:12 learning model
01:01:13 and then predict
01:01:14 the label, okay?
01:01:15 So this is a very
01:01:16 easy case.
01:01:18 Now, you see that
01:01:19 this is actually
01:01:20 a very sophisticated
01:01:21 machine learning
01:01:22 pipeline already,
01:01:24 and that's often
01:01:25 the case, right?
01:01:26 So let's see.
01:01:28 The error analysis
01:01:30 basically asks
01:01:31 how much error
01:01:32 is attributable
01:01:34 to each of the components,
01:01:37 okay?
01:01:37 So it's basically asking,
01:01:39 you know,
01:01:39 how much error
01:01:40 is attributable
01:01:41 to each component?
01:01:42 So in this case,
01:01:44 as I said,
01:01:45 for each component,
01:01:47 you can plug
01:01:48 in the ground truth
01:01:49 because the most
01:01:51 ideal component
01:01:53 here is to give you
01:01:55 the exact ground truth,
01:01:57 right?
01:01:57 For instance,
01:01:58 it's giving me
01:01:59 the face detection
01:02:00 perfectly, right?
01:02:02 In reality,
01:02:03 your preprocessing
01:02:05 may not give you
01:02:07 the ideal input
01:02:08 or your face detection
01:02:09 may not give you
01:02:10 the perfect face detection,
01:02:11 okay?
01:02:12 But let's say
01:02:13 that I replace
01:02:14 each component
01:02:16 with the ground truth,
01:02:18 okay?
01:02:19 If that's the best
01:02:20 you can do,
01:02:21 how does the accuracy
01:02:21 change this, okay?
01:02:23 For instance,
01:02:24 in this case,
01:02:25 the overall system
01:02:27 is 85%,
01:02:28 if I do the preprocessing
01:02:31 perfectly,
01:02:33 like expert,
01:02:34 just manually remove
01:02:35 the background
01:02:36 for instance,
01:02:37 you see the accuracy
01:02:38 only increased
01:02:38 by 0.1%,
01:02:41 okay?
01:02:41 That means,
01:02:42 all right,
01:02:43 stop there.
01:02:43 You don't need to spend
01:02:44 more time on background
01:02:46 preprocessing anymore,
01:02:48 okay?
01:02:49 But if it's a face detection,
01:02:50 let's say,
01:02:51 you know,
01:02:51 I always detect a face
01:02:53 successfully
01:02:54 without any error
01:02:56 and then pass
01:02:57 to the downstream components,
01:02:59 then you see
01:03:00 the accuracy increase
01:03:01 to 91%.
01:03:02 Great.
01:03:02 That's something
01:03:03 I can improve,
01:03:04 right?
01:03:05 And then let's say
01:03:07 starting from
01:03:08 whatever the
01:03:09 imperfect face detection,
01:03:11 right?
01:03:14 Oh, sorry.
01:03:14 I think in this case
01:03:16 it's probably,
01:03:17 let's see,
01:03:18 is it cumulative?
01:03:20 The numbers look increasing,
01:03:21 but I think he,
01:03:25 I think we are plugging
01:03:26 in this individually,
01:03:28 but do you think
01:03:30 that in this case,
01:03:32 in this case,
01:03:33 eyes segmentation,
01:03:35 we definitely plug in
01:03:36 the ground truth
01:03:37 for the eyes segmentation,
01:03:38 but do you think
01:03:39 we keep the ground truth
01:03:41 for the face detection
01:03:42 from the early stage
01:03:43 or not?
01:03:44 Based on the numbers,
01:03:48 it probably is the case.
01:03:50 I cannot tell for sure,
01:03:51 okay?
01:03:52 But it makes sense
01:03:53 to say that
01:03:54 let me incrementally
01:03:55 add things one by one,
01:03:56 but usually I do have
01:03:59 these individual perturbations
01:04:01 like the marginal perturbations.
01:04:02 I do perturb each one
01:04:04 separately
01:04:06 without incrementally,
01:04:07 okay?
01:04:08 You could have another lineage
01:04:09 of incremental,
01:04:12 like the ablation study,
01:04:13 you could, you know,
01:04:14 incrementally reduce
01:04:15 more and more components,
01:04:16 but usually you will also
01:04:18 leave some other combinations,
01:04:20 right?
01:04:21 Like I just remove
01:04:22 any one of these components,
01:04:25 right?
01:04:25 And then starting
01:04:25 from one of them,
01:04:26 I incrementally remove
01:04:28 more and more.
01:04:29 So in this case,
01:04:30 we see that eyes segmentation,
01:04:32 nose segmentation,
01:04:34 and mouth segmentation,
01:04:35 let's assume that
01:04:36 these are just
01:04:37 plugging in the
01:04:38 corresponding ground truth
01:04:40 alone without changing
01:04:41 the other parts,
01:04:42 okay?
01:04:43 So it looks like
01:04:44 if that's the case,
01:04:45 the most room
01:04:46 for improvement
01:04:47 is in face detection
01:04:49 and eyes segmentation.
01:04:52 If that's the conclusion,
01:04:54 then I think they are
01:04:55 doing this incrementally,
01:04:57 right?
01:04:57 If that's the conclusion.
01:04:58 Because they figured out,
01:05:00 oh, after adding
01:05:01 the face detection
01:05:02 perfectly,
01:05:03 I jumped,
01:05:04 increased 6%,
01:05:05 and then another jump,
01:05:07 4% is,
01:05:08 on top of the
01:05:09 perfect face detection,
01:05:10 I add the
01:05:11 perfect eyes segmentation.
01:05:13 So this one is incremental.
01:05:15 Okay?
01:05:15 Is this clear?
01:05:16 So that means what?
01:05:18 You want to spend
01:05:18 more time
01:05:19 on face detection
01:05:21 and eyes segmentation
01:05:23 and improve those components,
01:05:25 right?
01:05:26 You can save your efforts
01:05:27 and maximize your outcome.
01:05:30 So this is what
01:05:31 error analysis does
01:05:32 if you want to
01:05:33 improve your method.
01:05:34 Ablation analysis
01:05:37 is trying to explain
01:05:43 why your current model
01:05:44 is behaving like that.
01:05:47 Okay?
01:05:47 So error analysis
01:05:48 is explaining
01:05:49 the difference
01:05:49 between the current performance
01:05:51 and the perfect performance,
01:05:53 right?
01:05:53 What if I replace
01:05:54 the individual components
01:05:56 by the perfect ground truth?
01:05:58 Ablation analysis
01:05:59 is trying to explain
01:05:59 the difference
01:06:00 between your current performance
01:06:03 and some baseline
01:06:04 when you remove
01:06:06 some good properties
01:06:09 of that component,
01:06:11 right?
01:06:13 So,
01:06:14 for instance,
01:06:16 if you build
01:06:16 anti-spam,
01:06:17 right,
01:06:18 on top of this
01:06:19 logistical regression,
01:06:21 you also have
01:06:22 these clever contributions,
01:06:23 right?
01:06:23 You say that,
01:06:25 well,
01:06:25 these are my major contributions,
01:06:26 right?
01:06:27 In the end,
01:06:27 when I have
01:06:28 all these contributions
01:06:29 together,
01:06:30 I did improve
01:06:31 compared to Steve-Art,
01:06:33 by 10%,
01:06:34 right?
01:06:35 But then the question
01:06:36 is,
01:06:37 you know,
01:06:38 how much
01:06:39 did each of these components
01:06:41 actually help,
01:06:42 right?
01:06:43 So,
01:06:43 in other words,
01:06:44 you want to remove
01:06:45 some feature
01:06:46 in quotation,
01:06:47 I don't mean
01:06:47 the input feature,
01:06:48 I mean the properties
01:06:49 of the algorithm
01:06:50 and see the impact
01:06:52 to the performance,
01:06:53 right?
01:06:54 So,
01:06:54 in this case,
01:06:55 it's probably
01:06:56 incrementally
01:06:58 removing
01:06:59 the components,
01:07:00 right?
01:07:01 Right?
01:07:01 The reason I think
01:07:04 incremental
01:07:05 adding or removing
01:07:07 is tricky
01:07:08 is because
01:07:09 the order may not
01:07:10 be that clear.
01:07:11 So,
01:07:11 sometimes,
01:07:12 when we have
01:07:13 a few contributions
01:07:15 or components,
01:07:15 we usually have to combine
01:07:17 the incremental
01:07:18 and the individual,
01:07:19 you know,
01:07:20 any one or any two
01:07:21 sometimes even.
01:07:23 So,
01:07:23 in this case,
01:07:24 let's say the accuracy
01:07:25 is already 99.9%.
01:07:27 if we remove
01:07:29 our contribution
01:07:30 to the spelling correction,
01:07:32 right?
01:07:32 So,
01:07:33 the spelling correction.
01:07:34 If we don't do
01:07:35 spelling correction,
01:07:36 the accuracy
01:07:37 is 99.0.
01:07:39 If we don't include
01:07:41 the features
01:07:42 from the sender host,
01:07:43 the accuracy drops
01:07:44 to 98.9,
01:07:45 but,
01:07:46 you know,
01:07:46 you keep dropping
01:07:47 the major contributions
01:07:49 that you introduced
01:07:51 and that separates you
01:07:53 from the baseline
01:07:54 or state of art,
01:07:54 then you see that
01:07:56 eventually drops
01:07:57 to 94.0,
01:07:58 okay?
01:07:59 This is also
01:08:00 a good idea
01:08:01 sometimes
01:08:01 when your
01:08:02 implementation
01:08:04 is different
01:08:06 from the baseline,
01:08:07 right?
01:08:08 Sometimes,
01:08:08 you had to use
01:08:09 your own implementation.
01:08:10 You cannot change
01:08:11 the baseline repo
01:08:12 for your own contribution,
01:08:14 right?
01:08:14 So,
01:08:15 in that case,
01:08:16 you definitely want
01:08:17 to do the ablation study
01:08:18 all the way to see here
01:08:19 whether the performance
01:08:20 agree with the published
01:08:21 baseline performance
01:08:22 because it's possible
01:08:23 that your implementation
01:08:24 has some faulty issue
01:08:26 and you realize
01:08:27 that I'm always
01:08:28 like 80% here
01:08:29 compared to the baseline,
01:08:31 right?
01:08:31 Which is 82%.
01:08:33 What was wrong?
01:08:34 Then you keep
01:08:35 ablating your features
01:08:36 and realize that,
01:08:37 oh,
01:08:38 if I remove
01:08:38 all my contributions
01:08:39 and my accuracy
01:08:40 is only 70%,
01:08:41 right?
01:08:42 Then why is my
01:08:43 implementation
01:08:44 of the baseline
01:08:44 so different
01:08:45 from the reported baseline?
01:08:47 There must be
01:08:47 some implementation
01:08:48 issue as well.
01:08:49 So,
01:08:49 that's another reason
01:08:50 I like this kind
01:08:51 of ablation study,
01:08:52 right?
01:08:52 Because you think
01:08:53 conceptually,
01:08:54 if I,
01:08:55 you know,
01:08:55 make these contributions
01:08:56 they should help,
01:08:58 right?
01:08:58 But if you remove
01:08:59 all of them
01:09:00 you cannot even
01:09:00 replicate the baseline
01:09:01 then that means
01:09:02 some implementation
01:09:03 issues are there,
01:09:05 okay?
01:09:07 And in this case
01:09:09 we can conclude
01:09:10 where the biggest
01:09:11 job was
01:09:12 here,
01:09:15 98.9% to 95%,
01:09:17 right?
01:09:18 So it looks like
01:09:19 the email
01:09:19 text parser features
01:09:20 are most important
01:09:22 similarly,
01:09:22 right?
01:09:23 So,
01:09:24 and I think
01:09:25 now it's almost
01:09:26 the standard
01:09:26 that you need
01:09:28 to include
01:09:28 ablation study
01:09:29 in your paper
01:09:32 to show
01:09:32 that you understand
01:09:33 or you have
01:09:35 a proof
01:09:36 that your major
01:09:37 contributions
01:09:38 actually had
01:09:39 an impact
01:09:40 of your performance
01:09:42 improvement
01:09:43 against the state of the art,
01:09:45 okay?
01:09:46 So these are
01:09:47 the two kinds
01:09:47 of analysis.
01:09:48 Again,
01:09:48 error analysis
01:09:49 and ablation study,
01:09:50 okay?
01:09:51 Again,
01:09:51 during the development
01:09:52 process you may want
01:09:54 to do some error analysis
01:09:55 trying to see
01:09:56 how you can improve
01:09:58 but once you figure it out
01:09:59 you can do the ablation study,
01:10:01 right?
01:10:01 Usually in our research
01:10:03 what we do
01:10:04 is the reverse
01:10:05 ablation study.
01:10:06 We start with a baseline
01:10:07 and we think
01:10:08 that we have several ideas
01:10:09 and then we
01:10:10 build upon this idea,
01:10:13 build upon this
01:10:14 baseline
01:10:15 and add these ideas
01:10:16 one by one.