Lecture 13.1: Evaluating Classifiers

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 So this week and the next Tuesday were expected to finish the machine learning module because this is a module that many of you have already taken.

00:00:10 So I'm just trying to review some key concepts and I hope that you have gone through the complementary slackness after our lecture last Thursday and understood why those Lagrange multipliers or dual variables

00:00:29 in the support vector machines formulation were zero if or non-zero if and only if the corresponding primal constraints are not active.

00:00:46 So those multi-variable, multi-pliers and dual variables will be non-zero only when the primal constraints are active.

00:00:58 Those are support vectors because each primal constraint is corresponding to a training example, right?

00:01:08 If that constraint is active, then that example is a support vector.

00:01:14 Okay, so today and Thursday we're going to talk about classification and regression.

00:01:21 Two key concepts are about evaluating classifiers and integrating heterogeneous data today.

00:01:30 Then on Thursday we're going to talk about the regression and bias variance trade-off and how you can be guided under such principles to regularize the model complexity for the optimal trade-off.

00:01:48 And next Tuesday we're going to wrap up the machine learning module with some very practical knowledge about how do you diagnose your machine learning models.

00:02:02 Okay, so then we'll finish the machine learning module and advance to the deep learning module.

00:02:08 Any questions so far?

00:02:11 Okay, so let's see where we are.

00:02:15 Previously we talked about support vector machine for binary classification which can be extended in optimization formulation very easily to address, for instance, the soft margins versus the hard margins.

00:02:36 Right?

00:02:37 And you can also address multi-class classification versus binary in this formulation.

00:02:45 You can also change classification to regression in this formulation.

00:02:49 So it's a very flexible formulation.

00:02:52 And thanks to the fact that the training examples only enter the dual formulations through the inner products or the kernels and the optimal solution also only relies on the kernels between the support vectors and your x.

00:03:14 And due to that reason, support vector machines is a kernel method which can be easily extended from the linear inner product kind of kernel to nonlinear kernels.

00:03:29 So, and you can go beyond this feature space into a higher dimensional kernel space.

00:03:38 Right?

00:03:39 And we talked about the fact that this is a simple quadratic programming which can be solved analytically.

00:03:46 Right?

00:03:47 And then there's an article that probably gave you some historical context about why there was a time that support vector machine completely

00:03:58 beat the earlier neural networks.

00:04:01 Hope you enjoyed that.

00:04:02 So today I'm going to review first quickly, I'm going to talk about classification, but it's more than that.

00:04:08 How do you evaluate classifiers?

00:04:11 So it's just a quick review, but then I would spend more time talking about how do you integrate heterogeneous data.

00:04:18 Don't forget that our motivating example is protein function prediction.

00:04:23 Right?

00:04:24 And we'll use that example to show you how to integrate heterogeneous data and theoretical ideas behind them.

00:04:32 So let's talk about how do you evaluate classifiers.

00:04:36 So how do we evaluate classifiers?

00:04:38 Anyone?

00:04:41 So you train a machine learning model in this case, for instance, a binary classifier.

00:04:46 How do you evaluate your classifier?

00:04:47 You all know.

00:04:48 So in other words, you're going to have to look at a confusion matrix and then do some statistics.

00:05:05 All right?

00:05:06 So very good.

00:05:07 Anything else?

00:05:08 Depends upon a problem.

00:05:09 So if you want to solve a problem where we need to decrease false positive and then we

00:05:32 validate based on that.

00:05:33 So basically we need some extra samples.

00:05:34 You can say validation for testing samples and validate performance.

00:05:39 So you're saying that depending on the problem and the data characteristics as well as the desired

00:05:50 performance, you may favor one assessment versus the other.

00:05:55 And you also mentioned the validation.

00:05:57 We'll talk about that at the end as well.

00:05:59 So yes, so as we said before, you're going to evaluate with validation set or the development

00:06:09 set.

00:06:10 So the first one is just like Nami said, is the confusion matrix or the contingency table.

00:06:16 All right?

00:06:17 So what you do is you first split your data into positives or negatives based on the data.

00:06:28 The actual class, right?

00:06:29 Then you also further split them from the second axis that is a predicted class.

00:06:37 Then you basically split all the training example or validation set into four quadrants, right?

00:06:45 And these quadrants are corresponding to true positives, false positives and so on.

00:06:50 When it's a true positive, it means that actual class is positive and you predict it to be positive.

00:06:54 It is positive and it's a true positive.

00:06:57 False positive means that it's predicted to be a positive but the actual class is negative.

00:07:03 Okay?

00:07:04 So the positives and negatives are talking about the prediction and the true or false is about

00:07:12 the agreement between the prediction and the actual class.

00:07:15 Okay?

00:07:16 Based on these values, right?

00:07:19 Because you split the cardinality, you split the data set into four quadrants.

00:07:24 So the cardinality, so you have four cardinalities.

00:07:29 So these four subsets and you can use these cardinalities and calculate a lot of things, right?

00:07:35 So the true positive rate is called the recall.

00:07:39 It's also called sensitivity, which is out of everything that's predicted to be positive, what portion is true positive?

00:07:47 In other words, it's looking at the first row, right?

00:07:53 You're looking at the block of true positive divided by the true positive plus false positives.

00:08:00 Okay?

00:08:01 So that's a sensitivity.

00:08:03 False positive rate is a false alarm rate.

00:08:06 That's the number of false positives divided by the number of negatives.

00:08:11 So the false positive divided by this second column.

00:08:16 All right?

00:08:18 Oh, wait a second.

00:08:21 When I said the true positive, did I say the first row?

00:08:24 I apologize for that.

00:08:26 It's the first column, okay?

00:08:29 So the true positives divided by everything that is actually positive.

00:08:34 Okay?

00:08:35 So out of everything that's positive, including the false positives that you made the wrong prediction,

00:08:41 what percentage is true positive?

00:08:43 Okay?

00:08:44 So the sensitivity, this is the false alarm rate, false positive rate.

00:08:49 So accuracy is this diagonal, true positives plus true negatives divided by the total.

00:08:58 So this could be, depending on whether, you know, your positive and the negatives are balanced or not,

00:09:06 this accuracy could be largely dominated by the positive or the negative class, right?

00:09:12 So then there is a balanced accuracy, right?

00:09:15 And precision is out of everything that's predicted to be positive, right?

00:09:20 So that's the first column.

00:09:22 That's the first row.

00:09:24 Everything that's predicted to be positive, right?

00:09:27 So that includes true positives and false positives.

00:09:31 And among them, what portion is true positive?

00:09:34 And then F score is blending precision and recall together, and specificity is one minus the false positive rate, right?

00:09:46 Right?

00:09:47 So in other words, it is true negatives divided by the negatives.

00:09:53 Okay?

00:09:54 So there are all kinds of measures, right?

00:09:57 So you can choose one of them, right?

00:09:59 So what is the limitation of these measures?

00:10:02 Do we need anything else to assess your classifier?

00:10:07 What other assessment have you seen in classification?

00:10:18 And when would you use those assessments?

00:10:22 Of course, there's other kind of measures to summarize, right?

00:10:28 MCC for instance, right?

00:10:30 You can summarize these performances based on these measures.

00:10:35 Is there anything else?

00:10:37 You must have used the ROCK curve, right?

00:10:43 Receiver operating characteristic curves.

00:10:47 So when do you use them?

00:10:49 So the receiver operating characteristic curve is one minus specificity or false positive rate versus sensitivity, which is the two positive rate.

00:11:06 So the fact that you have more than one point in this two-dimensional space is because your classifier could have different thresholds, right?

00:11:18 So depending on the classifier predicted values, above or below this moving threshold, you could have different points, right?

00:11:28 Let's look at this case.

00:11:31 This point is zero, zero.

00:11:34 So the true positive rate is zero and the false positive rate is zero.

00:11:40 What scenario is this?

00:11:42 When would you have this?

00:11:44 True positive rate is zero and the false positive rate is also zero.

00:11:49 When would that happen?

00:11:50 Huh?

00:11:51 No data?

00:11:52 No data?

00:11:53 False positive rate equal to zero.

00:11:54 Let's look at false positive rate.

00:11:55 It is the number of false positives divided by negatives.

00:12:12 So that means false positive equal to zero.

00:12:16 That means two negatives equal to negatives, right?

00:12:23 In the meantime, what?

00:12:26 The true positive rate is zero.

00:12:29 So that means this is zero, right?

00:12:37 And false negatives equals two positives.

00:12:40 That means for all the negatives, you predict them to be negatives.

00:12:46 So the false negative rate is zero.

00:12:49 But in the meantime, for all the positives, you also predict them to be negatives, right?

00:12:55 So your predictions are focused in the second row.

00:12:59 There are either true negatives or false negatives.

00:13:02 In other words, you predict everything to be negative, okay?

00:13:06 Similarly, if you are here, then the sensitivity equal to one.

00:13:12 So the true positive equal to positive.

00:13:15 But the false positive rate equals to one as well.

00:13:21 So that means what?

00:13:23 That means all the negatives were predicted to be positives as well, right?

00:13:29 So in this case, we have everything to be predicted to be positives, right?

00:13:39 And what about this point?

00:13:41 This point is false positive equal to zero and two positive equal to one.

00:13:48 What does it mean?

00:13:50 Two positive equal to one, that means out of all the positives, they are indeed all predicted to be positives.

00:13:58 But in the meantime, the false positive rate equal to zero, that means out of all the negatives, none of them were mistakenly classified either.

00:14:07 So that means perfectly classified, right?

00:14:12 All positives are predicted to be two positives and all negatives are predicted.

00:14:16 So that is a perfect prediction.

00:14:20 So if you change the threshold of your classifier, you may end up having multiple points.

00:14:27 For instance, you reach A and you reach B and you, you know, reach this point, right?

00:14:33 So in this case, we may never get this perfect point, right?

00:14:45 But this tells you that this is a rock curve that basically tells you the threshold independent behavior of your classifier.

00:14:55 Then if you want to quantify this one, you can just look at the area under the curve.

00:15:01 Apparently, the highest IR under the curve is this perfect condition, that's one.

00:15:07 The lowest is totally opposite, that's zero.

00:15:10 So if it's between zero and one, the higher the better, okay?

00:15:15 Okay, so how do you do that?

00:15:17 As I said, again, you know, you should know this very well, that for instance, if you have this class distribution,

00:15:24 and you also have the model or classifier predicted score, right?

00:15:29 So of course, if your threshold is fixed, like 0.5, then you can calculate given a fixed number of sensitivity or specificity, right?

00:15:40 But if your threshold is changing, your threshold is changing, then you end up having a rock curve, right?

00:15:49 Like this, right?

00:15:51 So again, it always passed 0, 0 and 1, 1, right?

00:15:57 But then you see, as we go above, right?

00:16:00 For instance, we can start from this side, right?

00:16:04 Where we predict everything to be positive, right?

00:16:08 Then we start, so that's like the threshold is minus infinity, right?

00:16:12 So everything is predicted to be negative, positive, excuse me.

00:16:17 So that's where we are.

00:16:19 Then we gradually increase the threshold from minus infinity to something, and the first time that threshold change will result in a sensitivity or specificity change is when that threshold reaches the smallest score.

00:16:37 Is that right?

00:16:38 Right?

00:16:39 Like 0.1, right?

00:16:41 So once you are above 0.1, you can say that, well, everything, right?

00:16:47 Everything is positive except number 20, because that's below, right, 0.1.

00:16:54 So in that case, it is negative.

00:16:57 So in this case, we predicted all the positives correctly, because they are all above 0.1.

00:17:04 So the true positive rate is 1.

00:17:09 But the false, so, and then we keep going and reach 0.3, right?

00:17:17 Then we finally reached a wrong classification.

00:17:20 When you say that the threshold is slightly above 0.3, then we mistakenly think that number 19 is a negative, right?

00:17:29 Because the higher the score, we think it's positive.

00:17:31 So in that case, your false negative rate is going to be 0.9 here, right?

00:17:39 So you're going to keep dropping.

00:17:41 And why do you have this staircase kind of shape?

00:17:46 Why?

00:17:47 Why do we have this staircase shape?

00:17:53 Because we have finite samples, right?

00:18:02 Again, these sensitivity specificity only change when your threshold hits one of these score values, isn't it?

00:18:11 If you have infinite amount of samples, then all the scores are very close to each other, right, like a fluid distribution or continuum distribution, then your curve will be smoother and smoother, isn't it?

00:18:26 Right?

00:18:27 But anyway, you can calculate the area under the curve.

00:18:31 So this is just a review.

00:18:32 I believe that you have learned this before.

00:18:35 But can anyone tell me why do we want to bother to do this, use a threshold-independent rock curve and quantify the classifier's performance using the area under the curve?

00:18:48 Why do we bother to do this?

00:18:50 Why aren't we content with just a fixed point for a given threshold?

00:18:56 Why do we need the curve?

00:18:59 Anyone?

00:19:01 Or when do we need the curve?

00:19:05 Any ideas?

00:19:18 Okay, so, as I said, the rock curve gives you the ability to rank the two classes, right?

00:19:34 So there's no doubt of that.

00:19:36 And it's going through all the thresholds and see this distribution and look at the curve and the area under the curve, right?

00:19:45 So the benefit of looking at the rock curve is that it's not that sensitive to the class distributions, okay?

00:19:52 So you may look at different accuracy or sensitivity level for different data sets simply because they have different class distributions, right?

00:20:03 So, you know, then, right?

00:20:06 So that's one issue you don't want to be biased about, right?

00:20:12 So that's one reason that you'd like to use these kind of curves, okay?

00:20:17 So, besides, and then, you know, saying that, oh, this is a single value performance between zero and one, very convenient, there is also a very deep reason of utilizing this area under the curve.

00:20:37 It's not something heuristic.

00:20:39 It actually equals to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

00:20:52 So this is a chance that a randomly chosen positive is going to be ranked higher than the randomly chosen positive, negative.

00:21:02 So if it's one, it's really ideal.

00:21:04 If it's zero, it's very bad.

00:21:06 And a random predictor like a flip coin is going to give you 0.5, right?

00:21:13 So that corresponds to this dashed line exactly.

00:21:20 So this is a random classifier whose area under the curve is 0.5, which exactly gives half chance to rank the random positive above the random negative, okay?

00:21:31 So there's a deep reason of using this area under the curve.

00:21:36 It's not heuristic, okay?

00:21:39 Besides the area under the rock curve, there's also the area under the precision recall curve, okay?

00:21:48 So sometimes the rock curve can still be misleading, especially for imbalanced classification with a severe skew and a few examples of the minority classes, right?

00:22:00 Sometimes we have very small percentage in the data set that belongs to the positive.

00:22:07 For instance, if you look at the protein-protein interactions, right?

00:22:12 So if you look at the residues in protein 1 and the residues in protein 2, right?

00:22:17 And then we have M times M pairs, right?

00:22:20 If you classify those M times M pairs into positives, interacting pairs, versus negatives, non-interacting pairs, the interacting pairs are minority, isn't it?

00:22:32 That's like less than 1%, isn't it?

00:22:35 So in those cases, we have a severe skew, and the rock AUC can be extremely misleading.

00:22:42 You can get rock AUC like above 90% very easily, to be honest.

00:22:48 And in those cases, so I put some codes here.

00:22:52 You can read them into more details, right?

00:22:55 So in those cases, the precision recall curves are recommended for the highly skewed domains, okay?

00:23:02 So in those cases, you know, your x-axis is the recall.

00:23:06 Like the, you know, sensitivities, 1-specificity, the y-axis was the recall, okay?

00:23:12 But your y-axis here is the precision.

00:23:16 That's a true positive divided by everything that's predicted to be positive, okay?

00:23:23 So in this case, the perfect point is here, at 1 and 1.

00:23:29 Precision equal to 1 and recall equal to 1.

00:23:32 That means, you know, recall equal to 1 means out of all the positives, they are all correctly predicted to be positive.

00:23:40 Right?

00:23:41 So that's perfect.

00:23:42 Okay?

00:23:43 But of course, in reality, your model might be more like this, okay?

00:23:44 It will always pass this 0, 1 and 1, 0, okay?

00:23:59 And in this case, what's a random classifier like?

00:24:03 In this case, a random classifier will have a flat line with a constant precision, right?

00:24:09 And this precision equals to the minority positive class percentage, okay?

00:24:15 If it's balanced, then this level will be 0.5, okay?

00:24:20 So that's the control you compare to.

00:24:22 It's no longer you're comparing to 0.5 anymore.

00:24:25 Sometimes you could have area under the precision recall curve extremely small, like 0.01,

00:24:31 but you have to compare to the baseline, which could be one out of a million, okay?

00:24:36 So you have to use these numbers in their contexts, okay?

00:24:43 All right.

00:24:46 Any questions so far?

00:24:49 Okay, after reviewing what metrics are used, you know, just as you said that you have learned before that,

00:24:56 now that we have learned each classifier like the W in the linear classifier or learned from the training set,

00:25:02 we can evaluate the performance over the validation or the test set, right?

00:25:09 But before doing that, there are still so-called hyperparameters to choose.

00:25:14 For instance, if you're using the Gaussian kernel in the support vector machine, there is a sigma there.

00:25:21 That is not a parameter, but you're really treated as a hyperparameter, okay?

00:25:27 So, you know, in the neural networks, the number of layers, the number of neurons, those are like hyperparameters, right?

00:25:36 So for those kind of hyperparameters, their optimization or gradients are difficult, right?

00:25:41 So there's a reason that they are treated as hyperparameters.

00:25:44 But even in this case, sigma is continuous, you still treat it as hyperparameters because, you know,

00:25:50 they're in nature different from the parameters.

00:25:53 So there is a question that remains is that is how do we optimize these hyperparameters, right?

00:26:00 As you usually do is that you do like a nested loop, right?

00:26:05 In the outer loop, you optimize the hyperparameters and then for each in the inner loop,

00:26:11 for each hyperparameter combination, you train the model to optimize the parameters using the gradient-based algorithms, right?

00:26:21 So one way to optimize the hyperparameter is using the validation set

00:26:28 and do the k-fold cross-validation, right?

00:26:31 So as a reminder, you just randomly split your training set into training and validation set into equal k partitions.

00:26:43 k-1 of them will be used as training and one-fold will be used as the validation set.

00:26:50 And you can keep doing this and in total you have k ways to do that, right?

00:26:55 So if it's tenfold cross-validation, you can just repeat this process of training ten times and validating ten times, right?

00:27:05 Okay, then for each value of a parameter, you have to average the performance in all rounds, right?

00:27:13 Of a hyperparameter, I'm sorry about that, right?

00:27:16 So for instance, in each round, you can use ARC, AUPRC, we have talked about before, right?

00:27:26 And then you can average them over the ten, right?

00:27:29 And then that is the average performance for that hyperparameter combination choice, okay?

00:27:37 So this is a very typical method, k-fold cross-validation.

00:27:41 Again, I recommend you to take the course pattern recognition from Dr. Bragagnetto, who actually talked a lot about the small sample issue in the training machine learning.

00:27:53 So in those cases, k-fold cross-validation could be a very biased and in a non-optimal way to choose the hyperparameters, okay?

00:28:05 So, I'll see you next time.

00:28:06 Thank you.