Lecture 13.2: Integrating Heterogeneous Data

Recording. Download the text transcript.

Automatic transcript; technical terms and equations may contain recognition errors.

00:00:00 I think I'm ready to talk about how do you integrate heterogeneous data.

00:00:03 It's not just for classification, it's also for regression.

00:00:07 But for regression, I want to talk more about the bias-variance trade-off

00:00:12 and how do you regularize model complexities.

00:00:15 So those will be left in the next lecture.

00:00:18 Okay, so now let's talk about machine learning.

00:00:22 The perspective I want to emphasize is the heterogeneous data, right?

00:00:27 So again, using the machine, using the motivating question of

00:00:33 how do you predict a function of a new protein, right?

00:00:36 So the kind of data or the input you can give include

00:00:41 the amino acid sequences, 3D structures,

00:00:44 and the high-throughput data, and so on, right?

00:00:47 So you could have a sequence, but you actually could

00:00:52 featurize them in different ways, right?

00:00:55 You could have sequence similarity, local, global.

00:00:59 So you may wonder similarity to what?

00:01:03 This must be similarity to the training examples, right?

00:01:06 If you see the similarity, you should get reminded about kernels, right?

00:01:13 So in this case, when you use the similarity,

00:01:16 you can actually think about the kernel methods even.

00:01:19 So in the support vector machines, you don't have to give the features XI, right?

00:01:26 You can actually just directly give the kernels matrix for all the XI, XJ pairs

00:01:33 in the support vector machines directly, right?

00:01:36 So there's a genome context and then feature base like the subcellular location

00:01:41 and the function class and so on.

00:01:43 And then for the 3D structure, you could have the global structure and the local structure, right?

00:01:48 The local structure might be more responsible of finding small molecule

00:01:53 or, you know, catalyzing some chemical reactions, right?

00:01:58 Don't forget that we have also high-throughput data like a gene expression, right?

00:02:03 Birds of the same feather flock together, right?

00:02:09 So if they flock together, they might be of the same feather.

00:02:12 And protein-protein interaction, tell me who your friends are, I can tell who you are, right?

00:02:17 So this is...

00:02:18 Now, facing such heterogeneous data, right?

00:02:24 How do we integrate them together for our machine learning problem?

00:02:29 That is protein function prediction.

00:02:31 Again, we already formulated it as a machine learning problem because the principle is obscure,

00:02:38 but the data is abundant, okay?

00:02:41 So, what I want to say is the following.

00:02:47 Whatever data you use, you have to keep in mind that we are actually having some hypothesis

00:02:57 about the causal relationships.

00:02:58 Again, if you recall at the beginning, at the very, very beginning...

00:03:02 Okay, I don't have the figure here.

00:03:06 For this machine learning, what we are trying to do is to mimic the IO relationships of the natural processes.

00:03:16 In other words, there are some causal relationships of the input between the input and output, right?

00:03:22 When you actually feed some kind of input data to your machine learning models, you have already formulated

00:03:30 some assumptions or hypotheses about how they might act upon this natural process, right?

00:03:38 Right?

00:03:39 So, you know, for instance, if you use a sequence, you are using the assumption that, you know,

00:03:46 two proteins might share the similar function if they have the similar sequence and similar enough.

00:03:51 This similarity could be measured at a global level or local level or the genome context level, right?

00:03:57 Because, for instance, co-located on the chromosome in multiple genomes,

00:04:01 or they may be evolving together and having similar phylogenetic profiles, right?

00:04:06 So, you actually have different hypotheses and assumptions when you are using those features, right?

00:04:12 And similarly, when you use the structure features like global or local similarity at different levels

00:04:18 or fold or 3D motif or binding site, again, right, you are using different assumptions, right?

00:04:25 These different data are also different in the sense whether they are high-throughput enough or not, right?

00:04:30 And then there's a high-throughput data we talked about, right?

00:04:35 Gene expression, right?

00:04:36 And then protein-protein interactions, right?

00:04:39 So, we have learned this concept already when we motivate machine learning problem using protein function prediction, right?

00:04:50 The question is how do we integrate them together, right?

00:04:53 How do we integrate this different data together, right?

00:04:55 You have to look at this different data.

00:04:58 So, right now, I don't say it that way, but I want you to think beyond what I currently say

00:05:07 and think about this data in their different modalities, okay?

00:05:11 I want you to keep in mind that we have text data and we have, well, depending on how you represent them,

00:05:18 you could have graph data, you could have manifold data, you could have point cloud data for the structure, depending on how you...

00:05:26 You have the gene expression data, which is tabular.

00:05:30 You have the protein-protein interaction data that is graph.

00:05:32 So, I want you to now think about them as different modalities and how do you integrate them together.

00:05:38 I'm not going to talk about, you know, integrating different modalities in this lecture, okay?

00:05:43 I'm going to talk about something more fundamental, but I want you to think about them and keep those perspectives in mind when we go to the deep learning module, okay?

00:05:53 So, right? So, that's something to keep in mind.

00:05:57 So, some earlier methods are quite straightforward.

00:06:03 Well, not that straightforward. I can tell a few things, right?

00:06:06 For instance, the gene expression and the PPI data can be unified very easily.

00:06:11 For instance, you could use a functional linkage network, right?

00:06:15 And you have the topology that's determined by the protein-protein interaction networks.

00:06:20 Then, the coloring, if two nodes are over the similar color, then you use...

00:06:26 Then, that's based on the co-expression, right?

00:06:29 But, you don't have to do that right now, right?

00:06:32 Right now, you can use a graph modeling, right?

00:06:34 We're using the expression data as node features, isn't it?

00:06:39 Right? So, you can do that, right?

00:06:41 And there's probabilistic models like Bayesian networks and Markov random fields, right?

00:06:49 And then, machine learning, including the earlier support vector machines.

00:06:54 You will see why support vector machines were popular, because of one thing that is very good at...

00:07:00 Beyond everything I said last time, it's also a kernel method.

00:07:06 So, you will see why kernel methods are useful.

00:07:08 And there's earlier versions of deep learning models, including Boseman machines, restricted Boseman machines, and the earlier deep learning models.

00:07:19 Okay?

00:07:20 But, I want you to, again, think about the data in different modalities, and think about the most recent or advanced deep learning models.

00:07:29 And ask yourself, you have different modalities of the input.

00:07:32 How do you integrate them together?

00:07:34 Okay?

00:07:35 And there are a lot of more thorough reviews here.

00:07:41 So, the earlier methods can be based on the homology.

00:07:44 We talked about BLAST or SciBLAST.

00:07:46 So, it's like searching the queries sequence in a template library, where sequences are already functionally annotated.

00:07:55 So, BLAST or SciBLAST are good at finding homology or even remote homologies.

00:08:01 But, remember, BLAST and SciBLAST, even though they are heuristic database search, they're still too slow.

00:08:08 Okay?

00:08:09 Because we have so many sequences.

00:08:10 So, something even faster called diamond is giving you a heuristic, even more heuristic, but it's super fast.

00:08:18 Okay?

00:08:19 But, that's still based on homology.

00:08:21 And then, let's look at the machine learning methods, right?

00:08:24 So, actually, there are a lot of different sources depending on what's the input of your model, right?

00:08:30 Some are using sequence.

00:08:32 So, they are increasingly using transformers, for instance.

00:08:36 There is a structure-based methods.

00:08:38 There is network-based methods.

00:08:40 Then, there is integration of the sequence plus network.

00:08:45 Then, there's text-based methods based on the text mining.

00:08:49 There is a sequence plus structure plus network-based.

00:08:53 In other words, they are already integrating.

00:08:55 So, you may wonder how they integrate.

00:08:57 So, it's worthwhile looking into them.

00:09:00 Then, there is a sequence plus structure plus network plus text mining altogether.

00:09:05 Okay?

00:09:06 So, in other words, this multi-modal data integration is definitely being picked up, right?

00:09:17 So, I'm going to talk about some classical view of data integration.

00:09:23 You know, how do you integrate heterogeneous data in machine learning?

00:09:26 You know, it's machine learning.

00:09:28 I'm not talking about deep learning yet, right?

00:09:32 Although, you could argue that the feature integration can be extended to latent variable integration in deep learning, right?

00:09:41 So, the first one is very simple.

00:09:44 Feature integration, right?

00:09:46 You have different features and you just remember they are all meaningful to the application domain.

00:09:52 You just integrate them together, concatenate them, right?

00:09:56 But, what's the problem of doing that?

00:09:59 Is there a problem doing the feature integration?

00:10:06 It's straightforward, but do you see a problem?

00:10:19 What could be a problem?

00:10:25 Huh?

00:10:30 Well, just all kinds of features.

00:10:32 Like what I said earlier, for the protein function prediction, I just stack all the sequence, structure, features, network features.

00:10:41 I stack all of them together.

00:10:46 They are already featurized, right?

00:10:53 Right?

00:10:54 So, they are already all featurized.

00:10:57 Let's say each feature is already a vector and just concatenate them together.

00:11:04 Could there be any limitations?

00:11:08 Can you give me this look for the code?

00:11:12 All right.

00:11:16 The biggest challenge is in the downstream.

00:11:23 You can, of course, pursue the convenience of just concatenating all the features together.

00:11:31 But you put the burden on what?

00:11:40 What's downstream to the input features?

00:11:48 The machine learning models, isn't it?

00:11:51 Right?

00:11:53 So for all this heterogeneous data,

00:11:55 you are using a universal function approximator

00:11:58 with limited, theoretically universal function approximator,

00:12:02 with very limited number of layers and neurons

00:12:09 and representing the architectural biases, all that.

00:12:15 So you put a lot of burden on the model itself.

00:12:21 Not to mention that when you need to figure out

00:12:24 the importance of each feature,

00:12:27 these different features are not comparable to each other even.

00:12:31 Right?

00:12:32 Because they are in different units and scales and everything.

00:12:34 Of course, that you can still deal with

00:12:37 by just normalizing everything into these scores, for instance.

00:12:43 But still, you have additional issues.

00:12:46 For instance, your features could have,

00:12:48 you know, your sequence features can be highly correlated with each other.

00:12:52 Right?

00:12:53 And then the structure features can be correlated.

00:12:55 In other words, you have some inner structures of your feature space

00:13:00 that you know based on how you introduce it.

00:13:05 But the machine learning model does not know and, you know,

00:13:10 it assumes something else or does not know about it.

00:13:14 And it's very difficult for them to learn the very complicated IO relationships.

00:13:19 Okay?

00:13:20 Another kind of integration is using the kernel integration.

00:13:25 Again, the kernel is about similarity.

00:13:28 Okay?

00:13:29 So in different contexts, similarities may be measured differently.

00:13:33 What's the sequence similarity including local and global?

00:13:37 What's the structure similarity including local and global?

00:13:41 And so on.

00:13:42 So these different kernel matrices, right?

00:13:46 Again, you know, it's a matrix, right?

00:13:48 Whose ij component or element is about the similarity or dissimilarity between example i and example j.

00:13:57 So you have different kernel matrices, each corresponds to one feature, right?

00:14:05 And then you can combine them together using basic algebraic operations, right?

00:14:12 You could add them together, you could multiply them together, you can use exponentiation together, right?

00:14:21 There's no strong theoretical reasons about which one is more reasonable or optimal for the application.

00:14:29 There's none.

00:14:30 Okay?

00:14:31 The last one is classifier integration.

00:14:36 So you can train multiple classifier, each trained using some features, and then you can combine them together somehow, right?

00:14:48 For instance, you can use majority vote or something.

00:14:54 But the trick lies in how to combine, okay?

00:14:57 That there's some science we can talk about.

00:15:00 How do you combine?

00:15:01 Okay?

00:15:02 So this is actually something I want to emphasize in today's lecture, so-called ensemble methods.

00:15:08 Okay?

00:15:09 It's about how do you combine these classifiers or regressors, okay?

00:15:15 Any questions so far?

00:15:18 Again, I'm not saying that ensemble methods are better than kernel integration or feature integration, okay?

00:15:23 Again, there are tons of methods nowadays, including our own, just concatenating the latent variables, right?

00:15:31 In other words, the individual features were encoded by their respective encoders, and then in the latent space they are concatenated together.

00:15:40 So in that kind of approach, a lot of people are doing that still, okay?

00:15:46 In the deep learning world.

00:15:48 So I want to talk about the ensemble methods, right?

00:15:53 Because deep learning or not, you could use them, right?

00:15:56 The general idea is the following.

00:15:59 So you have a, again, you know, we are talking about how do you combine the machine learning models, okay?

00:16:08 It's not just the one way to integrate heterogeneous data, it's also one way to improve your accuracy.

00:16:15 Because one criteria to choose ways to integrate data is about its accuracy, okay?

00:16:22 So we start with this training data, and then we split them into, we draw samples into multiple data sets, okay?

00:16:34 S1, S2, SN, and you could allow, you know, these multiple data sets, all right?

00:16:42 And you could allow replacement when you are drawing these samples, okay?

00:16:51 And based on each training data sets or subsets, we can train multiple classifier, right?

00:16:59 And then combine them together.

00:17:01 So far, I still haven't told you how to combine, but I already introduced the one concept, right here.

00:17:07 I'm going to name them later, okay?

00:17:10 But can someone tell me what am I doing here?

00:17:13 So, when I am splitting the training data into multiple data sets, what's the concept or terminology is being involved?

00:17:23 Batch.

00:17:24 Pardon me?

00:17:25 Batch.

00:17:26 Batch what?

00:17:27 Different batch training.

00:17:29 Different batches of training, yes, that's used in the deep learning, yes.

00:17:36 Cross-contination.

00:17:38 I see.

00:17:45 I was thinking about bootstrapping, but I think your answers are also fine.

00:17:49 If you're thinking about multiple batches in the deep learning, that's also fine.

00:17:55 But the multiple batches in deep learning are not used in this way.

00:18:01 Yeah.

00:18:02 Okay.

00:18:03 So, multiple batches of, their classifiers are not integrated at the end.

00:18:08 Okay.

00:18:09 It's more for big data issue, right?

00:18:12 Right?

00:18:13 So that you can use batches and then calculate stochastic gradients, right?

00:18:18 So, these ensemble approaches, I want to talk about bagging, boosting, and the random forest.

00:18:27 What's behind the random forest, okay?

00:18:29 Again, for tabular data, in the Kaggle challenges, for tabular data, random forest is still not surprisingly one of the best performing methods for tabular data.

00:18:45 Okay?

00:18:46 And you will see why.

00:18:48 Okay.

00:18:49 So, bagging.

00:18:50 Okay?

00:18:51 So, I was saying that, what was the concept behind, you know, splitting a dataset and sampling with replacement?

00:19:00 The keyword I was thinking about is bootstrapping, okay?

00:19:04 So, by bootstrapping, what I mean is, again, starting with the dataset S at iteration I, I'm going to draw a random sample SI with replacement.

00:19:15 Okay?

00:19:16 So, in other words, I allow repeats from S, okay?

00:19:19 And then I can train a predictor CI from each SI, okay?

00:19:24 And, yes, so, do you allow repeats when you construct batches?

00:19:35 I don't think so.

00:19:36 Is there a particular requirement about whether this replacement is allowed or not when you are?

00:19:44 I think it's just a random subset, is it?

00:19:47 I see, okay.

00:19:48 I don't know whether there's a deep reason of allowing replacement there or not.

00:19:53 Okay.

00:19:54 So, I mean, in the deep learning context.

00:19:57 So, in this case, we first do bootstrapping, okay?

00:20:02 They don't replace.

00:20:04 Huh?

00:20:05 No, they don't replace the data.

00:20:08 They don't replace the data.

00:20:10 They don't replace as long as the epoch is finished and all the data are covered?

00:20:27 Yeah.

00:20:28 Okay.

00:20:29 We're asking whether within each batch you could allow repeats.

00:20:33 Usually, we don't recommend doing that.

00:20:35 No, that's not definitely why would we do that?

00:20:40 I don't know why either.

00:20:41 It's like taking some more weight is to one example and less weight is to the remaining examples.

00:20:49 So, it's like that.

00:20:50 So, basically, we are multiplying the loss for that example by two and, like, the loss for the remaining examples would be just multiplied by one.

00:21:01 I don't see a deep reason about why you have to do it one way or another.

00:21:07 But I'd like to say that bootstrapping in this way, allowing replacement, was introduced in the early days when data are limited.

00:21:16 So, in those cases, you know, this is something that's more desired.

00:21:24 But when you enter the deep learning era, you really have big data for your supervised machine learning.

00:21:32 In those cases, it doesn't really matter whether you allow it or not, but you really, you know, you don't see this need of replacement.

00:21:41 That's one thing I think I can think of.

00:21:44 Okay.

00:21:45 Okay.

00:21:46 So, after bootstrapping, you can do the aggregation, right?

00:21:51 Aggregating.

00:21:52 So, bagging is actually a coined word using bootstrap and aggregating together, right?

00:21:59 Bag, right?

00:22:00 So, aggregating says that this bagged classifier can be based on the majority vote for the classifiers or can be the average value for regressions.

00:22:11 Okay.

00:22:12 So, the question is the following.

00:22:15 Why bagging works, right?

00:22:18 So, we haven't talked about bias variance straight off.

00:22:21 So, you don't know that, but let me give it as a conclusion.

00:22:29 First, that is, the generalization error is equal to the squared bias plus variance.

00:22:36 So, bagging works by reducing the variance.

00:22:41 Okay.

00:22:42 You know that the variance of the average, right?

00:22:45 If you have iid, xi's, right?

00:22:48 Each xi is of standard deviation sigma, right?

00:22:53 Okay.

00:22:54 Then, if you take the average of n xi's, what's the standard deviation of the average of n xi's?

00:23:12 The variance is sigma squared divided by n.

00:23:20 So, it's decreasing, right?

00:23:23 That's why when you are doing experiments in high school, usually they ask you to measure three times and take the average.

00:23:30 Because the average is unbiased estimate of the true mean, right?

00:23:37 And its standard deviation is getting smaller and smaller.

00:23:41 Meaning that it's unbiased estimation of the true mean and its deviation from the true mean is getting smaller and is expected to be smaller and smaller as you average more and more times, right?

00:23:54 Because it's divided by n.

00:23:56 So, the variance is reduced by voting or averaging, right?

00:24:02 And then, the more predictor, bigger n, the better.

00:24:06 But if you don't have enough data, see, you are doing bootstrapping.

00:24:09 So, this probably also echoes my conjecture that you probably allow such replacement, especially in the early days when you have the data issue, right?

00:24:18 Because you want bigger data, but you don't have them.

00:24:21 So, you could construct these bootstrapping strategies.

00:24:27 So, something I want to mention briefly, for instance, the boosting strategy, right?

00:24:34 So, the boosting strategy is that, you know, you start with the training example, right?

00:24:40 And you have the labels, right?

00:24:43 Again, for this t equal to 1 to capital T, right?

00:24:47 And then, you are finding these distributions on these m subsets.

00:24:55 And then, you can construct classifiers.

00:24:58 What boosting is saying is that you can combine weak classifiers together into strong ones.

00:25:07 Okay, I have an illustration for that, okay?

00:25:10 For instance, this is what Ada Boosty is saying.

00:25:13 This is a two-dimensional space with plus and minuses, okay?

00:25:18 And we are supposed to find a classifier.

00:25:21 So, apparently, a linear classifier isn't going to be working fantastically well, right?

00:25:28 Due to the distribution, but let's start with a linear classifier, right?

00:25:33 Predicting everything on the left to be positive, everything on the right to predict everything on the left.

00:25:36 Right?

00:25:37 To be negative, right?

00:25:38 So, it's a weak classifier.

00:25:40 And then, you could have another weak classifier, right?

00:25:44 Right?

00:25:45 Each of them can be measured by the sensitivity, specificity, right?

00:25:49 And then, you could have another one, right?

00:25:51 Then, you could combine these three together with some weights.

00:25:55 Again, this is a very simple linear weight.

00:25:58 Okay?

00:25:59 You are now looking at individual classifiers reported sign on that sample, but you are using the weighted sum.

00:26:07 And then, you end up finding that, huh, actually, I could have the perfect classification in this case.

00:26:13 Okay?

00:26:14 So, of course, this is just some ideal illustration, but this tells you how you can boost the weak classifiers.

00:26:24 Okay?

00:26:25 I think there is such idiom in every country, every culture saying that, you know, if we can integrate the wisdom from many average people, we can get, you know, the genius idea, right?

00:26:37 Every country, every culture has such an idiom, isn't it?

00:26:41 There must be?

00:26:42 No?

00:26:43 Yeah.

00:26:44 You think so?

00:26:45 You think so?

00:26:46 Okay.

00:26:47 Right?

00:26:48 Right?

00:26:49 Is there any?

00:26:50 Something similar?

00:26:52 If we integrate or combine a lot of average people's ideas together, we may get some genius ideas.

00:27:02 Right?

00:27:03 Okay.

00:27:04 So this is Ada boosting.

00:27:08 I want to talk a little bit more about the random forest because this is, first of all, performing very well for tabular data, again, on Kaggle.

00:27:19 Okay?

00:27:20 Tabular data is still something that hasn't found a lot of success in the latest developments, largely because of this lack of storage.

00:27:33 Okay?

00:27:34 Let's look at what random forest is doing.

00:27:37 So the random forest, right?

00:27:40 So it's, what is a forest?

00:27:43 A forest is a collection of trees.

00:27:47 Right?

00:27:48 What is a tree here?

00:27:50 The tree here is a decision tree.

00:27:53 Right?

00:27:54 For instance, if feature one is bigger than three and the feature two is less than two, then I think this sample is belonging to the plus class.

00:28:05 This is one decision rule.

00:28:08 Right?

00:28:09 So if we, you know, think about a random forest at the root, maybe it's, you know, in this case, you can think of it as no rules at the beginning.

00:28:20 And then when you make this branching, you're asking whether feature one is bigger than something or smaller than something.

00:28:27 And here you're asking whether feature two is bigger than or smaller than something like that.

00:28:32 Okay?

00:28:33 So you're, as you go from the root to the leaf, these rules are getting more and more specific because you aggregate all the decision making along the path.

00:28:43 Right?

00:28:44 But the way you split, the way you split these nodes or decision is not as simple as what I said.

00:28:54 You know, first feature one, next feature two.

00:28:56 That splitting is something tricky.

00:28:58 Okay?

00:28:59 What we're going to talk about.

00:29:01 So, again, random forest are combination of decision trees and then what it's doing at each split is feature bagging.

00:29:11 You already know what bagging is.

00:29:13 Bagging is bootstrapping and aggregating.

00:29:15 What it's doing here is bagging not in the sample space but in the feature space.

00:29:20 Because why?

00:29:21 At each split point, we need to figure out, you know, how to make a decision rule.

00:29:28 And the decision rules are based on the feature.

00:29:30 Right?

00:29:31 So, at each split, a random subset of the features are being used.

00:29:37 Okay?

00:29:38 So, at that branches.

00:29:39 Okay?

00:29:40 And the generalization error depends on the strengths of the individual trees as well as the correlation between them.

00:29:47 There are strong theoretical proofs about that.

00:29:52 And you can use a random selection of features, you know, and that's more favorable compared to the AdaBoost.

00:30:01 And that's all more robust with regard to noise.

00:30:05 Okay?

00:30:06 So, this is what's behind the random forest.

00:30:10 So, let's look at one example, right?

00:30:14 So, what's the random forest algorithm?

00:30:17 So, given a training set S for I equal to 1 to K, what we do is bagging.

00:30:23 Right?

00:30:24 Right?

00:30:25 So, for each I, we first build a subset S I, right?

00:30:30 By sampling with replacement from the training set.

00:30:33 Right?

00:30:34 And then we also do the feature, sorry, and then we learn a tree based on this subset of training example.

00:30:45 And at each node for this tree, we actually do the feature bagging.

00:30:50 So, we're choosing the best split from a random subset of F features.

00:30:55 Okay?

00:30:56 And then we keep growing the tree and so on.

00:30:59 Right?

00:31:00 So, and then at the end the predictions are made based on the majority vote from the set of K trees.

00:31:07 Right?

00:31:08 Each, you know, each tree is trained over a subset of the training set and it's also split based on its own random selection of features.

00:31:21 Okay?

00:31:22 So, if you look at the SK learn implementation of random forest, right?

00:31:29 There are more advanced versions of random forest, of course, but even for the basic version, right?

00:31:35 They do give you some parameters, not model parameters.

00:31:40 I mean the algorithm parameters, right?

00:31:42 Like how many, you know, branches was a maximum, was a maximum number of trees, was a maximum number of split.

00:31:49 So, those can be set at default values or set at the values you prefer.

00:31:55 So, you know, you may want to understand that what options are given there.

00:31:59 And more importantly, how those parameters affect the generalization error, right?

00:32:07 So, some might be linearly dependent, some might be quadratically dependent and so on.

00:32:11 So, those things you may want to pay attention to.

00:32:14 Okay?

00:32:15 Right?

00:32:16 For instance, does it matter more the number of trees or the depth of the trees?

00:32:20 Right?

00:32:21 Does it matter more the maximum number of splits you can have at each node or, you know, the size of the subset of the features you have?

00:32:32 So, all these things you can, of course, choose based on your empirical experiences, but it's very hard to form that experiences when you don't have, you know, infinite amount of time to test, right?

00:32:50 So, again, theories give you a guideline about what features might be, sorry, parameters can be important in what way.

00:32:59 So, I would, you know, if you're interested, I would strongly recommend you to look at the theoretical papers about what's the expression of the generalization error, okay?

00:33:08 And how do you measure the strength of individual trees and how do you measure the correlation between them?

00:33:15 And, let's see, okay, this is a random forest algorithm.

00:33:21 So, something I want to say about the random forest, okay, so, besides that it's still very successful for tabular data.

00:33:30 So, it's unexcelled in accuracy among the current very shallow learning algorithms, okay?

00:33:37 So, again, the estimate is unbiased.

00:33:42 The generalization, if you look at, you know, the expectation of the generalization error, right, it's unbiased, right?

00:33:51 As the forest building progresses, okay?

00:33:54 And it runs efficiently on large data sets, right?

00:33:58 And it gives estimates of what variables are important in what classification.

00:34:03 So, again, you could look at something that's returned from the sklearn implementation of random forest about the importance of each feature.

00:34:15 I'm sorry, each, yeah, what variable in this case, the input variable, what features are important.

00:34:21 So, that's very important as well.

00:34:23 That's part of the interpretability.

00:34:26 So, it has an effective way of estimating missing data and maintaining accuracy when the large portion of data are missing, okay?

00:34:36 So, but again, about the data imputation, there are different ways.

00:34:41 In the past, there are final projects investigating data imputation, whether you want to impute first, then make some machine learning, or you want to do them together.

00:34:52 So, you know, there are a lot of research into that.

00:34:55 And it also has ways to balance the error, especially when your data's class distribution is imbalanced.

00:35:08 Okay?

00:35:09 So, these are all the good part about random forest.

00:35:12 So, in general, in general, let's talk about the pros and cons of ensemble methods, not just the random forest, but ensemble methods in general.

00:35:25 Remember, we had three ways, right?

00:35:27 To integrate heterogeneous data, right?

00:35:31 The pros is that the prediction accuracy is going to be improved.

00:35:36 And we know why, especially in the case of ensemble methods, right?

00:35:41 They will reduce the variance part of the generalization error, right?

00:35:45 It can easily integrate various types of classifiers, right?

00:35:49 You don't worry about, you know, the data integration because it's now in the classifier integration, right?

00:35:56 But, of course, there's, you know, limitations because you're using different features independently, for instance, right?

00:36:05 And you don't consider their correlations.

00:36:07 But I think random forest more or less overcome that because it always choose a random subset of features, right?

00:36:14 There's not too much little, there's not too much parameter tuning as I said in the random forest.

00:36:20 Many of you don't even tune those parameters like the maximum number of trees, maximum tree depth, right?

00:36:28 The con is that sometimes the learned classifier is a little bit black box.

00:36:38 You may be wondering, what do you mean you just told me that random forest can give me the weight or importance of each feature or variable?

00:36:47 But what I mean black box here is the learned classifier itself, okay?

00:36:55 It could be very zigzagging decision boundaries like feature one is less than this, feature two is bigger than that, and so on.

00:37:03 And you combine each, sorry, and each such decision is already complex, and you combine them together with the majority vote.

00:37:12 It's not exactly telling you why.

00:37:17 So that is a little bit black box, okay?

00:37:21 I think nowadays a lot of this need is addressed through regularizing your model architectures, especially in deep learning.

00:37:32 So it's either, well, it's in your model architecture or slash end in your loss function.

00:37:41 There's some, you know, architectural bias and loss regularization for that.

00:37:47 So let's see what do we have here.

00:37:52 So what do we have learned today are two things.

00:37:55 One is how do you evaluate classifiers.

00:37:58 It's quick review.

00:38:00 We started with a contingency table.

00:38:02 We split the data set into four partitions, and then we can play with those four numbers, right?

00:38:10 And their sums and calculate various measures, including sensitivity, specificity, or recall, precision, and so on.

00:38:19 But we also learned that those, you know, fixed threshold or determined measures can be misleading sometimes because it's heavily tied to the data set distribution, right?

00:38:34 So in those cases, you may want to also look into something that's more like overall assessment that is moving the threshold all over and see what's, you know, the performance using, for instance, the rock curve, receiver operating characteristic, right?

00:38:54 We learned that this area under the curve actually has a deep theoretical origin that is measure the probabilities that you rank a random positive above a random negative.

00:39:07 So you know that in this case, a random classifier like a coin, fair coin, would have a rock curve value area under the rock curve to be 0.5, right?

00:39:20 When the data set is extremely imbalanced, the rock curve can be misleading, right?

00:39:26 You can easily get AUC a rock curve above 0.9 in those cases.

00:39:32 So you may want to also look into the precision recall curve in those cases, right?

00:39:37 And in those cases, the random performer would get a value which is equal to the minority rate, essentially.

00:39:47 And we also talked about how you can choose the hyperparameters using cross-validation.

00:39:53 But keep in mind, I'm not looking at the small sample regime, which I would strongly recommend you to learn from the pattern recognition course.

00:40:03 We also talked about data integration, which is needed in so many domains.

00:40:11 As I said, for now, I'm looking at very classical views that you can integrate these data at the feature level, at the kernel level, or at the model level, right?

00:40:24 But I also want you to keep in mind that in the deep learning world, these are data of different modalities, right?

00:40:32 There's text, there's graph, there's image, there's manifold, and so on, right?

00:40:37 So how do you integrate them together, right?

00:40:40 And in the model integration where we spend a lot of time, we talked about bagging, including bootstrapping and aggregating, as well as boosting.

00:40:53 And lastly, the random forest, which is using feature bagging.

00:40:58 We explained what they do, but more importantly, why they were useful for accuracy, right?

00:41:06 But of course, there are some limitations, okay?

00:41:09 So with this, I would conclude today's lecture.

00:41:13 [Uncertain recognition] Again, next lecture, we're going to talk about