Feature Importance -- How's and Why's

In this article, we will be exploring various feature selection techniques that we need to be familiar with, in order to get the best performance out of your model. SelectKbest is a method provided by sklearn to rank features of a dataset by their "importance "with respect to the target variable. This "importance" is calculated using a score function which can be one of the following: All of the above-mentioned scoring functions are based on statistics. For instance, the f_regression function arranges the p_values of each of the variables in increasing order and picks the best K columns with the least p_value. Features with a p_value of less than 0.05 are considered "significant" and only these features should be used in the predictive model.