Diagonal Discriminant Analysis with Feature Selection for High Dimensional Data

Classification problems involving high dimensional data are extensive in many fields such as finance, marketing, and bioinformatics. Unique challenges with high dimensional datasets are numerous and well known, with many classifiers built under traditional low dimensional frameworks simply unable to be applied to such high dimensional data. Discriminant Analysis (DA) is one such example (Fisher, 1936). DA classifiers work by assuming the distribution of the features is strictly Gaussian at the class level, and assign a particular point to the class label which minimises the Mahalanobis (for linear discriminant analysis (LDA)) distance between that point and the mean of the multivariate normal corresponding to such class. Although extraordinary simple and easy to use in low dimensional settings, DA is well known to be unusable in high dimensions due to the maximum likelihood estimate of the corresponding covariance matrix being singular when the number of features is greater than that of the observations.