A Study on Variants of Conventional, Fuzzy, and Nullspace-Based Independence Criteria for Improving Supervised and Unsupervised Learning

-- Unsupervised and supervised learning methods conventionally use kernels to capture nonlinearities inherent in data structure. However experts have to ensure their proposed nonlinearity maximizes variability and capture inherent diversity of data. We revie wed all independenc e criteria to design unsupervised learners. Then we proposed 3 independence criteria and used them to design unsupervised and supervised dimensionality reduction methods. We evaluated contrast, accuracy and interpretability of these meth ods in both linear and neural nonlinear settings. The results show that the methods have outperformed the baseline (tSNE, PCA, regularized LDA, VAE with (un)supervised learner and layer sharing) and opened a new line of interpretable machine learning (ML) for the researchers. Small amount of research is conducted on the role and nature of statistical independence for Machine Learning (ML). Independency criteria are mainly used in the context of Independent Component Analysis (ICA). However learning more about capability of them, gives a wide variety of tools for processing and interpreting supervised and unsupervised learning. As uncorrelatedness is a specific type of independence (linear independence), most of PCA - based approaches gets summariz ed into a special case of independenc y . Another insight about independenc e is the mechanism of Linear Discriminant Analysis (LDA) [15], Independent Component Analysis ( ICA) [1], and Variational Autoencoder ( VAE) [13] based on independency criteria. LDA seeks for a linear projection with least between - class and highest within - class linear dependence. ICA seeks for an unmixing matrix with least statistical dependency between projected components. Finally, VAE seeks for a nonlinear projection to mixtures with minimum correlation (linear independency), minimum mean, and agreed variance. Yet, despite proposing many variations of Kernel PCA [ 1, 19 ] (least between sample dependency criterion), there is no publication in liter ature with neural version of PCA and LDA.