Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data

Tabular datasets are widely used in scientific disciplines such as biology. While these disciplines have already adopted AI methods to enhance their findings and analysis, they mainly use tree-based methods due to their interpretability. At the same time, artificial neural networks have been shown to offer superior flexibility and depth for rich and complex non-tabular problems, but they are falling behind tree-based models for tabular data in terms of performance and interpretability. Although sparsity has been shown to improve the interpretability and performance of ANN models for complex non-tabular datasets, enforcing sparsity structurally and formatively for tabular data before training the model, remains an open question. To address this question, we establish a method that infuses sparsity in neural networks by utilising attention mechanisms to capture the features' importance in tabular datasets. They further permit the extraction of insights from these datasets and achieve better performance than post-hoc methods like SHAP. Although tabular data are widespread, they have been left behind by recent advances in artificial intelligence (1). Recent literature suggests that tree-based models (such as Random Forest or XGBoost (2)) outperform neural networks for tabular data (3). Despite this, there remains a strong interest in neural networks in this area (1). Research in artificial intelligence has moved toward foundation models, and a foundation model for tabular data is lacking to date (4). The main challenges for deep learning models for tabular data are competitiveness with tree-based models, extracting meaningful features and dealing with hetero-modals data (heterogenous features). A natural advantage of a neural network model is the ability to perform other tasks, with minimal training, via transfer learning or fine-tuning.