In many learning problems, the domain scientist is often interested in discovering the groups of features that are redundant and are important for classification. Moreover, the features that belong to each group, and the important feature groups may vary per sample. But what do we mean by feature redundancy? In this paper, we formally define two types of redundancies using information theory: \textit{Representation} and \textit{Relevant redundancies}. We leverage these redundancies to design a formulation for instance-wise feature group discovery and reveal a theoretical guideline to help discover the appropriate number of groups. We approximate mutual information via a variational lower bound and learn the feature group and selector indicators with Gumbel-Softmax in optimizing our formulation.

In many learning problems, the domain scientist is often interested in discovering the groups of features that are redundant and are important for classification.

Additional Feedback: A few other minor comments: Other methods to compare? For gene expression experiments, I've found k-means clustering using R 2 on columns to be a surprisingly good baseline for global feature clustering (and it allows for getting groups that are anti-correlated). I've also seen methods for finding groups of variables with high multivariate mutual information / total correlation / redundancy. Those ideas could be interesting for comparison or for improving the representation redundancy part to allow for groups that are not always positively correlated. Attention-based learning methods are natural comparisons for things like Figure 1.

In many learning problems, the domain scientist is often interested in discovering the groups of features that are redundant and are important for classification. Moreover, the features that belong to each group, and the important feature groups may vary per sample. But what do we mean by feature redundancy? In this paper, we formally define two types of redundancies using information theory: \textit{Representation} and \textit{Relevant redundancies}. We leverage these redundancies to design a formulation for instance-wise feature group discovery and reveal a theoretical guideline to help discover the appropriate number of groups. We approximate mutual information via a variational lower bound and learn the feature group and selector indicators with Gumbel-Softmax in optimizing our formulation.