A Limitations and Future Work

We perform large-scale analysis on a variety of modular systems through simple rule-based data generating distributions. Through our analysis, we uncover several interesting insights into the regimes where modularity, sparsity and perfect specialization helps and how sub-optimal standard modular systems are in terms of collapse and specialization. While we provide an in-depth analysis of modular systems and the sub-optimalities they suffer from, we do realize that our experimentation is done from a purely synthetic perspective. It would be quite interesting to see how it extends and extrapolates to more real-world domains. To this end, we would point the reader to the list of future works outlined below to extend it to more complex, but still synthetic, domains and also to Appendix B which discusses in detail ways to leverage insights from this work to perform experimentation in real-world settings as well as considerations that should be kept in mind when designing the same. We believe that this is a first step towards better benchmarking and understanding of modular systems. However, there are still a number of important and interesting directions that have not been explored in this work. We discuss some of these important future directions here. One direction of exploration is to extend the settings considered here to noisy domains and investigate whether similar analyses still hold. In this work, we only considered simple soft-attention based activation decisions in end-to-end trained modular systems described in Section 4. An interesting future work involves benchmarking hard-attention based modular systems to explore whether they perform or specialize better, and also if the problem of collapse is exacerbated in such models. Since this is the first work that provides such an analysis in this field, we decided to stick to the simplest setting of infinite-data regime to limit the effects of overfitting. We believe that a useful future work would be to consider a low-data regime to see whether the inductive biases of modularity and sparsity lead to even better generalization when there is only little data to learn from. While we consider the simplest setting for each rule, one could consider more complex distributions where x is also conditional on the rule c and where the labeling function, denoted by y in Section 3, is a complex non-linear function instead of a simple linear function. Analysis on trends between modular and monolithic models with increasing complexity of rules would not only lead to better understanding of such systems but also bridge it closer to real-world settings. Modular systems are often shown to lead to better OoD generalization.