Yoshua Bengio Team's Large-Scale Analysis Reveals the Benefits of Modularity and Sparsity for DNNs

Deep neural networks (DNNs) have drawn much inspiration from the human cognitive process, evidenced recently in their incorporation of modular structures and attention mechanisms. By representing knowledge in a modular manner and selecting relevant information via attention mechanisms, DNN models can develop meaningful inductive biases, boost their out-of-distribution generalization abilities, and manipulate concepts at higher levels of cognition. While modular architectures provide proven advantages for DNNs, there currently exists no rigorous quantitative assessment method for them due to the complexity and unknown nature of real-world data distributions. As such, it is unclear whether or to what extent the performance gains obtained by modular systems are actually attributable to good modular architecture design. In the new paper Is a Modular Architecture Enough, a research team from Mila and the Université de Montréal conducts a rigorous and thorough quantitative assessment of common modular architectures that reveals the benefits of modularity and sparsity for DNNs and the sub-optimality of existing end-to-end learned modular systems.