Joint visual and language modeling on large-scale datasets has recently shown good progress in multi-modal tasks when compared to single modal learning.
While current works account for the importance ofdepth for the expressivepower ofneural-networks, itremains an open question whether these benefits are exploited during a gradient-based optimization process.