On Single Source Robustness in Deep Fusion Models