Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System