Probing the effects of broken symmetries in machine learning