SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency

Extensive studies have shown that deep learning models are vulnerable to adversarial and natural noises, yet little is known about model robustness on noises caused by different system implementations. In this paper, we for the first time introduce SysNoise, a frequently occurred but often overlooked noise in the deep learning training-deployment cycle. In particular, SysNoise happens when the source training system switches to a disparate target system in deployments, where various tiny system mismatch adds up to a non-negligible difference. We first identify and classify SysNoise into three categories based on the inference stage; we then build a holistic benchmark to quantitatively measure the impact of SysNoise on 20+ models, comprehending image classification, object detection, instance segmentation and natural language processing tasks. Our extensive experiments revealed that SysNoise could bring certain impacts on model robustness across different tasks and common mitigations like data augmentation and adversarial training show limited effects on it. Together, our findings open a new research topic and we hope this work will raise research attention to deep learning deployment systems accounting for model performance. Based in handling multiple tasks (Krizhevsky et al., 2012; Simonyan on where SysNoise could happen, we classify it into three & Zisserman, 2014; He et al., 2016a; Devlin et al., different types. Pre-processing: Depends on the implementation 2018; Brown et al., 2020), yet they are vulnerable against of input data. Despite the progress devoted to noises made by decoding (JPEG2RGB) algorithms and different interpolation human-being or nature (e.g., adversarial noises (Goodfellow methods for image resize and crop. In practice, have different results when the upsampling operator is different.