Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition

--Pedestrian Attribute Recognition (PAR) is an indispensable task in human-centered research and has made great progress in recent years with the development of deep neural networks. However, the potential vulnerability and anti-interference ability have still not been fully explored. T o bridge this gap, this paper proposes the first adversarial attack and defense framework for pedestrian attribute recognition. Specifically, we exploit both global-and patch-level attacks on the pedestrian images, based on the pre-trained CLIP-based PAR framework. It first divides the input pedestrian image into non-overlapping patches and embeds them into feature embeddings using a projection layer . Meanwhile, the attribute set is expanded into sentences using prompts and embedded into attribute features using a pre-trained CLIP text encoder . A multi-modal Transformer is adopted to fuse the obtained vision and text tokens, and a feed-forward network is utilized for attribute recognition. Based on the aforementioned PAR framework, we adopt the adversarial semantic and label-perturbation to generate the adversarial noise, termed ASL-PAR. We also design a semantic offset defense strategy to suppress the influence of adversarial attacks. Extensive experiments conducted on both digital domains (i.e., PET A, PA100K, MSP60K, RAPv2) and physical domains fully validated the effectiveness of our proposed adversarial attack and defense strategies for the pedestrian attribute recognition. EDESTRIAN attribute recognition (P AR) [1] is a basic and important task for human-centric perception, which targets describing the appearance of a given pedestrian image using a set of attributes like " gender (male, female), hair style, shoes, carrying things ", etc. In addition, it can also be treated as middle-level semantic features and help other human-related tasks, such as person re-identification [2], multi-object tracking [3], and pedestrian retrieval [4]. With the help of deep learning, pedestrian attribute recognition has also been developed rapidly.