Deep Neural Networks (DNNs) are shown vulnerable to various security threats.

IoT devices, where limitations in memory, processing power and energy are critical [Sze et al., 2017]. To address these issues, Muller and Indiveri [2015] and Courbariaux et al. [2015b] noted that

Generative Adversarial Networks (GAN) [1] are known for the forefront approach to generating high-fidelity images of diverse categories [2,3,4,5,6,7,8,9].

Deep active learning aims to reduce the annotation cost for the training of deep models, which is notoriously data-hungry. Until recently, deep active learning methods were ineffectual inthelow-budgetregime, where only asmall number ofexamples areannotated. Thesituation hasbeen alleviated byrecent advances inrepresentation andself-supervised learning, which impart thegeometry ofthe data representation with rich information about the points.

For the latter, PT -based methods adaptively extract a matching width-based slimmed-down sub-model from the global model as a local model according to each client's budget, thus averting the requirements for public data. As with FedAvg, PT -based methods require the server to periodically communicate with the clients. Existing PT -based methods focus on how to extract width-based sub-models from the global model. DFKD methods are promising, which transfer knowledge from the teacher model to another student model without any real data. Existing DFKD methods can be broadly classified into non-adversarial and adversarial training methods. They take the quality and/or diversity of the synthetic data as important objectives.