Because the deep learning models for the classification task always have a normalization operation (e.g., Softmax) to make the final unconstrained According to the definition of the Consistency of Representations Complexity Measure Eq.(??), itiseasy toseethatwehaveaninfinite number oflocal minima where themeasuresSi andMi,j for arbitrary classesiandj are finite positivenumbers. The python libraries we use to implement our experiments are PyTorch1.7.1andPyG1.6.3. C.2 DetailsofBaselineMethods In this subsection, we detailed the network architectures and the baseline methods used in our experiments. ImageNet is a benchmark dataset used for ResNet-50, which contains 14,197,122 annotated images with 1000 classes. For GNN, we selected four real-world graph datasets: PubMed [11] is a paper citation network where nodes represent documents and edgesrepresentcitationlinks.

For the latter, PT -based methods adaptively extract a matching width-based slimmed-down sub-model from the global model as a local model according to each client's budget, thus averting the requirements for public data. As with FedAvg, PT -based methods require the server to periodically communicate with the clients. Existing PT -based methods focus on how to extract width-based sub-models from the global model. DFKD methods are promising, which transfer knowledge from the teacher model to another student model without any real data. Existing DFKD methods can be broadly classified into non-adversarial and adversarial training methods. They take the quality and/or diversity of the synthetic data as important objectives.

This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs). Second-order techniques, while enjoying higher convergence rates vs first-order counterparts, have cubic complexity with respect to either the model size and/or the training batch size.

This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs). Second-order techniques, while enjoying higher convergence rates vs first-order counterparts, have cubic complexity with respect to either the model size and/or the training batch size.

Recently, two families of self-supervised methods, contrastive learning and latent bootstrapping, exemplified by SimCLR and BYOL respectively,havemade significant progress.

Recent approaches include priors on the feature attribution of a deep neural network (DNN) into the training process to reduce the dependence on unwanted features. However, until now one needed to trade off high-quality attributions, satisfying desirable axioms, against the time required to compute them. This in turn either led to long training times or ineffective attribution priors.