Goto

Collaborating Authors

 Oceania



de7858e3e7f9f0f7b2c7bfdc86f6d928-Supplemental-Conference.pdf

Neural Information Processing Systems

Inourexperiments, we visualize images for the mini-ImageNet dataset, and thus we utilize the image generator from the work [10] asG, which is pre-trained on ImageNet, and we pre-trainf on the mini-ImageNet dataset.





Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks

Neural Information Processing Systems

Theperformance ofneural networksonhigh-dimensional datadistributions suggests that it may be possible to parameterize a representation of agiven highdimensional function with controllably small errors, potentially outperforming standard interpolation methods. We demonstrate, both theoretically and numerically, that this is indeed the case. We map the parameters of a neural network to a system of particles relaxing with an interaction potential determined by the lossfunction.




A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication

Neural Information Processing Systems

Algorithm Thei Requirinitialx0,i, 1: forj =0 ,1,2,..., 1do 2: Randomlymtraining 3: Compute 4: Update 5: if((j+ 1)p)=0 then 6: Compute 7: Quantize 8: Av 9: Update 10: end 11: end Inthe achie O(1/ p MK)con limited impair gradient 2-bit ratio 32/2 =(if We the communicate issho each parameters.


Y our representations are in the network: composable and parallel adaptation for large scale models

Neural Information Processing Systems

On the ViT -L/16 architecture, our experiments show that a single adapter, 1.3% of the full model, is able to reach full fine-tuning accuracy on average across 11 challenging downstream classification tasks. Compared with other forms of parameter-efficient adaptation, the isolated nature of the InCA adaptation is computationally desirable for large-scale models. For instance, we adapt ViT -G/14 (1.8B+ parameters) quickly with 20+ adapters in parallel on a single V100 GPU (76% GPU memory reduction) and exhaustively identify its