Deep Learning
MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superposition
Finally, we provide mathematical bounds on the interference between superposition channels in MIMOFormer.
Birder: Communication-Efficient 1-bit Adaptive Optimizer for Practical Distributed DNN Training
Therefore, from a system-level perspective, the design ethos of a system-efficient communication-compression algorithm is that we should guarantee that the compression/decompression of the algorithm is computationally light and takes less time, and it should also be friendly to efficient collective communication primitives.
LoRA-GA: Low-Rank Adaptation with Gradient Approximation
Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters.
