memory requirement
LoRO: Real-Time on-Device Secure Inference for LLMs via TEE-Based Low Rank Obfuscation
While Large Language Models (LLMs) have gained remarkable success, they are consistently at risk of being stolen when deployed on untrusted edge devices. As a solution, TEE-based secure inference has been proposed to protect valuable model property. However, we identify a statistical vulnerability in existing protection methods, and furtherly compromise their security guarantees by proposed Model Stealing Attack with Prior. To eliminate this vulnerability, LoRO is presented in this paper, which leverages dense mask to completely obfuscate parameters. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to generate dense masks efficiently. The computing complexity in TEE is hence reduced by an exponential amount to achieve inference speed up, while providing robust model confidentiality.
Exact Gaussian Processes on a Million Data Points
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q. Weinberger, Andrew Gordon Wilson
Neural Information Processing Systems http://nips.cc/
RNNPool: EfficientNon-linearPoolingforRAM ConstrainedInference
RNNPool can be applied to any tensor structuredproblem,butwefocuson2Dimages for ease of exposition.
Reversible Recurrent Neural Networks
Matthew MacKay, Paul Vicol, Jimmy Ba, Roger B. Grosse
Neural Information Processing Systems http://nips.cc/
VeLoRA: MemoryEfficientTrainingusing Rank-1Sub-TokenProjections
Using a single projection vector, we then project these individual sub-tokens onto a one-dimensional subspace. Importantly, we notice that we can initialize this projection vector cheaply using first-order batch statistics andthen keepitfixedthroughout training. Wethen reconstruct the original tokens using the same vector during the backward pass.
Exact Gaussian Processes on a Million Data Points
Ke Wang, Geoff Pleiss, Jacob Gardner, Stephen Tyree, Kilian Q. Weinberger, Andrew Gordon Wilson
Neural Information Processing Systems http://nips.cc/
Blockwise Parallel Transformers for Large Context Models
Model sizes range from 1B to 70B.
A Appendix
Perplexity vs. FLOP count of MIM compared to left-to-right baselines across model sizes. To evaluate the effectiveness of "Meet in the Middle" (MIM) pre-training compared to left-to-right Perplexity vs. training time of MIM compared to left-to-right baselines across model sizes. Our largest models of size 2.7B parameters are trained using 128 A100 GPU with 80GB See Table 10 for the details of all the training runs. This paper presents "Meet in the Middle", a novel pretraining paradigm for language models that The proposed method's secondary benefits in the infilling task could also improve several NLP tasks, such as text summarization and question answering, leading to better usability and overall