PRISM: Distributed Inference for Foundation Models at Edge

--Foundation models (FMs) have achieved remarkable success across a wide range of applications, from image classification to natural langurage processing, but pose significant challenges for deployment at edge. This has sparked growing interest in developing practical and efficient strategies for bringing foundation models to edge environments. Additionally, we restructure the self-attention mechanism to eliminate redundant computations caused by per-device Key/V alue calculation in position-wise partitioning and design a partition-aware causal masking scheme tailored for autoregressive models. Our results demonstrate substantial reductions in communication overhead (up to 99.2% for BERT at compression rate CR = 128) and per-device computation (51.24% for BERT at the same setting), with only minor accuracy degradation. This method offers a scalable and practical solution for deploying foundation models in distributed resource-constrained environments. Index T erms --Distributed Inference, Edge Inference, Foundation model, Transformer, Causal mask, Large Language Models, and AIoT . HE introduction of the Transformer architecture has revolutionized the field of artificial intelligence, leading to groundbreaking advancements across natural language processing (NLP), computer vision, and multimodal learning. Foundation models (FMs) such as GPT [1], [2], LLaMA [3], [4], BERT [5] and ViT [6], DeepSeek's R1 [7], and V eo-3 have significantly transformed how people interact with AI, becoming integral to everyday applications, from document summarization and dialogue systems to image and video generation. However, the extraordinary capabilities of these models come at the cost of enormous model size.