Aligning future system design with the ever-increasing compute needs of large language models (LLMs) is undoubtedly an important problem in today's world. Here, we propose a general performance modeling methodology and workload analysis of distributed LLM training and inference through an analytical framework that accurately considers compute, memory sub-system, network, and various parallelization strategies (model parallel, data parallel, pipeline parallel, and sequence parallel). We validate our performance predictions with published data from literature and relevant industry vendors (e.g., NVIDIA). For distributed training, we investigate the memory footprint of LLMs for different activation re-computation methods, dissect the key factors behind the massive performance gain from A100 to B200 ($\sim$ 35x speed-up closely following NVIDIA's scaling trend), and further run a design space exploration at different technology nodes (12 nm to 1 nm) to study the impact of logic, memory, and network scaling on the performance. For inference, we analyze the compute versus memory boundedness of different operations at a matrix-multiply level for different GPU systems and further explore the impact of DRAM memory technology scaling on inference latency. Utilizing our modeling framework, we reveal the evolution of performance bottlenecks for both LLM training and inference with technology scaling, thus, providing insights to design future systems for LLM training and inference.

Computer systems are becoming increasingly heterogeneous with the emergence of new memory technologies and compute devices. GPUs alongside CPUs have become commonplace and CXL is poised to be a mainstay of cloud systems. The operating system is responsible for managing these hardware resources, requiring modification every time a new device is released. Years of research and development are sunk into tuning the OS for high performance with each new heterogeneous device. With the recent explosion in memory technologies and domain-specific accelerators, it would be beneficial to have an OS that could provide high performance for new devices without significant effort. We propose LLaMaS which can adapt to new devices easily. LLaMaS uses Large Language Models (LLMs) to extract the useful features of new devices from their textual description and uses these features to make operating system decisions at runtime. Adding support to LLaMaS for a new device is as simple as describing the system and new device properties in plaintext. LLaMaS reduces the burden on system administrators to enable easy integration of new devices into production systems. Preliminary evaluation using ChatGPT shows that LLMs are capable of extracting device features from text and make correct OS decisions based on those features.

HOD HASHARON, Israel – Jan. 3, 2023 – Weebit Nano Limited (ASX:WBT), a leading developer of next-generation memory technologies for the global semiconductor industry, has taped-out (released to manufacturing) demonstration chips integrating its embedded Resistive Random-Access Memory (ReRAM or RRAM) module in an advanced 22nm FD-SOI (fully depleted silicon on insulator) process technology. This is the first tape-out of Weebit ReRAM in 22nm, one of the industry's most common process nodes, and a geometry where embedded flash is not viable. Weebit worked with its development partners CEA-Leti and CEA-List to successfully scale its ReRAM technology down to 22nm. The teams designed a full IP memory module that integrates a multi-megabit ReRAM block targeting the 22nm FD-SOI process which is designed to deliver outstanding performance for connected and ultra-low power applications such as IoT and edge AI. As embedded flash is unable to scale below 28nm, new non-volatile memory (NVM) technology is needed for smaller process geometries.

The market for memory ICs has always been dynamic, but with edge computing, artificial intelligence (AI), 5G and autonomous driving all taking off, the demand for memory technology is both expanding and evolving. Add dramatic shifts in the world of work and business because of the ongoing pandemic, and the memory industry has more challenges to solve all at once than ever before. Our latest EE Times Special Project looks at how diverse memory technologies are advancing and what's driving their evolution. Contributing editor Gary Hilson has been on the memory beat for a long time and he helped develop this Special Project and wrote many of the stories. One of those takes a look at a set of memory technologies that are still described as "emerging," even though most of them have been around for years.

IMAGE: Like any recipe, an ideal memristive neuromorphic computing system requires a special blend of CMOS circuits and memristive devices, as well as spatial resources and temporal dynamics that must be... view more WASHINGTON, March 24, 2020 -- During the 1990s, Carver Mead and colleagues combined basic research in neuroscience with elegant analog circuit design in electronic engineering. This pioneering work on neuromorphic electronic circuits inspired researchers in Germany and Switzerland to explore the possibility of reproducing the physics of real neural circuits by using the physics of silicon. The field of "brain-mimicking" neuromorphic electronics shows great potential not only for basic research but also for commercial exploitation of always-on edge computing and "internet of things" applications. In Applied Physics Letters, from AIP Publishing, Elisabetta Chicca, from Bielefeld University, and Giacomo Indiveri, from the University of Zurich and ETH Zurich, present their work to understand how neural processing systems in biology carry out computation, as well as a recipe to reproduce these computing principles in mixed signal analog/digital electronics and novel materials. One of the most distinctive computational features of neural networks is learning, so Chicca and Indiveri are particularly interested in reproducing the adaptive and plastic properties of real synapses.

When Apple CEO Tim Cook introduced the iPhone X, he claimed it would "set the path for technology for the next decade." While it is too early to tell, the neural engine used for face recognition was the first of its kind. Today deep neural networks are a reality, and neuromorphic appears to be the only practical path to make continuing progress in AI. Facing data bandwidth constraints and ever-rising computational requirements, sensing and computing must reinvent themselves by mimicking neurobiological architectures, claimed a recently published report by Yole Développement (Lyon, France). In an interview with EE Times, Pierre Cambou, Principal Analyst for Imaging at Yole, explained that neuromorphic sensing and computing could solve most of AI's current issues while opening new application perspectives in the next decades.

On August 29, 2019 I put on a workshop on Emerging Memories and Artificial Intelligence at Stanford University put on by the Stanford Center for Magnetic Nanotechnology and Coughlin Associates. We had several interesting speakers talking about various types of artificial intelligence and the role that new non-volatile memories will play in both training AI models and implementing them in the field using inference engines. This piece will talk about some of the material presented at this workshop. Dr. Shan Wang, co-organizer of the event gave a introduction, talking about emerging non-volatile memories and in particular on Magnetic Random Access Memory (MRAM). He spoke about how various new memories work--in particular Resistive RAM (RRAM), Phase Change Memory (PCM), MRAM and Ferrroelectic RAM (FRAM).

On August 29, 2019 I put on a workshop on Emerging Memories and Artificial Intelligence at Stanford University put on by the Stanford Center for Magnetic Nanotechnology and Coughlin Associates. We had several interesting speakers talking about various types of artificial intelligence and the role that new non-volatile memories will play in both training AI models and implementing them in the field using inference engines. This piece will talk about some of the material presented at this workshop. Dr. Shan Wang, co-organizer of the event gave a introduction, talking about emerging non-volatile memories and in particular on Magnetic Random Access Memory (MRAM). He spoke about how various new memories work--in particular Resistive RAM (RRAM), Phase Change Memory (PCM), MRAM and Ferrroelectic RAM (FRAM).

On August 29, 2019 I put on a workshop on Emerging Memories and Artificial Intelligence at Stanford University put on by the Stanford Center for Magnetic Nanotechnology and Coughlin Associates. We had several interesting speakers talking about various types of artificial intelligence and the role that new non-volatile memories will play in both training AI models and implementing them in the field using inference engines. This piece will talk about some of the material presented at this workshop. Dr. Shan Wang, co-organizer of the event gave a introduction, talking about emerging non-volatile memories and in particular on Magnetic Random Access Memory (MRAM). He spoke about how various new memories work--in particular Resistive RAM (RRAM), Phase Change Memory (PCM), MRAM and Ferrroelectic RAM (FRAM).