The pervasive integration of Artificial Intelligence models into contemporary mobile computing is notable across numerous use cases, from virtual assistants to advanced image processing. Optimizing the mobile user experience involves minimal latency and high responsiveness from deployed AI models with challenges from execution strategies that fully leverage real time constraints to the exploitation of heterogeneous hardware architecture. In this paper, we research and propose the optimal execution configurations for AI models on an Android system, focusing on two critical tasks: object detection (YOLO family) and image classification (ResNet). These configurations evaluate various model quantization schemes and the utilization of on device accelerators, specifically the GPU and NPU. Our core objective is to empirically determine the combination that achieves the best trade-off between minimal accuracy degradation and maximal inference speed-up.

We introduce the Neural Power Unit (NPU).

When you purchase through links in our articles, we may earn a small commission. Intel Core Ultra & Windows Copilot: What local AI can (and can't) do Using AI locally while retaining full data control - we explain how this works and what you need to know about it. Microsoft has introduced a new device category with Copilot+. Only laptops with a dedicated Neural Processing Unit (NPU), at least 16 GB of RAM and a fast NVMe SSD fulfil the minimum requirements. Intel is addressing these requirements with the Core Ultra series, which combines classic CPU cores with GPU acceleration and a hardwired NPU.

We introduce the Neural Power Unit (NPU).

When you purchase through links in our articles, we may earn a small commission. More and more applications support the use of NPUs in modern AI notebooks. Since Intel integrated a dedicated Neural Processing Unit (NPU) into modern notebooks with the Core Ultra processors and AMD with the Ryzen AI series, the software landscape has visibly changed. Applications from various fields, such as image processing, video production, communication and document processing, are increasingly using this specialized hardware to execute AI functions locally, becoming faster and more energy-efficient. Frameworks help developers to program applications that also offer NPU support.

On-device running Large Language Models (LLMs) is nowadays a critical enabler towards preserving user privacy. We observe that the attention operator falls back from the special-purpose NPU to the general-purpose CPU/GPU because of quantization sensitivity in state-of-the-art frameworks. This fallback results in a degraded user experience and increased complexity in system scheduling. To this end, this paper presents shadowAttn, a system-algorithm codesigned sparse attention module with minimal reliance on CPU/GPU by only sparsely calculating the attention on a tiny portion of tokens. The key idea is to hide the overhead of estimating the important tokens with a NPU-based pilot compute. Further, shadowAttn proposes insightful techniques such as NPU compute graph bucketing, head-wise NPU-CPU/GPU pipeline and per-head fine-grained sparsity ratio to achieve high accuracy and efficiency. shadowAttn delivers the best performance with highly limited CPU/GPU resource; it requires much less CPU/GPU resource to deliver on-par performance of SoTA frameworks.

MSI has a new OLED gaming monitor. It also has a built-in neural processing unit (or NPU). If you're familiar with that term, you know what comes next: this OLED monitor has "AI" built into it. After reading the official promo for the MSI MPG 271QR QD-OLED X50 and an extended session of bouncing various four-letter words off the walls of my office, I have to admit that this isn't the worst way to jump on the "AI" bandwagon. The NPU is tied into a CMOS sensor (a very basic camera) and a presence detection system, which detects whether a real human is sitting in front of it.

I don't use a Windows Copilot PC as a daily driver, though I have several in my office. But there's one absolutely critical Copilot feature that forces me to swap out my current laptop, attach a Copilot PC to my docking station, and boot it up. Very few people have bought a Copilot PC in the last year. So these features, which are currently locked to Copilot PCs and their NPU, aren't well known: Windows Recall; Paint's Cocreator, Generative Erase, Object Select, and Sticker Generator; Click-to-Do; Photos' Super Resolution, Relight and Restyle Image; the intelligent search features within the Settings menu; Windows Studio Effects; and Live Captions. My editor assumed I would prefer the last feature, Live Captions, probably because it's both useful and cool.

A year ago, Microsoft hyped Copilot PCs as the next big thing. Twelve months later, it's hard not to see them as one of the tech industry's more significant flops. The question is whether they'll stay that way. Many Copilot PCs began shipping on June 18, 2024, about a month after Microsoft announced the program at the company's headquarters a month earlier. Acer, Asus, Dell, HP, Lenovo, Samsung, and Microsoft's own Surface division committed to shipping Copilot PCs, whose centerpiece was a processor with an embedded Neural Processing Unit -- the engine of AI -- capable of 40 trillion operations per second, or TOPS.

Should Microsoft and the PC industry have paid more attention to the GPU during the development of AI and Copilot PCs? After a year's time waiting for Copilot PCs (and their newfangled "Neural Processing Units" to take off, I can't help but wonder. Microsoft launched the Copilot PC initiative on May 20, 2024, and began shipping them on June 18. Since then, Microsoft has supported Copilot PCs with a handful of features, rolling them out first for PCs with the Qualcomm Snapdragon chips inside and then later for PCs powered by Intel Core Ultra Series 2 chips and the AMD Ryzen AI 300 processor. Qualcomm is essentially blameless, delivering a potent PC processor with most AI capabilities and long battery life.