NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference

In this study, we introduce NeuralMatrix, a framework that transforms the computation of entire DNNs into linear matrix operations, effectively enabling their execution with one general-purpose matrix multiplication (GEMM) accelerator. By surmounting the constraints posed by the diverse computation types required by individual network models, this approach provides both generality, allowing a wide range of DNN models to be executed using a single GEMM accelerator and application-specific acceleration levels without extra special function units, which are validated through main stream DNNs and their variant models. In recent years, the development of various types of deep neural networks (DNNs) has found applications in a wide range of scenarios. As neural network architectures continue to expand in size and complexity, they pose substantial computational challenges, especially for resource-constrained platforms and budget-conscious organizations. Application-specific integrated circuits (ASICs) offer a promising solution for supporting DNNs on mobile and edge devices. For example, Bai et al. (2018) introduced a CNN accelerator design that incorporates a multiplier array, add tree, normalization, ReLU, and pooling units. Similarly, Thierry Tambe et al. (2021) proposed an edge transformer accelerator featuring processing units (with floating-point vector and accumulate) and dedicated function units for layer normalization, softmax, and other unique operators in each layer. ASIC-based accelerators are known for their efficient execution of specific DNN applications. However, their inherent specificity, including the type and number of computation units, can restrict their adaptability from one DNN to another.