Field-programmable gate arrays (FPGAs) are widely used to implement deep learning inference. Standard deep neural network inference involves the computation of interleaved linear maps and nonlinear activation functions. Prior work for ultra-low latency implementations has hardcoded the combination of linear maps and nonlinear activations inside FPGA lookup tables (LUTs). Our work is motivated by the idea that the LUTs in an FPGA can be used to implement a much greater variety of functions than this. In this paper, we propose a novel approach to training neural networks for FPGA deployment using multivariate polynomials as the basic building block. Our method takes advantage of the flexibility offered by the soft logic, hiding the polynomial evaluation inside the LUTs with minimal overhead. We show that by using polynomial building blocks, we can achieve the same accuracy using considerably fewer layers of soft logic than by using linear functions, leading to significant latency and area improvements. We demonstrate the effectiveness of this approach in three tasks: network intrusion detection, jet identification at the CERN Large Hadron Collider, and handwritten digit recognition using the MNIST dataset.

K. Cheung, Senior Member, IEEE, and George A. Constantinides, Senior Member, IEEE Abstract--Research has shown that deep neural networks contain significant redundancy, and thus that high classification accuracy can be achieved even when weights and activations are quantised down to binary values. Network binarisation on FPGAs greatly increases area efficiency by replacing resource-hungry multipliers with lightweight XNOR gates. However, an FPGA's fundamental building block, the K -LUT, is capable of implementing far more than an XNOR: it can perform any K -input Boolean operation. Inspired by this observation, we propose LUTNet, an end-to-end hardware-software framework for the construction of area-efficient FPGA-based neural network accelerators using the native LUTs as inference operators. We describe the realisation of both unrolled and tiled LUTNet architectures, with the latter facilitating smaller, less power-hungry deployment over the former while sacrificing area and energy efficiency along with throughput. For both varieties, we demonstrate that the exploitation of LUT flexibility allows for far heavier pruning than possible in prior works, resulting in significant area savings while achieving comparable accuracy . Against the state-of-the-art binarised neural network implementation, we achieve up to twice the area efficiency for several standard network models when inferencing popular datasets. We also demonstrate that even greater energy efficiency improvements are obtainable. Index Terms --Deep neural network, hardware architecture, field-programmable gate array, lookup table.null 1 I NTRODUCTION AND M OTIVATION D URING inference, the most common--and expensive-- computational node in a deep neural network (DNN) performs a function of the form in (1), calculating a channel output y . Each weight w n is a constant determined during training, x a vector of N channel inputs and f an activation function such as the widely used rectified linear unit. In the extreme case where w { 1, 1} N --so-called binarised neural networks (BNNs)--the multiplications become cheap or free to implement. With weight inputs left variable, multipliers become XNOR gates. When networks are unrolled, weights are fixed, and so the XNOR gates can be further simplified into buffers and inverters, all of which are usually subsumed into the downstream adder logic. Also beneficial for BNNs is the ability to use a population count (popcount) for summation: an operation that consumes half the LUT s of the otherwise-throughput-optimal balanced adder tree [1]. In modern networks, N commonly reaches numbers in the thousands [2], [3]. T o tackle this, we propose the replacement of (1) with the specifically FPGA-inspired function (2), wherein the activation function is unchanged but each product is replaced with an arbitrary term-specific Boolean function g n: { 1, 1} K { 1, 1 } .

Research has shown that deep neural networks contain significant redundancy, and that high classification accuracies can be achieved even when weights and activations are quantised down to binary values. Network binarisation on FPGAs greatly increases area efficiency by replacing resource-hungry multipliers with lightweight XNOR gates. However, an FPGA's fundamental building block, the K-LUT, is capable of implementing far more than an XNOR: it can perform any K-input Boolean operation. Inspired by this observation, we propose LUTNet, an end-to-end hardware-software framework for the construction of area-efficient FPGA-based neural network accelerators using the native LUTs as inference operators. We demonstrate that the exploitation of LUT flexibility allows for far heavier pruning than possible in prior works, resulting in significant area savings while achieving comparable accuracy. Against the state-of-the-art binarised neural network implementation, we achieve twice the area efficiency for several standard network models when inferencing popular datasets. We also demonstrate that even greater energy efficiency improvements are obtainable.