Can We Do Deep Learning Without Multiplications?

A neural network is built around simple linear equations like Y WX B, which contain something called as weights W. These weights get multiplied with the input X and thus plays a crucial in how the model predicts. Most of the computations in deep neural networks are multiplications between float-valued weights and float-valued activations during the forward inference. The prediction scores can even go downhill if a wrong weight gets updated and as the network gets deeper i.e addition of more layers and columns of connected nodes, the error gets magnified and the results miss the target. To make models lighter while not keeping the efficiency intact, many solutions have been developed, and one such solution is neural compression.