Stochastic Deep Learning in Memristive Networks

Inspired by the computational efficiency of human brain in processing unstructured data, neural networks have been explored since 1940s for a wide variety of data analytics applications. The latest generation of Deep Neural networks (DNNs) have achieved impressive successes rivaling typical human performance, thanks to their ability to capture hidden features from unstructured data using multiple layers of neurons [1]. However, as the number of layers (depth) of the networks increase, DNN training becomes computationally intense and time consuming due to the physically separated execution and memory units in conventional von Neumann machines. This has motivated the exploration of non-von Neumann architectures with closely integrated processing units and local memory elements in dense cross bar arrays with memristive devices [2]. It has been recently proposed that DNNs can be implemented by 2D cross bar arrays of resistive processing units (RPUs) that can store multiple analog states and adjust its conductivity with simple voltage pulses [3]. These RPU devices when implemented in a cross bar array can accelerate DNN training if all the weights in the array can be updated in parallel.