Min-Max-Plus Neural Networks

Conventional artificial neural networks typically have a fixed nonlinear activation function that applies to all neurons. Introducing a trainable nonlinear part of the network usually further enhances fitting capability of the network. For example, the seminal work of He et al. [1] showed for the first time that human-level performance on ImageNet Classification (experimentally with an error rate of 5.1%) could be surpassed by the performance of a large scale deep neural network (experimentally with an error rate of 4.94%). A key ingredient of their work is to make an extension of the classical Rectified Linear Unit (ReLU) as the nonlinear activation function to Parametric Rectified Linear Unit (PReLU) in which the slope of the negative part of the input is learnable. In this paper, we propose a new framework of neural networks called Min-Max-Plus Neural Networks (MMP-NNs) whose nonlinear part is systematically complexified. The mathematical foundation of this model is called tropical mathematics [2] which is a fast developing area in mathematics and whose connections to neural networks have been established only very recently. A special feature of tropical mathematics is that the operations of usual multiplications and additions degenerate to operations of additions (called "tropical multiplications") and min/max operations (called "tropical additions") respectively. Consequently, the nonlinear part of an MMP-NN only involves additions and min/max operations. Since the nonlinear part of an MMP-NN is trainable, the overall fitting capability of the model is determined by the fitting capabilities of both the linear and nonlinear parts of the network.