Training Feedforward Neural Networks with Standard Logistic Activations is Feasible

Deep learning models are impactful in many real-world applications, and the transfer of this technology to society has created new emerging issues, like the need of model interpretability [14]. The General Data Protection Regulation approved in 2016 by the European parliament, which will be effective in 2018, is a concrete example of the need to provide human understandable justifications for decisions taken by automated data-processing systems [15]. Research could probably be inspired by old literature in neural networks to find better explanations about the dynamics of deep learning and provide more human interpretable solutions. An example of such process is found in standard logistic activation functions, that have been studied extensively in the past, but tend to be substituted by other activation functions in modern neural networks. To understand why this may be the case, it is important to recall the unique properties of the logistic function and therefore analyze the reasons why it has been introduced in neural networks. Firstly, the standard logistic function is biologically plausible.