Injectivity capacity of ReLU gates

Over the last 15-20 years we have been witnessing a rapid development of machine learning (ML) and neural networks (NN) concepts. As the need for efficient processing and interpretation of large data sets is estimated to further grow in the years to come, many fundamental algorithmic and theoretical NN breakthroughs are to be expected. To be able to adequately address upcoming challenges an excellent understanding of the ultimate limits of the employed technologies is needed. We in this paper study a mathematical problem that is directly connected to a notion of network capacity which is an example of such a limit. Characterizing presence or absence of injectivity as a property of random functions is the mathematical problem of our interest here. The mere definition of the functional injectivity implies its critical role in studying inverse problems. Namely, well-or ill-posedness of these problems is in a direct correspondence with the associated injectivity. Recent utilization of neural networks in studying (nonlinear) inverse problems therefore critically relies on their injectivity properties (see, e.g., [6,11,15,16,19,31,36,38]). Consequently, injectivity as a purely mathematical object is in these contexts transformed into a practically rather important NN architectures feature.