Goto

Collaborating Authors

 Country




Breaking the Activation Function Bottleneck through Adaptive Parameterization

Neural Information Processing Systems

Adaptive parameterization is a means of increasing this flexibility and thereby increasing the model's capacity to learn non-linear patterns. We focus on the feed-forward layer, f(x):= φ(W x+b),for some activation functionφ: R 7 R. Define the pre-activation layer as a = A(x):= Wx+band denote byg(a):= φ(a)/athe activation effect ofφgivena, where divisioniselement-wise.


On Exact Computation with an Infinitely Wide Neural Net

Neural Information Processing Systems

Moreo randominitializationH( 0)conv deterministic H asthewidthNeur ker ( , ) (Equation (2)) evaluatedH(t)= H forallt, then (3) becomes du(t) dt = H (u(t) y). Suppose (z)= max ( 0,z), 1/ = poly ( 1/ ,log (n / )) and d1 = d2 = = dL = m with m poly ( 1/ , L,1/ 0,n,log ( 1/ )).




Diminishing Returns Shape Constraints for Interpretability and Regularization

Neural Information Processing Systems

Similarly, a model that predicts the time it will take a customer to grocery shop should decrease in the number of cashiers, but each addedcashierreduces average wait time by less. In both cases, we would like to be able to incorporate this prior knowledge by constraining the machine learned model's output to have a diminishing returns response to the size of the apartment or number of cashiers.