From acomputational point of view,this problem is equivalent to the quadratic assignment problem, and as such is an NP-hard problem (Burkard et al., 1998).

While ridge regression provides shrinkage for the regression coefficients, manyofthecoefficients remain smallbutnon-zero. Performing ridgeregression with the matrix sketch returned by our algorithm and a particular regularization parameter forces coefficients to zero and has a provable(1+) bound on the statisticalrisk.

However,in light of4 stochastic optimization, we argue that our random permutation does not seem problematic. For the imperfect ground truth experiments, SURE requires knownσnoisy, but our eSURE requires knownσgt7 (otherwise, they are not working). In Tables 2, 3, bothσnoisy and σgt are described in the second and third rows,8 respectively. However,your comment is correct for practical sense, so we did train one deep neural network with12 varyingσgt [1 10]andσnoisy [10.1 55]for blind color image denoising and tested onimages with afixed13 noise level (just like Table 1) as shown in the below table. ToReviewer2It is indeed a good idea to be more explicit in some explanations for easier understanding.

Learned representations are a central component in modern ML systems, serving a multitude of downstream tasks. When training such representations, it is often the case that computational and statistical constraints for each downstream task are unknown. In this context, rigid fixed-capacity representations can be either over or under-accommodating to the task at hand.

However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent andcanbedifficultinpractice.

Broadly,aneural network will be better at learning to execute a reasoning task (in terms of samplecomplexity) ifitsindividual components align wellwiththetargetalgorithm.

Generative Adversarial Networks (GANs) produce high-quality images but are challenging totrain.

Overall, we argue that stability enforcement (through dynamics and/or plasticity) seems reasonable given current29 neuroscientific theories (e.g., Zenke, Ganguli & Gerstner 2017, "The temporal paradox of Hebbian learning and30 homeostatic plasticity."). Lastly, R3's mention that the pre-activation variableal doesn't appear in the update can be readily explained.