We study the set of continuous functions that admit no spurious local optima (i.e.

Several additional assumptions were suggested to make the ICA problem identifiable.

Several additional assumptions were suggested to make the ICA problem identifiable.

A.2 Training Settings of T eacher We provide training settings of the teacher w.r.t. In practice, we do not optimize the student and the generator via the plain losses in Eq. 4 and Eq. 6, Number of steps for pretraining G, δ: the bound in Eqs. A.4 Generator Architectures In Table 8, we show different architectures of the generator w.r.t. ResNetBlockY are provided in Table 9. ConvBlockX(c This is because the "uncond" generator has learned to jump "sum" generator enables stable training of our model and gives the best accuracy and crossentropy The "cat" generator only yields good results at "uncond" generator does not encounter any problem with MAD to learn faster than the "cat" generator. An important question is "What is a reasonable upper bound

This paper studies the condition for absence of spurious optimality. In particular, the authors introduce'global functions' to define the set of continuous functions that admit no spurious local optima (in the sense of sets), and develop some corresponding definitions and propositions for an extending characterization of continuous functions that admit no spurious strict local optima. The authors also apply their theory to l1-norm minimization in tensor decomposition. Pros: In my opinion, the main contribution of this paper is to establish a general math result and apply it to study the absence of spurious optimality for a specific problem. I also find some mathematical discoveries on global functions interesting, which include: -- In section 2, the paper provides two examples to show that: (i).

Matrix sensing problems exhibit pervasive non-convexity, plaguing optimization with a proliferation of suboptimal spurious solutions. Avoiding convergence to these critical points poses a major challenge. This work provides new theoretical insights that help demystify the intricacies of the non-convex landscape. In this work, we prove that under certain conditions, critical points sufficiently distant from the ground truth matrix exhibit favorable geometry by being strict saddle points rather than troublesome local minima. Moreover, we introduce the notion of higher-order losses for the matrix sensing problem and show that the incorporation of such losses into the objective function amplifies the negative curvature around those distant critical points. This implies that increasing the complexity of the objective function via high-order losses accelerates the escape from such critical points and acts as a desirable alternative to increasing the complexity of the optimization problem via over-parametrization. By elucidating key characteristics of the non-convex optimization landscape, this work makes progress towards a comprehensive framework for tackling broader machine learning objectives plagued by non-convexity.