Likelihood Landscape and Local Minima Structures of Gaussian Mixture Models

Mixture models, as exemplified by the Gaussian mixture model (GMM), are widely used for approximating complex multi-modal distributions. They can also be viewed as a form of latent variable models that provide a flexible approach for statistical inference with heterogeneous data. To estimate the parameters of GMM, a standard approach is via the maximum likelihood principle. When the global optimum of the likelihood function can be computed, the statistical properties of the maximum likelihood estimator is relatively well studied, including its asymptotic consistency [25] and finite-sample error rates [5, 21, 14]. Much less understood are the computational challenges associated with estimating GMMs. The negative log-likelihood function of GMM is nonconvex and in general has multiple local minima. Standard iterative algorithms, such as Expectation-Maximization (EM) [9], are only guaranteed to converge to a local minimum [27, 16].