We consider the optimization problem associated with training simple ReLU neural networks of the form $\mathbf{x}\mapsto \sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the squared loss. We provide a computer-assisted proof that even if the input distribution is standard Gaussian, even if the dimension is unrestricted, and even if the target values are generated by such a network, with orthonormal parameter vectors, the problem can still have spurious local minima once $k\geq 6$. By a continuity argument, this implies that in high dimensions, \emph{nearly all} target networks of the relevant sizes lead to spurious local minima. Moreover, we conduct experiments which show that the probability of hitting such local minima is quite high, and increasing with the network size. On the positive side, mild over-parameterization appears to drastically reduce such local minima, indicating that an over-parameterization assumption is necessary to get a positive result in this setting.

The sparse representation classifier (SRC) has been utilized in various classification problems, which makes use of L1 minimization and is shown to work well for image recognition problems that satisfy a subspace assumption. In this paper we propose a new implementation of SRC via screening, establish its equivalence to the original SRC under regularity conditions, and prove its classification consistency under a latent subspace model. The results are demonstrated via simulations and real data experiments, where the new algorithm achieves comparable numerical performance but significantly faster.

In this work we introduce declarative statistics, a suite of declarative modelling tools for statistical analysis. Statistical constraints represent the key building block of declarative statistics. First, we introduce a range of relevant counting and matrix constraints and associated decompositions, some of which novel, that are instrumental in the design of statistical constraints. Second, we introduce a selection of novel statistical constraints and associated decompositions, which constitute a self-contained toolbox that can be used to tackle a wide range of problems typically encountered by statisticians. Finally, we deploy these statistical constraints to a wide range of application areas drawn from classical statistics and we contrast our framework against established practices.

If my model is not learning I have a better idea of what to address rather than blindly wasting time switching optimisers (or even frameworks).

In the first part in a series on Tidy Time Series Analysis, we'll use tidyquant to investigate CRAN downloads. You're probably thinking, "Why tidyquant?" Most people think of tidyquant as purely a financial package and rightfully so. However, because of its integration with xts, zoo and TTR, it's naturally suited for "tidy" time series analysis. In this post, we'll discuss the the "period apply" functions from the xts package, which make it easy to apply functions to time intervals in a "tidy" way using tq_transmute()!

Get accustom to Predictive Analytics as career option with practical knowledge on some of the techniques that are currently in demand, such as Hypothesis Testing, Linear Regression, Multiple Regression, Logistic Regression, Correlations, Chi-Square Test etc.

At this year's BlockShow Asia, Yangdong Deng, chief AI scientist of Blockchain startup Matrix, explained how inserting Artificial Intelligence (AI) into the Blockchain ecosystem would make it possible to use Bitcoin mining computational power for scientific innovation. According to Deng, the current computing power being used in Bitcoin mining operations is 8.23x10²² floating point operations per second (FLOPS for short), while the total computing power in the world is 1.2x10²³ FLOPS. According to these calculations, Bitcoin mining is consuming 17 percent of total global computing power, justifying the frequent accusations that Bitcoin mining is wasteful. Matrix is seeking to reinvent mining algorithms by including AI into the equation through a Bayesian mining system that utilizes a Markov chain Monte Carlo algorithm (MCMC). Because these computations function similarly to traditional mining functions, they work well for Bitcoin mining.

It is a practical, hands-on course, i.e. we will spend some time dealing with some of the theoretical concepts related to both statistical and machine learning regression analysis. However, majority of the course will focus on implementing different techniques on real data and interpret the results. After each video you will learn a new concept or technique which you may apply to your own projects.

Many machine learning tasks are interested in recognizing or identifying an individual instance within a large set of possible candidates. These problems are usually modeled as multi-class classification problems, with a large and possibly complex label set. Leading examples include detecting the speaker from his voice patterns (Togneri and Pullella, 2011), identifying the author from her written text (Stamatatos et al., 2014), or labeling the object category from its image (Duygulu et al., 2002, Deng et al., 2010, Oquab et al., 2014). In all these examples, the algorithm observes an input x, and uses the classifier function h to guess the label y from a large label set S. 1 There are multiple practical challenges in developing classifiers for large label sets. Collecting high quality training data is perhaps the main obstacle, as the costs scale with the number of classes. It can be affordable to first collect data for a small set of classes, even if the long-term goal is to generalize to a larger set. Furthermore, classifier development can be accelerated by training first on fewer classes, as each training cycle may require substantially less resources. Indeed, due to interest in how small-set performance generalizes to larger sets, such comparisons can found in the literature (Oquab et al., 2014, Griffin et al., 2007). A natural question is: how does changing the size of the label set affect the classification accuracy?

We consider a probit model without covariates, but the latent Gaussian variables having compound symmetry covariance structure with a single parameter characterizing the common correlation. We study the parameter estimation problem under such one-parameter probit models. As a surprise, we demonstrate that the likelihood function does not yield consistent estimates for the correlation. We then formally prove the parameter's nonestimability by deriving a non-vanishing minimax lower bound. This counter-intuitive phenomenon provides an interesting insight that one bit information of the latent Gaussian variables is not sufficient to consistently recover their correlation. On the other hand, we further show that trinary data generated from the Gaussian variables can consistently estimate the correlation with parametric convergence rate. Hence we reveal a phase transition phenomenon regarding the discretization of latent Gaussian variables while preserving the estimability of the correlation.