In the square linear blind source separation problem, one must find a linear unmixing operator which can detangle the result Xi(t) of mixing n unknown independent sources 8i(t) through an unknown n x n mixing matrix A( t) of causal linear filters: Xi E j aij * 8 j . We cast the problem as one of maximum likelihood density estima(cid:173) tion, and in that framework introduce an algorithm that searches for independent components using both temporal and spatial cues. We call the resulting algorithm "Contextual ICA," after the (Bell and Sejnowski 1995) Infomax algorithm, which we show to be a special case of cICA. Because cICA can make use of the temporal structure of its input, it is able separate in a number of situations where standard methods cannot, including sources with low kur(cid:173) tosis, colored Gaussian sources, and sources which have Gaussian histograms. Consider a set of n indepent sources 81 (t), .

Multilayer architectures such as those used in Bayesian belief net(cid:173) works and Helmholtz machines provide a powerful framework for representing and learning higher order statistical relations among inputs. Because exact probability calculations with these mod(cid:173) els are often intractable, there is much interest in finding approxi(cid:173) mate algorithms. We present an algorithm that efficiently discovers higher order structure using EM and Gibbs sampling. The model can be interpreted as a stochastic recurrent network in which ambi(cid:173) guity in lower-level states is resolved through feedback from higher levels. We demonstrate the performance of the algorithm on bench(cid:173) mark problems.

The techniques of Bayesian inference have been applied with great success to many problems in neural computing including evaluation of regression functions, determination of error bars on predictions, and the treatment of hyper-parameters. However, the problem of model comparison is a much more challenging one for which current techniques have significant limitations. In this paper we show how an extended form of Markov chain Monte Carlo, called chaining, is able to provide effective estimates of the relative probabilities of different models. We present results from the robot arm problem and compare them with the corresponding results obtained using the standard Gaussian approximation framework. In a Bayesian treatment of statistical inference, our state of knowledge of the values of the parameters w in a model M is described in terms of a probability distribution function. Initially this is chosen to be some prior distribution p(wIM), which can be combined with a likelihood function p( Dlw, M) using Bayes' theorem to give a posterior distribution p(wID, M) in the form

A central theme of computational vision research has been the re(cid:173) alization that reliable estimation of local scene properties requires propagating measurements across the image. Many authors have therefore suggested solving vision problems using architectures of locally connected units updating their activity in parallel. Unfor(cid:173) tunately, the convergence of traditional relaxation methods on such architectures has proven to be excruciatingly slow and in general they do not guarantee that the stable point will be a global mini(cid:173) mum. In this paper we show that an architecture in which Bayesian Be(cid:173) liefs about image properties are propagated between neighboring units yields convergence times which are several orders of magni(cid:173) tude faster than traditional methods and avoids local minima. In particular our architecture is non-iterative in the sense of Marr [5]: at every time step, the local estimates at a given location are op(cid:173) timal given the information which has already been propagated to that location.

Our modifications of the self-organizing map (SOM) algorithm results in purely digital learning rules which perform non-parametric his(cid:173) togram density estimation. The non-parametric nature of the sep(cid:173) aration allows for source separation of non-linear mixtures. An anisotropic coupling is introduced into our SOM with the role of aligning the network locally with the independent component con(cid:173) tours. This approach provides an exact verification condition for source separation with no prior on the source distributions.

In this paper, the technique of stacking, previously only used for supervised learning, is applied to unsupervised learning. Specifi(cid:173) cally, it is used for non-parametric multivariate density estimation, to combine finite mixture model and kernel density estimators. Ex(cid:173) perimental results on both simulated data and real world data sets clearly demonstrate that stacked density estimation outperforms other strategies such as choosing the single best model based on cross-validation, combining with uniform weights, and even the sin(cid:173) gle best model chosen by "cheating" by looking at the data used for independent testing.

This paper reports about an application of Bayes' inferred neu(cid:173) ral network classifiers in the field of automatic sleep staging. The reason for using Bayesian learning for this task is two-fold. First, Bayesian inference is known to embody regularization automati(cid:173) cally. Second, a side effect of Bayesian learning leads to larger variance of network outputs in regions without training data. This results in well known moderation effects, which can be used to detect outliers.

Image intensity variations can result from several different object surface effects, including shading from 3-dimensional relief of the object, or paint on the surface itself. An essential problem in vision, which people solve naturally, is to attribute the proper physical cause, e.g. We ad(cid:173) dressed this problem with an approach combining psychophysical and Bayesian computational methods. We assessed human performance on a set of test images, and found that people made fairly consistent judgements of surface properties. Our computational model assigned simple prior probabilities to different relief or paint explanations for an image, and solved for the most probable interpretation in a Bayesian framework.

We derive a learning algorithm for inferring an overcomplete basis by viewing it as probabilistic model of the observed data. Over(cid:173) complete bases allow for better approximation of the underlying statistical density. Using a Laplacian prior on the basis coefficients removes redundancy and leads to representations that are sparse and are a nonlinear function of the data. This can be viewed as a generalization of the technique of independent component anal(cid:173) ysis and provides a method for blind source separation of fewer mixtures than sources. We demonstrate the utility of overcom(cid:173) plete representations on natural speech and show that compared to the traditional Fourier basis the inferred representations poten(cid:173) tially have much greater coding efficiency.

We present a computationally efficient algorithm for function ap(cid:173) proximation with piecewise linear sigmoidal nodes. A one hidden layer network is constructed one node at a time using the method of fitting the residual. The task of fitting individual nodes is accom(cid:173) plished using a new algorithm that searchs for the best fit by solving a sequence of Quadratic Programming problems. Unique characteristics of this algorithm include: finite step convergence, a simple stop(cid:173) ping criterion, a deterministic methodology for seeking "good" local minima, good scaling properties and a robust numerical implemen(cid:173) tation.