We are developing a phoneme based. In [Bourlard & Wellekens]. it was shown that MLPs were approximating Maximum a Posteriori (MAP) probabilities and could thus be embedded as an emission probability estimator in HMMs. It is shown here that word recognition performance for a simple discrete density HMM system appears to be somewhat better when MLP methods are used to estimate the emission probabilities.

Two approaches were explored which integrate neural net classifiers with Hidden Markov Model (HMM) speech recognizers. Both at(cid:173) tempt to improve speech pattern discrimination while retaining the temporal processing advantages of HMMs. One approach used neu(cid:173) ral nets to provide second-stage discrimination following an HMM recognizer. On a small vocabulary task, Radial Basis Function (RBF) and back-propagation neural nets reduced the error rate substantially (from 7.9% to 4.2% for the RBF classifier). In a larger vocabulary task, neural net classifiers did not reduce the error rate.

In recent years many researchers have investigated the use of Markov Random Fields (MRFs) for computer vision. They can be applied for example to reconstruct surfaces from sparse and noisy depth data coming from the output of a visual process, or to integrate early vision processes to label physical discontinuities. In this pa(cid:173) per we show that by applying mean field theory to those MRFs models a class of neural networks is obtained. Those networks can speed up the solution for the MRFs models. The method is not restricted to computer vision.

A rigorous analysis on the finite precision computational)Spects of neural network as a pattern classifier via a probabilistic approach is presented. Even though there exist negative results on the capa(cid:173) bility of perceptron, we show the following positive results: Given n pattern vectors each represented by en bits where e 1, that are uniformly distributed, with high probability the perceptron can perform all possible binary classifications of the patterns. More(cid:173) over, the resulting neural network requires a vanishingly small pro(cid:173) portion O(log n/n) of the memory that would be required for com(cid:173) plete storage of the patterns. Further, the perceptron algorithm takes O(n2) arithmetic operations with high probability, whereas other methods such as linear programming takes O(n3 .5) in the worst case. We also indicate some mathematical connections with VLSI circuit testing and the theory of random matrices.

One of the attractions of neural network approaches to pattern recognition is the use of a discrimination-based training method. We show that once we have modified the output layer of a multi(cid:173) layer perceptron to provide mathematically correct probability dis(cid:173) tributions, and replaced the usual squared error criterion with a probability-based score, the result is equivalent to Maximum Mu(cid:173) tual Information training, which has been used successfully to im(cid:173) prove the performance of hidden Markov models for speech recog(cid:173) nition. If the network is specially constructed to perform the recog(cid:173) nition computations of a given kind of stochastic model based clas(cid:173) sifier then we obtain a method for discrimination-based training of the parameters of the models. Examples include an HMM-based word discriminator, which we call an'Alphanet'.

In this paper, we prove that the vectors in the LVQ learning algorithm converge. We do this by showing that the learning algorithm performs stochastic approximation. Convergence is then obtained by identifying the appropriate conditions on the learning rate and on the underlying statistics of the classification problem. We also present a modification to the learning algorithm which we argue results in convergence of the LVQ error to the Bayesian optimal error as the appropriate parameters become large.

In this paper we examine in a general sense the application of Minimum Description Length (MDL) techniques to the problem of selecting a good classifier from a large set of candidate models or hypotheses. Pattern recognition algorithms differ from more conventional statistical modeling techniques in the sense that they typically choose from a very large number of candidate models to describe the available data. Hence, the problem of searching through this set of candidate models is frequently a formidable one, often approached in practice by the use of greedy algorithms. In this context, techniques which allow us to eliminate portions of the hypothesis space are of considerable interest. We will show in this paper that it is possible to use the intrinsic structure of the MDL formalism to eliminate large numbers of candidate models given only minimal information about the data.

We introduce a method for the efficient design of a Boltzmann machine (or a Hopfield net) that computes an arbitrary given Boolean function f . This method is based on an efficient simulation of acyclic circuits with threshold gates by Boltzmann machines. As a consequence we can show that various concrete Boolean functions f that are relevant for classification problems can be computed by scalable Boltzmann machines that are guaranteed to converge to their global maximum configuration with high probability after constantly many steps.

A particular form of neural network is described, which has terminals for acoustic patterns, class labels and speaker parameters. A method of training this network to "tune in" the speaker parameters to a particular speaker is outlined, based on a trick for converting a supervised network to an unsupervised mode. We describe experiments using this approach in isolated word recognition based on whole-word hidden Markov models. The results indicate an improvement over speaker-independent perfor(cid:173) mance and, for unlabelled data, a performance close to that achieved on labelled data.

Previous work has shown the ability of Multilayer Perceptrons (MLPs) to estimate emission probabilities for Hidden Markov Mod(cid:173) els (HMMs). The advantages of a speech recognition system incor(cid:173) porating both MLPs and HMMs are the best discrimination and the ability to incorporate multiple sources of evidence (features, temporal context) without restrictive assumptions of distributions or statistical independence. This paper presents results on the speaker-dependent portion of DARPA's English language Resource Management database. Results support the previously reported utility of MLP probability estimation for continuous speech recog(cid:173) nition. An additional approach we are pursuing is to use MLPs as nonlinear predictors for autoregressive HMMs.