Memory-based classification algorithms such as radial basis func(cid:173) tions or K-nearest neighbors typically rely on simple distances (Eu(cid:173) clidean, dot product ...), which are not particularly meaningful on pattern vectors. More complex, better suited distance measures are often expensive and rather ad-hoc (elastic matching, deformable templates). We propose a new distance measure which (a) can be made locally invariant to any set of transformations of the input and (b) can be computed efficiently. We tested the method on large handwritten character databases provided by the Post Office and the NIST. Using invariances with respect to translation, rota(cid:173) tion, scaling, shearing and line thickness, the method consistently outperformed all other systems tested on the same databases.

Typical methods for gradient descent in neural network learning involve calculation of derivatives based on a detailed knowledge of the network model. This requires extensive, time consuming calculations for each pat(cid:173) tern presentation and high precision that makes it difficult to implement in VLSI. We present here a perturbation technique that measures, not calculates, the gradient. Since the technique uses the actual network as a measuring device, errors in modeling neuron activation and synaptic weights do not cause errors in gradient descent. The method is parallel in nature and easy to implement in VLSI.

In the presence of outliers, the existing self-organizing rules for Principal Component Analysis (PCA) perform poorly. Using sta(cid:173) tistical physics techniques including the Gibbs distribution, binary decision fields and effective energies, we propose self-organizing PCA rules which are capable of resisting outliers while fulfilling various PCA-related tasks such as obtaining the first principal com(cid:173) ponent vector, the first k principal component vectors, and directly finding the subspace spanned by the first k vector principal com(cid:173) ponent vectors without solving for each vector individually. Com(cid:173) parative experiments have shown that the proposed robust rules improve the performances of the existing PCA algorithms signifi(cid:173) cantly when outliers are present.

We analyse the effects of analog noise on the synaptic arithmetic during MultiLayer Perceptron training, by expanding the cost func(cid:173) tion to include noise-mediated penalty terms. Predictions are made in the light of these calculations which suggest that fault tolerance, generalisation ability and learning trajectory should be improved by such noise-injection. Extensive simulation experiments on two distinct classification problems substantiate the claims. The re(cid:173) sults appear to be perfectly general for all training schemes where weights are adjusted incrementally, and have wide-ranging implica(cid:173) tions for all applications, particularly those involving "inaccurate" analog neural VLSI.

With a point matching distance measure which is invariant under translation, rotation and permutation, we learn 2-D point-set ob(cid:173) jects, by clustering noisy point-set images. Unlike traditional clus(cid:173) tering methods which use distance measures that operate on feature vectors - a representation common to most problem domains - this object-based clustering technique employs a distance measure spe(cid:173) cific to a type of object within a problem domain. Formulating the clustering problem as two nested objective functions, we derive optimization dynamics similar to the Expectation-Maximization algorithm used in mixture models.

Stochastic optimization algorithms typically use learning rate schedules that behave asymptotically as J.t(t) J.to/t. The ensem(cid:173) ble dynamics (Leen and Moody, 1993) for such algorithms provides an easy path to results on mean squared weight error and asymp(cid:173) totic normality. We apply this approach to stochastic gradient algorithms with momentum. We show that at late times, learning is governed by an effective learning rate J.tejJ J.to/(l - f3) where f3 is the momentum parameter. We describe the behavior of the asymptotic weight error and give conditions on J.tejJ that insure optimal convergence speed.

A neurocomputer was implemented using radial basis functions and a combination of analog and digital VLSI circuits. The hybrid system uses custom analog circuits for the input layer and a digital signal processing board for the hidden and output layers. The system combines the advantages of both analog and digital circuits. The analog circuits have been fabricated and tested, the system has been built, and several applications have been executed on the system. One application provides significantly better results for a remote sensing problem than have been previously obtained using conventional methods.

We describe a number of learning rules that can be used to train un(cid:173) supervised parallel feature extraction systems. The learning rules are derived using gradient ascent of a quality function. We con(cid:173) sider a number of quality functions that are rational functions of higher order moments of the extracted feature values. We show that one system learns the principle components of the correla(cid:173) tion matrix. Principal component analysis systems are usually not optimal feature extractors for classification.

Although recurrent neural nets have been moderately successful in learning to emulate finite-state machines (FSMs), the continu(cid:173) ous internal state dynamics of a neural net are not well matched to the discrete behavior of an FSM. We describe an architecture, called DOLCE, that allows discrete states to evolve in a net as learn(cid:173) ing progresses. DOLCE consists of a standard recurrent neural net trained by gradient descent and an adaptive clustering technique that quantizes the state space. DOLCE is based on the assumption that a finite set of discrete internal states is required for the task, and that the actual network state belongs to this set but has been corrupted by noise due to inaccuracy in the weights. DOLCE learns to recover the discrete state with maximum a posteriori probabil(cid:173) ity from the noisy state.

We present a method for learning, tracking, and recognizing human hand gestures recorded by a conventional CCD camera without any special gloves or other sensors. A view-based representation is used to model aspects of the hand relevant to the trained gestures, and is found using an unsupervised clustering technique. We use normalized correlation net(cid:173) works, with dynamic time warping in the temporal domain, as a distance function for unsupervised clustering. Views are computed separably for space and time dimensions; the distributed response of the combination of these units characterizes the input data with a low dimensional repre(cid:173) sentation. A supervised classification stage uses labeled outputs of the spatio-temporal units as training data.