We introduce oriented non-radial basis function networks (ONRBF) as a generalization of Radial Basis Function networks (RBF)- wherein the Euclidean distance metric in the exponent of the Gaussian is re(cid:173) placed by a more general polynomial. This permits the definition of more general regions and in particular- hyper-ellipses with orienta(cid:173) tions. In the case of hyper-surface estimation this scheme requires a smaller number of hidden units and alleviates the "curse of dimen(cid:173) sionality" associated kernel type approximators.In the case of an im(cid:173) age, the hidden units correspond to features in the image and the parameters associated with each unit correspond to the rotation, scal(cid:173) ing and translation properties of that particular "feature". In the con(cid:173) text of the ONBF scheme, this means that an image can be represented by a small number of features. Since, transformation of an image by rotation, scaling and translation correspond to identical transformations of the individual features, the ONBF scheme can be used to considerable advantage for the purposes of image recognition and analysis.

The problem of color clustering is defined and shown to be a problem of assigning a large number (hundreds of thousands) of 3-vectors to a small number (256) of clusters. Finding those clusters in such a way that they best represent a full color image using only 256 distinct colors is a burdensome computational problem. In this paper, the problem is solved using "classical" techniques -- k-means clustering, vector quantization (which turns out to be the same thing in this application), competitive learning, and Kohonen self-organizing feature maps. Quality of the result is judged subjectively by how much the pseudo-color result resembles the true color image, by RMS quantization error, and by run time. The Kohonen map provides the best solution.

Genetic algorithms were used to select and create features and to select reference exemplar patterns for machine vision and speech pattern classi(cid:173) fication tasks. For a complex speech recognition task, genetic algorithms required no more computation time than traditional approaches to feature selection but reduced the number of input features required by a factor of five (from 153 to 33 features). On a difficult artificial machine-vision task, genetic algorithms were able to create new features (polynomial functions of the original features) which reduced classification error rates from 19% to almost 0%. Neural net and k nearest neighbor (KNN) classifiers were unable to provide such low error rates using only the original features. Ge(cid:173) netic algorithms were also used to reduce the number of reference exemplar patterns for a KNN classifier.

Learning is posed as a problem of function estimation, for which two princi(cid:173) ples of solution are considered: empirical risk minimization and structural risk minimization. These two principles are applied to two different state(cid:173) ments of the function estimation problem: global and local. Systematic improvements in prediction power are illustrated in application to zip-code recognition.

Batch gradient descent, w(t) -7JdE/dw(t), conver es to a minimum of quadratic form with a time constant no better than '4Amax/ Amin where Amin and Amax are the minimum and maximum eigenvalues of the Hessian matrix of E with respect to w. It was recently shown that adding a momentum term w(t) -7JdE/dw(t) Q' w(t - 1) improves this to VAmax/ Amin, although only in the batch case. Here we show that second(cid:173) order momentum, w(t) -7JdE/dw(t) Q' w(t -1) (3 w(t - 2), can lower this no further. We then regard gradient descent with momentum as a dynamic system and explore a non quadratic error surface, showing that saturation of the error accounts for a variety of effects observed in simulations and justifies some popular heuristics.

Stochastic gradient descent is a general algorithm which includes LMS, on-line backpropagation, and adaptive k-means clustering as special cases. The standard choices of the learning rate 1] (both adaptive and fixed func(cid:173) tions of time) often perform quite poorly. In contrast, our recently pro(cid:173) posed class of "search then converge" learning rate schedules (Darken and Moody, 1990) display the theoretically optimal asymptotic convergence rate and a superior ability to escape from poor local minima. However, the user is responsible for setting a key parameter. We propose here a new method(cid:173) ology for creating the first completely automatic adaptive learning rates which achieve the optimal rate of convergence.

We present an analysis of how the generalization performance (expected test set error) relates to the expected training set error for nonlinear learn(cid:173) ing systems, such as multilayer perceptrons and radial basis functions. The expectations () of training set and test set errors are taken over possible training sets e and training and test sets e' respec(cid:173) tively. The effective number of parameters Peff(,x) usually differs from the true number of model parameters P for nonlinear or regularized models; this theoretical conclusion is supported by Monte Carlo experiments. In addition to the surprising result that Peff(,x);/; p, we propose an estimate of (1) called the generalized prediction error (GPE) which generalizes well established estimates of prediction risk such as Akaike's F P E and AI C, Mallows Cp, and Barron's PSE to the nonlinear setting.!

Two projection based feedforward network learning methods for model(cid:173) free regression problems are studied and compared in this paper: one is the popular back-propagation learning (BPL); the other is the projection pursuit learning (PPL). In terms of learning efficiency, both methods have comparable training speed when based on a Gauss(cid:173) Newton optimization algorithm while the PPL is more parsimonious. In terms of learning robustness toward noise outliers, the BPL is more sensi(cid:173) tive to the outliers.

Several parallel analogue algorithms, based upon mean field theory (MFT) approximations to an underlying statistical mechanics formulation, and re(cid:173) quiring an externally prescribed annealing schedule, now exist for finding approximate solutions to difficult combinatorial optimisation problems. They have been applied to the Travelling Salesman Problem (TSP), as well as to various issues in computational vision and cluster analysis. I show here that any given MFT algorithm can be combined in a natural way with notions from the areas of constrained optimisation and adaptive simulated annealing to yield a single homogenous and efficient parallel re(cid:173) laxation technique, for which an externally prescribed annealing schedule is no longer required. The results of numerical simulations on 50-city and 100-city TSP problems are presented, which show that the ensuing algo(cid:173) rithms are typically an order of magnitude faster than the MFT algorithms alone, and which also show, on occasion, superior solutions as well.

Three methods for improving the performance of (gaussian) radial basis function (RBF) networks were tested on the NETtaik task. In RBF, a new example is classified by computing its Euclidean distance to a set of centers chosen by unsupervised methods. The application of supervised learning to learn a non-Euclidean distance metric was found to reduce the error rate of RBF networks, while supervised learning of each center's vari(cid:173) ance resulted in inferior performance. The best improvement in accuracy was achieved by networks called generalized radial basis function (GRBF) networks. In GRBF, the center locations are determined by supervised learning.