In this communication we present a new algorithm for solving Support Vector Classifiers (SVC) with large training data sets. The new algorithm is based on an Iterative Re-Weighted Least Squares procedure which is used to optimize the SVc. Moreover, a novel sample selection strategy for the working set is presented, which randomly chooses the working set among the training samples that do not fulfill the stopping criteria. The validity of both proposals, the optimization procedure and sample selection strategy, is shown by means of computer experiments using well-known data sets.

Modern classification applications necessitate supplementing the few available labeled examples with unlabeled examples to improve classi(cid:173) fication performance. We present a new tractable algorithm for exploit(cid:173) ing unlabeled examples in discriminative classification. This is achieved essentially by expanding the input vectors into longer feature vectors via both labeled and unlabeled examples. The resulting classification method can be interpreted as a discriminative kernel density estimate and is read(cid:173) ily trained via the EM algorithm, which in this case is both discriminative and achieves the optimal solution. We provide, in addition, a purely dis(cid:173) criminative formulation of the estimation problem by appealing to the maximum entropy framework.

Jensen's inequality is a powerful mathematical tool and one of the workhorses in statistical learning. Its applications therein include the EM algorithm, Bayesian estimation and Bayesian inference. Quite often (i.e. in discriminative learning) upper bounds are needed as well. We derive and prove an efficient analytic inequality that provides such variational upper bounds. This inequality holds for latent variable mixtures of exponential family distributions and thus spans a wide range of contemporary statis(cid:173) tical models.

The concept of averaging over classifiers is fundamental to the Bayesian analysis of learning. Based on this viewpoint, it has re(cid:173) cently been demonstrated for linear classifiers that the centre of mass of version space (the set of all classifiers consistent with the training set) - exhibits excel(cid:173) lent generalisation abilities. However, the billiard algorithm as pre(cid:173) sented in [4] is restricted to small sample size because it requires o (m 2) of memory and 0 (N . In this paper we present a method based on the simple perceptron learning algorithm which allows to overcome this algorithmic drawback. The method is al(cid:173) gorithmically simple and is easily extended to the multi-class case.

We investigate a new kernel-based classifier: the Kernel Fisher Discrim(cid:173) inant (KFD). A mathematical programming formulation based on the ob(cid:173) servation that KFD maximizes the average margin permits an interesting modification of the original KFD algorithm yielding the sparse KFD. We find that both, KFD and the proposed sparse KFD, can be understood in an unifying probabilistic context. Furthermore, we show connections to Support Vector Machines and Relevance Vector Machines. From this understanding, we are able to outline an interesting kernel-regression technique based upon the KFD algorithm.

To control the walking gaits of a four-legged robot we present a novel neuromorphic VLSI chip that coordinates the relative phasing of the robot's legs similar to how spinal Central Pattern Generators are believed to control vertebrate locomotion [3]. The chip controls the leg move(cid:173) ments by driving motors with time varying voltages which are the out(cid:173) puts of a small network of coupled oscillators. The characteristics of the chip's output voltages depend on a set of input parameters. The rela(cid:173) tionship between input parameters and output voltages can be computed analytically for an idealized system. In practice, however, this ideal re(cid:173) lationship is only approximately true due to transistor mismatch and off(cid:173) sets.

Nonlinear Support Vector Machines (SVMs) are investigated for visual sex classification with low resolution "thumbnail" faces (21- by-12 pixels) processed from 1,755 images from the FE RET face database. The performance of SVMs is shown to be superior to traditional pattern classifiers (Linear, Quadratic, Fisher Linear Dis(cid:173) criminant, Nearest-Neighbor) as well as more modern techniques such as Radial Basis Function (RBF) classifiers and large ensemble(cid:173) RBF networks. Furthermore, the SVM performance (3.4% error) is currently the best result reported in the open literature.

An on-line recursive algorithm for training support vector machines, one vector at a time, is presented. Adiabatic increments retain the Kuhn(cid:173) Tucker conditions on all previously seen training data, in a number of steps each computed analytically.

We present a novel method for clustering using the support vector ma(cid:173) chine approach. Data points are mapped to a high dimensional feature space, where support vectors are used to define a sphere enclosing them. The boundary of the sphere forms in data space a set of closed contours containing the data. Data points enclosed by each contour are defined as a cluster. As the width parameter of the Gaussian kernel is decreased, these contours fit the data more tightly and splitting of contours occurs.

Recent work has exploited boundedness of data in the unsupervised learning of new types of generative model. For nonnegative data it was recently shown that the maximum-entropy generative model is a Non(cid:173) negative Boltzmann Distribution not a Gaussian distribution, when the model is constrained to match the first and second order statistics of the data. Learning for practical sized problems is made difficult by the need to compute expectations under the model distribution. The computa(cid:173) tional cost of Markov chain Monte Carlo methods and low fidelity of naive mean field techniques has led to increasing interest in advanced mean field theories and variational methods. Here I present a second(cid:173) order mean-field approximation for the Nonnegative Boltzmann Machine model, obtained using a "high-temperature" expansion.