Industry
Relaxation Networks for Large Supervised Learning Problems
Alspector, Joshua, Allen, Robert B., Jayakumar, Anthony, Zeppenfeld, Torsten, Meir, Ronny
Feedback connections are required so that the teacher signal on the output neurons can modify weights during supervised learning. Relaxation methods are needed for learning static patterns with full-time feedback connections. Feedback network learning techniques have not achieved wide popularity because of the still greater computational efficiency of back-propagation. We show by simulation that relaxation networks of the kind we are implementing in VLSI are capable of learning large problems just like back-propagation networks. A microchip incorporates deterministic mean-field theory learning as well as stochastic Boltzmann learning. A multiple-chip electronic system implementing these networks will make high-speed parallel learning in them feasible in the future.
Asymptotic slowing down of the nearest-neighbor classifier
Snapp, Robert R., Psaltis, Demetri, Venkatesh, Santosh S.
Santosh S. Venkatesh Electrical Engineering University of Pennsylvania Philadelphia, PA 19104 If patterns are drawn from an n-dimensional feature space according to a probability distribution that obeys a weak smoothness criterion, we show that the probability that a random input pattern is misclassified by a nearest-neighbor classifier using M random reference patterns asymptotically satisfies a PM(error) "" Poo(error) M2/n' for sufficiently large values of M. Here, Poo(error) denotes the probability of error in the infinite sample limit, and is at most twice the error of a Bayes classifier. Although the value of the coefficient a depends upon the underlying probability distributions, the exponent of M is largely distribution free.We thus obtain a concise relation between a classifier's ability to generalize from a finite reference sample and the dimensionality of the feature space, as well as an analytic validation of Bellman's well known "curse of dimensionality." 1 INTRODUCTION One of the primary tasks assigned to neural networks is pattern classification.
A VLSI Neural Network for Color Constancy
Moore, Andrew W., Allman, John, Fox, Geoffrey, Goodman, Rodney
A system for color correction has been designed, built, and tested successfully; theessential components are three custom chips built using subthreshold analogCMOS VLSI. The system, based on Land's Retinex theory of color constancy, produces colors similar in many respects to those produced by the visual system. Resistive grids implemented in analog VLSI perform the smoothing operation central to the algorithm at video rates. With the electronic system, the strengths and weaknesses of the algorithm are explored.
Analog Neural Networks as Decoders
Erlanson, Ruth, Abu-Mostafa, Yaser
In turn, KWTA networks can be used as decoders of a class of nonlinear error-correcting codes. By interconnecting suchKWTA networks, we can construct decoders capable of decoding more powerful codes. We consider several families of interconnected KWTAnetworks, analyze their performance in terms of coding theory metrics, and consider the feasibility of embedding such networks in VLSI technologies.
Optimal Sampling of Natural Images: A Design Principle for the Visual System
Bialek, William, Ruderman, Daniel L., Zee, A.
One ofthe major theoretical issues in neural computation is to understand how this efficiency is reached given the constraints imposed by the biological hardware. Part of the problem [2] is simply to give an informative representation ofthe visual world using a limited number of neurons, each of which has a limited information capacity. The information capacity of the visual system is determined in part by the spatial transfer characteristics, or "receptive fields," of the individual cells. From a theoretical point of view we can ask if there exists an optimal choice for these receptive fields, a choice which maximizes the information transfer through the system given the hardware constraints. We show that this optimization problem has a simple formulation which allows us to use the intuition developed through the variational approach to quantum mechanics. In general our approach leads to receptive fields which are quite unlike those observed forcells in the visual cortex. In particular orientation selectivity is not a generic prediction. The optimal filters, however, depend on the statistical properties ofthe images we are trying to sample. Natural images have a symmetry - scale invariance [4] - which saves the theory: The optimal receptive fields for sampling of natural images are indeed orientation selective and bear a striking resemblance to observed receptive field characteristics in the mammalian visual cortex as well as the retinal ganglion of lower vertebrates.
Exploratory Feature Extraction in Speech Signals
A novel unsupervised neural network for dimensionality reduction which seeks directions emphasizing multimodality is presented, and its connection toexploratory projection pursuit methods is discussed. This leads to a new statistical insight to the synaptic modification equations governing learning in Bienenstock, Cooper, and Munro (BCM) neurons (1982). The importance of a dimensionality reduction principle based solely on distinguishing features, is demonstrated using a linguistically motivated phoneme recognition experiment, and compared with feature extraction using back-propagation network. 1 Introduction Due to the curse of dimensionality (Bellman, 1961) it is desirable to extract features froma high dimensional data space before attempting a classification. How to perform this feature extraction/dimensionality reduction is not that clear. A first simplification is to consider only features defined by linear (or semi-linear) projections ofhigh dimensional data.
Discrete Affine Wavelet Transforms For Anaylsis And Synthesis Of Feedfoward Neural Networks
Pati, Y. C., Krishnaprasad, P. S.
In this paper we show that discrete affine wavelet transforms can provide a tool for the analysis and synthesis of standard feedforward neural networks. Itis shown that wavelet frames for L2(IR) can be constructed based upon sigmoids. The spatia-spectral localization property of wavelets can be exploited in defining the topology and determining the weights of a feedforward network. Training a network constructed using the synthesis proceduredescribed here involves minimization of a convex cost functional andtherefore avoids pitfalls inherent in standard backpropagation algorithms. Extension of these methods to L2(IRN) is also discussed. 1 INTRODUCTION Feedforward type neural network models constructed from empirical data have been found to display significant predictive power [6]. Mathematical justification in support ofsuch predictive power may be drawn from various density and approximation theorems [1, 2, 5].
Rapidly Adapting Artificial Neural Networks for Autonomous Navigation
Dean A. Pomerleau School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Abstract The ALVINN (Autonomous Land Vehicle In a Neural Network) project addresses the problem of training artificial neural networks in real time to perform difficult perception tasks. ALVINN,is a back-propagation network that uses inputs from a video camera and an imaging laser rangefinder to drive the CMU Navlab, a modified Chevy van. This paper describes training techniques which allow ALVINN to learn in under 5 minutes to autonomously control the Navlab by watching a human driver's response to new situations. Using these techniques, ALVINN has been trained to drive in a variety of circumstances including single-lane paved and unpaved roads, multilane lined and unlined roads, and obstacle-ridden on-and off-road environments, at speeds of up to 20 miles per hour. 1 INTRODUCTION Previous trainable connectionist perception systems have often ignored important aspects of the form and content of available sensor data. Because of the assumed impracticality of training networks to perform realistic high level perception tasks, connectionist researchers have frequently restricted their task domains to either toy problems (e.g. the TC identification problem [11] [6]) or fixed low level operations (e.g.