There is a widespread misconception that the delta-rule is in some sense guaranteed to work on networks without hidden units. As previous authors have mentioned, there is no such guarantee for classification tasks. We will begin by presenting explicit counterexamples illustrating two different interesting ways in which the delta rule can fail. We go on to provide conditions which do guarantee that gradient descent will successfully train networks without hidden units to perform two-category classification tasks. We discuss the generalization of our ideas to networks with hidden units and to multicategory classification tasks.

In this article we consider two aspects of computation with neural networks. Firstly we consider the problem of the complexity of the network required to compute classes of specified (structured) functions. We give a brief overview of basic known complexity theorems for readers familiar with neural network models but less familiar with circuit complexity theories. We argue that there is considerable computational and physiological justification for the thesis that shallow circuits (Le., networks with relatively few layers) are computationally more efficient. We hence concentrate on structured (as opposed to random) problems that can be computed in shallow (constant depth) circuits with a relatively few number (polynomial) of elements, and demonstrate classes of structured problems that are amenable to such low cost solutions. We discuss an allied problem-the complexity of learning-and close with some open problems and a discussion of the observed limitations of the theoretical approach. We next turn to a rigourous classification of how much a network of given structure can do; i.e., the computational capacity of a given construct.

SYNCHRONIZATION IN NEURAL NETS Jacques J. Vidal University of California Los Angeles, Los Angeles, Ca. 90024 John Haggerty· ABSTRACT The paper presents an artificial neural network concept (the Synchronizable Oscillator Networks) where the instants of individual firings in the form of point processes constitute the only form of information transmitted between joining neurons. This type of communication contrasts with that which is assumed in most other models which typically are continuous or discrete value-passing networks. Limiting the messages received by each processing unit to time markers that signal the firing of other units presents significant implemen tation advantages. When interaction is present, the scheduled firings are advanced or delayed by the firing of neighboring neurons. Networks of such neurons become global oscillators which exhibit multiple synchronizing attractors.

Centric Models of the Orientation Map in Primary Visual Cortex William Baxter Department of Computer Science, S.U.N.Y. at Buffalo, NY 14620 Bruce Dow Department of Physiology, S.U.N.Y. at Buffalo, NY 14620 Abstract In the visual cortex of the monkey the horizontal organization of the preferred orientations of orientation-selective cells follows two opposing rules: 1) neighbors tend to have similar orientation preferences, and 2) many different orientations are observed in a local region. Several orientation models which satisfy these constraints are found to differ in the spacing and the topological index of their singularities. Using the rate of orientation change as a measure, the models are compared to published experimental results. Introduction It has been known for some years that there exist orientation-sensitive neurons in the visual cortex of cats and mOnkeysl,2. These cells react to highly specific patterns of light occurring in narrowly circumscribed regiOns of the visual field, i.e., the cell's receptive field. The best patterns for such cells are typically not diffuse levels of illumination, but elongated bars or edges oriented at specific angles.

ON TROPISTIC PROCESSING AND ITS APPLICATIONS Manuel F. Fernandez General Electric Advanced Technology Laboratories Syracuse, New York 13221 ABSTRACT The interaction of a set of tropisms is sufficient in many cases to explain the seemingly complex behavioral responses exhibited by varied classes of biological systems to combinations of stimuli. It can be shown that a straightforward generalization of the tropism phenomenon allows the efficient implementation of effective algorithms which appear to respond "intelligently" to changing environmental conditions. Examples of the utilization of tropistic processing techniques will be presented in this paper in applications entailing simulated behavior synthesis, path-planning, pattern analysis (clustering), and engineering design optimization. INTRODUCTION The goal of this paper is to present an intuitive overview of a general unsupervised procedure for addressing a variety of system control and cost minimization problems. This procedure is hased on the idea of utilizing "stimuli" produced by the environment in which the systems are designed to operate as basis for dynamically providing the necessary system parameter updates.

Many optimization models of neural networks need constraints to restrict the space of outputs to a subspace which satisfies external criteria. Optimizations using energy methods yield "forces" which act upon the state of the neural network. The penalty method, in which quadratic energy constraints are added to an existing optimization energy, has become popular recently, but is not guaranteed to satisfy the constraint conditions when there are other forces on the neural model or when there are multiple constraints. In this paper, we present the basic differential multiplier method (BDMM), which satisfies constraints exactly; we create forces which gradually apply the constraints over time, using "neurons" that estimate Lagrange multipliers. The basic differential multiplier method is a differential version of the method of multipliers from Numerical Analysis.

In biological systems, it relates to such issues as classical and operant conditioning, temporal coordination of sensorimotor systems and temporal reasoning. In artificial systems, it addresses such real-world tasks as robot control, speech recognition, dynamic image processing, moving target detection by sonars or radars, EEG diagnosis, and seismic signal processing.

ABSTRACT How does the connectivity of a neural network (number of synapses per neuron) relate to the complexity of the problems it can handle (measured by the entropy)? Switching theory would suggest no relation at all, since all Boolean functions can be implemented using a circuit with very low connectivity (e.g., using two-input NAND gates). However, for a network that learns a problem from examples using a local learning rule, we prove that the entropy of the problem becomes a lower bound for the connectivity of the network. INTRODUCTION The most distinguishing feature of neural networks is their ability to spontaneously learn the desired function from'training' samples, i.e., their ability to program themselves. Clearly, a given neural network cannot just learn any function, there must be some restrictions on which networks can learn which functions.

DISCOVERING STRUCfURE FROM MOTION IN MONKEY, MAN AND MACHINE Ralph M. Siegel· The Salk Institute of Biology, La Jolla, Ca. 92037 ABSTRACT The ability to obtain three-dimensional structure from visual motion is important for survival of human and nonhuman primates. Using a parallel processing model, the current work explores how the biological visual system might solve this problem and how the neurophysiologist might go about understanding the solution. In the present work, much effort has been expended mimicking the visual system. This was done for one main reason: the model was designed to help direct physiological experiments in the primate. It was hoped that if an approach for understanding the model could be developed, the approach could then be directed at the primate's visual system.

Recently, many modifications to the McCulloch/Pitts model have been proposed where both learning and forgetting occur. Given that the network never saturates (ceases to function effectively due to an overload of information), the learning updates can continue indefinitely. For these networks, we need to introduce performance measmes in addition to the information capacity to evaluate the different networks. We mathematically define quantities such as the plasticity of a network, the efficacy of an information vector, and the probability of network saturation. From these quantities we analytically compare different networks.