There is a widespread misconception that the delta-rule is in some sense guaranteed to work on networks without hidden units. As previous authors have mentioned, there is no such guarantee for classification tasks. We will begin by presenting explicit counterexamples illustrating two different interesting ways in which the delta rule can fail. We go on to provide conditions which do guarantee that gradient descent will successfully train networks without hidden units to perform two-category classification tasks. We discuss the generalization of our ideas to networks with hidden units and to multicategory classification tasks.

Department of Electrical and Computer Engineering University of Notre Dame Notre Dame, IN 46556 ABSTRACT In the present paper we survey and utilize results from the qualitative theory of large scale interconnected dynamical systems in order to develop a qualitative theory for the Hopfield model of neural networks. In our approach we view such networks as an interconnection of many single neurons. Our results are phrased in terms of the qualitative properties of the individual neurons and in terms of the properties of the interconnecting structure of the neural networks. Aspects of neural networks which we address include asymptotic stability, exponential stability, and instability of an equilibrium; estimates of trajectory bounds; estimates of the domain of attraction of an asymptotically stable equilibrium; and stability of neural networks under structural perturbations. INTRODUCTION In recent years, neural networks have attracted considerable attention as candidates for novel computational systemsl-3.

LEARNING IN NETWORKS OF NONDETERMINISTIC ADAPTIVE LOGIC ELEMENTS Richard C. Windecker* AT&T Bell Laboratories, Middletown, NJ 07748 ABSTRACT This paper presents a model of nondeterministic adaptive automata that are constructed from simpler nondeterministic adaptive information processing elements. The first half of the paper describes the model. Chief among these properties is that network aggregates of the model elements can adapt appropriately when a single reinforcement channel provides the same positive or negative reinforcement signal to all adaptive elements of the network at the same time. This holds for multiple-input, multiple-output, multiple-layered, combinational and sequential networks. It also holds when some network elements are "hidden" in that their outputs are not directly seen by the external environment. INTRODUCTION There are two primary motivations for studying models of adaptive automata constructed from simple parts. First, they let us learn things about real biological systems whose properties are difficult to study directly: We form a hypothesis about such systems, embody it in a model, and then see if the model has reasonable learning and behavioral properties. In the present work, the hypothesis being tested is: that much of an animal's behavior as determined by its nervous system is intrinsically nondeterministic; that learning consists of incremental changes in the probabilities governing the animal's behavior; and that this is a consequence of the animal's nervous system consisting of an aggregate of information processing elements some of which are individually nondeterministic and adaptive. The second motivation for studying models of this type is to find ways of building machines that can learn to do (artificially) intelligent and practical things.

This paper focuses on the issue of learning in these networks especially with regard to their implementation in an electronic system. Learning phenomena that have been studied include associative memoryllJ.

Centric Models of the Orientation Map in Primary Visual Cortex William Baxter Department of Computer Science, S.U.N.Y. at Buffalo, NY 14620 Bruce Dow Department of Physiology, S.U.N.Y. at Buffalo, NY 14620 Abstract In the visual cortex of the monkey the horizontal organization of the preferred orientations of orientation-selective cells follows two opposing rules: 1) neighbors tend to have similar orientation preferences, and 2) many different orientations are observed in a local region. Several orientation models which satisfy these constraints are found to differ in the spacing and the topological index of their singularities. Using the rate of orientation change as a measure, the models are compared to published experimental results. Introduction It has been known for some years that there exist orientation-sensitive neurons in the visual cortex of cats and mOnkeysl,2. These cells react to highly specific patterns of light occurring in narrowly circumscribed regiOns of the visual field, i.e., the cell's receptive field. The best patterns for such cells are typically not diffuse levels of illumination, but elongated bars or edges oriented at specific angles.

How Neural Nets Work Alan Lapedes Robert Farber Theoretical Division Los Alamos National Laboratory Los Alamos, NM 87545 Abstract: There is presently great interest in the abilities of neural networks to mimic "qualitative reasoning" by manipulating neural incodings of symbols. Less work has been performed on using neural networks to process floating point numbers and it is sometimes stated that neural networks are somehow inherently inaccurate and therefore best suited for "fuzzy" qualitative reasoning. Nevertheless, the potential speed of massively parallel operations make neural net "number crunching" an interesting topic to explore. In this paper we discuss some of our work in which we demonstrate that for certain applications neural networks can achieve significantly higher numerical accuracy than more conventional techniques. In particular, prediction of future values of a chaotic time series can be performed with exceptionally high accuracy. We analyze how a neural net is able to do this, and in the process show that a large class of functions from Rn. Rffl may be accurately approximated by a backpropagation neural net with just two "hidden" layers. The network uses this functional approximation to perform either interpolation (signal processing applications) or extrapolation (symbol processing applicationsJ.

This viewpoint allows the class of probability distributions, P, the neural network can acquire to be explicitly specified. Learning algorithms for the neural network which search for the "most probable" member of P can then be designed. Statistical tests which decide if the "true" or environmental probability distribution is in P can also be developed. Example applications of the theory to the highly nonlinear back-propagation learning algorithm, and the networks of Hopfield and Anderson are discussed. INTRODUCTION A connectionist system is a network of simple neuron-like computing elements which can store and retrieve information, and most importantly make generalizations. Using terminology suggested by Rumelhart & McClelland 1, the computing elements of a connectionist system are called units, and each unit is associated with a real number indicating its activity level. The activity level of a given unit in the system can also influence the activity level of another unit. The degree of influence between two such units is often characterized by a parameter of the system known as a connection strength. During the information retrieval process some subset of the units in the system are activated, and these units in turn activate neighboring units via the inter-unit connection strengths.

ENCODING GEOMETRIC INVARIANCES IN HIGHER-ORDER NEURAL NETWORKS C.L. Giles Air Force Office of Scientific Research, Bolling AFB, DC 20332 R.D. Griffin Naval Research Laboratory, Washington, DC 20375-5000 T. Maxwell Sachs-Freeman Associates, Landover, MD 20785 ABSTRACT We describe a method of constructing higher-order neural networks that respond invariantly under geometric transformations on the input space. By requiring each unit to satisfy a set of constraints on the interconnection weights, a particular structure is imposed on the network. A network built using such an architecture maintains its invariant performance independent of the values the weights assume, of the learning rules used, and of the form of the nonlinearities in the network. The invariance exhibited by a firstorder network is usually of a trivial sort, e.g., responding only to the average input in the case of translation invariance, whereas higher-order networks can perform useful functions and still exhibit the invariance. We derive the weight constraints for translation, rotation, scale, and several combinations of these transformations, and report results of simulation studies.

SYNCHRONIZATION IN NEURAL NETS Jacques J. Vidal University of California Los Angeles, Los Angeles, Ca. 90024 John Haggerty· ABSTRACT The paper presents an artificial neural network concept (the Synchronizable Oscillator Networks) where the instants of individual firings in the form of point processes constitute the only form of information transmitted between joining neurons. This type of communication contrasts with that which is assumed in most other models which typically are continuous or discrete value-passing networks. Limiting the messages received by each processing unit to time markers that signal the firing of other units presents significant implemen tation advantages. When interaction is present, the scheduled firings are advanced or delayed by the firing of neighboring neurons. Networks of such neurons become global oscillators which exhibit multiple synchronizing attractors.

In this article we consider two aspects of computation with neural networks. Firstly we consider the problem of the complexity of the network required to compute classes of specified (structured) functions. We give a brief overview of basic known complexity theorems for readers familiar with neural network models but less familiar with circuit complexity theories. We argue that there is considerable computational and physiological justification for the thesis that shallow circuits (Le., networks with relatively few layers) are computationally more efficient. We hence concentrate on structured (as opposed to random) problems that can be computed in shallow (constant depth) circuits with a relatively few number (polynomial) of elements, and demonstrate classes of structured problems that are amenable to such low cost solutions. We discuss an allied problem-the complexity of learning-and close with some open problems and a discussion of the observed limitations of the theoretical approach. We next turn to a rigourous classification of how much a network of given structure can do; i.e., the computational capacity of a given construct.