W. Scott Stornetta Stanford University, Physics Department, Stanford, Ca., 94305 Tad Hogg and B. A. Huberman Xerox Palo Alto Research Center, Palo Alto, Ca. 94304 ABSTRACT Recognizing patterns with temporal context is important for such tasks as speech recognition, motion detection and signature verification. We propose an architecture in which time serves as its own representation, and temporal context is encoded in the state of the nodes. We contrast this with the approach of replicating portions of the architecture to represent time. As one example of these ideas, we demonstrate an architecture with capacitive inputs serving as temporal feature detectors in an otherwise standard back propagation model. Experiments involving motion detection and word discrimination serve to illustrate novel features of the system.

ABSTRACT The aim of this paper is to explore the spatial organization of neural networks under Markovian assumptions, in what concerns the behaviour ofindividual cells and the interconnection mechanism. Spaceorganizational propertiesof neural nets are very relevant in image modeling and pattern analysis, where spatial computations on stochastic two-dimensionalimage fields are involved. As a first approach we develop a random neural network model, based upon simple probabilistic assumptions,whose organization is studied by means of discrete-event simulation.We then investigate the possibility of approXimating therandom network's behaviour by using an analytical approach originating from the theory of general product-form queueing networks. The neural network is described by an open network of nodes, inwhich customers moving from node to node represent stimulations andconnections between nodes are expressed in terms of suitably selectedrouting probabilities. We obtain the solution of the model under different disciplines affecting the time spent by a stimulation ateach node visited.

The method allows the fully distributed representation of symbolic structures: the roles in the structures, as well as the fillers for those roles, can be arbitrarily non-local. Fully and partially localized special cases reduce to existing cases of connectionist representations of structured data; the tensor product representation generalizes these and the few existing examples of fuUy distributed representations of structures. The representation saturates gracefully as larger structures are represented; it pennits recursive construction of complex representations from simpler ones; it respects the independence of the capacities to generate and maintain multiple bindings in parallel; it extends naturally to continuous structures and continuous representational patterns; it pennits values to also serve as variables; it enables analysis of the interference of symbolic structures stored in associative memories; and it leads to characterization of optimal distributed representations of roles and a recirculation algorithm for learning them. Introduction Any model of complex infonnation processing in networks of simple processors must solve the problem of representing complex structures over network elements. Connectionist models of realistic natural language processing, for example, must employ computationally adequate representations of complex sentences. Many connectionists feel that to develop connectionist systems with the computational power required by complex tasks, distributed representations must be used: an individual processing unit must participate in the representation of multiple items, and each item must be represented as a pattern of activity of multiple processors. Connectionist models have used more or less distributed representations of more or less complex structures, but little if any general analysis of the problem of distributed representation of complex infonnation has been carried out This paper reports results of an analysis of a general method called the tensor product representation.

To demonstrate these methods we have trained an ANS network to drive a vehicle through simulated rreeway traffic. I ntJooducticn Computational systems employing fine grained parallelism are revolutionizing the way we approach a number or long standing problems involving pattern recognition and cognitive processing. Thefield spans a wide variety or computational networks, rrom constructs emulating neural runctions, to more crystalline configurations that resemble systolic arrays. Several titles are used to describe this broad area or research, we use the term artificial neural systems (ANS). Our concern inthis work is the use or ANS ror manually training certain types or autonomous systems where the desired rules of behavior are difficult to rormulate. Artificial neural systems consist of a number or processing elements interconnected in a weighted, user-specified fashion, the interconnection weights acting as memory ror the system. Each processing element calculatE', an output value based on the weighted sum or its inputs. In addition, the input data is correlated with the output or desired output (specified by an instructive agent) in a training rule that is used to adjust the interconnection weights.

ABSTRACT The study of distributed memory systems has produced a number of models which work well in limited domains. However, until recently, the application of such systems to realworld problemshas been difficult because of storage limitations, and their inherent architectural (and for serial simulation, computational) complexity. Recent development of memories with unrestricted storage capacity and economical feedforward architectures has opened the way to the application of such systems to complex pattern recognition problems. However, such problems are sometimes underspecified by the features which describe the environment, and thus a significant portion of the pattern environment is often non-separable. We will review current work on high density memory systems and their network implementations.

DCPS' memory scheme is a modified version of the Random Receptors method [5]. The symbol space is the set of all triples over a 25 letter alphabet. Units have fixed-size receptive fields organized as 6 x 6 x 6 subspaces. Patterns are manipulated to minimize the variance in pattern size across symbols.

The approach is applied both to Michie and Chambers BOXES algorithm and to Barto, Sutton and Anderson's extension, the ASE/ACE system, and has significantly improved the convergence rate of stochastically based learning automata. Recurrencelearning is a new nonlinear reward-penalty algorithm. It exploits information found during learning trials to reinforce decisions resulting in the recurrence of nonfailing states. Recurrence learning applies positive reinforcement during the exploration of the search space, whereas in the BOXES or ASE algorithms, only negative weight reinforcement is applied, and then only on failure. Simulation results show that the added information from recurrence learning increases the learning rate.

Serial Order: A Parallel Distributed Processing Approach.

The matrix chip contains a programmable 32X32 array of "long channel" NMOSFET binary connection elements implemented ina 3-um bulk CMOS process. Since the neurons are kept offchip, the synaptic chip serves as a "cascadable" building block for a multi-chip synaptic network as large as 512X512 in size. As an alternative to the programmable NMOSFET (long channel) connection elements, tailored thin film resistors are deposited, in series with FET switches, on some CMOS test chips, to obtain the weak synaptic connections. Although deposition and patterning of the resistors require additional processing steps, they promise substantial savings in silcon area. The performance of a synaptic chip in a 32-neuron breadboard system in an associative memory test application is discussed. INTRODUCTION The highly parallel and distributive architecture of neural networks offers potential advantages in fault-tolerant and high speed associative information processing.

Current Focus Of Learning Research Most connectionist learning algorithms may be grouped into three general catagories, commonly referred to as supenJised, unsupenJised, and reinforcement learning. Supervised learning requires the explicit participation of an intelligent teacher, usually to provide the learning system with task-relevant input-output pairs (for two recent examples, see [1,2]). Unsupervised learning, exemplified by "clustering" algorithms, are generally concerned with detecting structure in a stream of input patterns [3,4,5,6,7]. In its final state, an unsupervised learning system will typically represent the discovered structure as a set of categories representing regions of the input space, or, more generally, as a mapping from the input space into a space of lower dimension that is somehow better suited to the task at hand. In reinforcement learning, a "critic" rewards or penalizes the learning system, until the system ultimately produces the correct output in response to a given input pattern [8].