Global Optimisation of Neural Network Models via Sequential Sampling

Andrew H Gee Cambridge University Engineering Department Cambridge CB2 1PZ England ahg@eng.cam.ac.uk Abstract We propose a novel strategy for training neural networks using sequential sampling-importanceresampling algorithms. This global optimisation strategy allows us to learn the probability distribution ofthe network weights in a sequential framework. It is well suited to applications involving online, nonlinear, non-Gaussian or non-stationary signal processing. 1 INTRODUCTION This paper addresses sequential training of neural networks using powerful sampling techniques. Sequential techniques are important in many applications of neural networks involvingreal-time signal processing, where data arrival is inherently sequential. Furthermore, one might wish to adopt a sequential training strategy to deal with non-stationarity in signals, so that information from the recent past is lent more credence than information from the distant past. One way to sequentially estimate neural network models is to use a state space formulation and the extended Kalman filter (Singhal and Wu 1988, de Freitas, Niranjan and Gee 1998).