Learning and Predicting Sequential Tasks Using Recurrent Neural Networks and Multiple Model Filtering

An integral part of human-robot collaboration is the ability ofthe robot to understand and predict human motion. Predicting what the human collaborator will do next is very useful in planning the robot’s response. In this paper, an algorithm for early detection and prediction of human activities is presented. For a given sequential task composed of many steps, a long short-term memory (LSTM) recurrent neural network (RNN) model is trained to learn the underlying sequence of steps. The trained network is then used to make predictions about the subsequent steps the human is about to carry out. The prediction of next steps requires information about the current step that is being carried out. The steps are inferred by observing the motion trajectories of the human arm and predicting where the human is reaching. The trajectories of the arm motion are modeled by using a dynamical system with contracting behavior towards the object. A neural network (NN) is used to learn the dynamics under the contraction analysis constraints. An interacting multiple model (IMM) framework is used for the early prediction of the goal locations of reaching motions. Since humans tend to look in the direction of the object they are reaching for, the prior probabilities of the models are calculated based on the human eye gaze. Experimental results based on an audio amplifier circuit assembly task are used to validate the proposed algorithm.