A Tree Architecture of LSTM Networks for Sequential Regression with Missing Data

A Tree Architecture of LSTM Networks for Sequential Regression with Missing Data S. Onur Sahin and Suleyman S. Kozat, Senior Member, IEEE Abstract --We investigate regression for variable length sequential data containing missing samples and introduce a novel tree architecture based on the Long Short-T erm Memory (LSTM) networks. In our architecture, we employ a variable number of LSTM networks, which use only the existing inputs in the sequence, in a treelike architecture without any statistical assumptions or imputations on the missing data, unlike all the previous approaches. In particular, we incorporate the missingness information by selecting a subset of these LSTM networks based on "presence-pattern" of a certain number of previous inputs. From the mixture of experts perspective, we train different LSTM networks as our experts for various missingness patterns and then combine their outputs to generate the final prediction. We also provide the computational complexity analysis of the proposed architecture, which is in the same order of the complexity of the conventional LSTM architectures for the sequence length. Our method can be readily extended to similar structures such as GRUs, RNNs as remarked in the paper . In the experiments, we achieve significant performance improvements with respect to the state-of-the-art methods for the well-known financial and real life datasets. I NTRODUCTION A. Preliminaries We study regression of variable length sequential data containing missing samples. Here, we sequentially receive a data sequence suffering from missing input values and estimate an unknown desired signal related to this data sequence. In most regression tasks involving sequential data, one usually assumes that we have the complete data sequence [1]. However, nearly in every real life application, the data sequences usually contain missing input values due to various reasons such as inconvenience, anomalies and cost savings [2], [3]. Furthermore, in many real life problems such as medical imaging applications [4] and finance [5], we encounter nonuniformly sampled data, which can be modelled as a missing data case [6]. To mitigate these issues, the widely used approaches make certain statistical assumptions on the missing data [7], [8], however, these assumptions usually do not hold and the This works is in part supported by Turkish Academy of Sciences Outstanding Researcher Programme and TUBIT AK Project No: 117E153.