PerceptionNet: A Deep Convolutional Neural Network for Late Sensor Fusion

Abstract-- Human Activity Recognition (HAR) based on motion sensors has drawn a lot of attention over the last few years, since perceiving the human status enables context-aware applications to adapt their services on users' needs. However, motion sensor fusion and feature extraction have not reached their full potentials, remaining still an open issue. In this paper, we introduce PerceptionNet, a deep Convolutional Neural Network (CNN) that applies a late 2D convolution to multimodal time-series sensor data, in order to extract automatically efficient features for HAR. We evaluate our approach on two public available HAR datasets to demonstrate that the proposed model fuses effectively multimodal sensors and improves the performance of HAR. In particular, PerceptionNet surpasses the performance of state-of-the-art HAR methods based on: (i) features extracted from humans, (ii) deep CNNs exploiting early fusion approaches, and (iii) Long Short-Term Memory (LSTM), by an average accuracy of more than 3%. The proliferation of the Internet of Things (IoT) over the last few years, has contributed to the collection of huge amounts of time-series data. An IoT device with high sampling rates, such as a wearable, produces hundreds of data every second, resulting to a data explosion, considering the vast number of such devices connected over the internet. Through real-time or batch data processing, meaningful information is extracted, revealing daily patterns of individual owners or social groups.