EgoCHARM: Resource-Efficient Hierarchical Activity Recognition using an Egocentric IMU Sensor

Figure 1: We propose a resource-efficient hierarchical architecture, EgoCHARM, to classify both high and low level activities using a single egocentric, head-mounted IMU. On the left, we show the Aria smartglasses [9, 27] featuring the IMU along with sample acceleration data from selected low and high level activities within our dataset. Our hierarchical architecture, shown on the right, features a low level encoder that inputs 1s low level windows of IMU data and extracts motion embeddings, which can be used to predict low level activities or aggregated over time (30s) and inputted into a high level architecture to predict high level activities. Human activity recognition (HAR) on smartglasses has various use cases, including health/fitness tracking and input for context-aware AI assistants. However, current approaches for egocentric activity recognition suffer from low performance or are resource-intensive. In this work, we introduce a resource (memory, compute, power, sample) efficient machine learning algorithm, EgoCHARM, for recognizing both high level and low level activities using a single egocentric (head-mounted) Inertial Measurement Unit (IMU). Our hierarchical algorithm employs a semi-supervised learning strategy, requiring primarily high level activity labels for training, to learn generalizable low level motion embeddings that can be effectively utilized for low level activity recognition. We evaluate our method on 9 high level and 3 low level activities achieving 0.826 and 0.855 F1 scores on high level and low level activity recognition respectively, with just 63k high level and 22k low level model parameters, allowing the low level encoder to be deployed directly on current IMU chips with compute. Lastly, we present results and insights from a sensitivity analysis and highlight the opportunities and limitations of activity recognition using egocentric IMUs. Work done while at Meta. email: akhil.padmanabha@gmail.com The proliferation of wearable devices and sensor-enabled technologies in portable form factors has created numerous opportunities for tracking, analyzing, and generating insights into human actions and behaviors.