Matching Skeleton-based Activity Representations with Heterogeneous Signals for HAR