Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data