How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching