Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing