Continual Vision-Language Representation Learning with Off-Diagonal Information