Test of Time: Instilling Video-Language Models with a Sense of Time