'What did the Robot do in my Absence?' Video Foundation Models to Enhance Intermittent Supervision