Analyzing the Roles of Language and Vision in Learning from Limited Data