The influence of visual and linguistic cues on ignorance inference in Vision-Language Models