Probing Language Models on Their Knowledge Source