Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models