Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty