Language Models Struggle to Achieve a Consistent Temporal Representation of Facts