From Confidence to Collapse in LLM Factual Robustness