Trust, but Verify: Using Self-Supervised Probing to Improve Trustworthiness