Reliable Test-Time Adaptation via Agreement-on-the-Line