SNAP: Low-Latency Test-Time Adaptation with Sparse Updates