Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation