Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptation