Plug-and-Play Transformer Modules for Test-Time Adaptation