STAMP: Spatial-Temporal Adapter with Multi-Head Pooling