Robust Noise Attenuation via Adaptive Pooling of Transformer Outputs