Efficient Transformer Encoders for Mask2Former-style models