M2T: Masking Transformers Twice for Faster Decoding