Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model