BASS: Block-wise Adaptation for Speech Summarization