Retrieval Augmented End-to-End Spoken Dialog Models