Enhancing Speech-to-Speech Dialogue Modeling with End-to-End Retrieval-Augmented Generation