Continual Speech Learning with Fused Speech Features