Efficient Model Adaptation for Continual Learning at the Edge