AdaptVC: High Quality Voice Conversion with Adaptive Learning