Few-shot Adaptation for Manipulating Granular Materials Under Domain Shift