Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation