CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model