Learning Robot Safety from Sparse Human Feedback using Conformal Prediction