With or Without Replacement? Improving Confidence in Fourier Imaging