How many samples are needed to leverage smoothness?