Comparisons Are All You Need for Optimizing Smooth Functions