Keep Guessing? When Considering Inference Scaling, Mind the Baselines