Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective