In other words, the assumptions in these works imply that the state space is already well-explored.

ZO methods only assume an oracle access to the function value at any given point, instead of gradients as in first-order methods.