A Appendix

A.1 Pseudocode for our search algorithm Our framework follows a standard search pipeline: 1. Candidate proposal: the search algorithm samples an optimizer from the search space. Search: The optimizer score is used to guide the search algorithm to propose new optimizers. Our set of operators is a subset of the full operator set presented in Section 4.1 of the NOS-RL paper. We refer the readers to "Further discussions on NOS-RL baseline" in Appendix D.3 for more For inspiration on what to add, the user might look into 1). With an augmented operator set, other components in our algorithm can largely remain the same.