A Appendix

A.1 Pseudocode for our search algorithm Our framework follows a standard search pipeline: 1. Candidate proposal: the search algorithm samples an optimizer from the search space. This procedure is commonly used in other AutoML domains, such as Neural Architecture Search [47, 67] and Hyperparameter Optimization [23]. Algorithm 1 and 2 summarize the complete search process. Input: Candidate set A, constraints C, operator set O, maximum super-tree depth D, maximum traversal level L, MC sample size M for each level, score threshold, proposal size K. Following NOS-RL, we use n =0.5 for cosine decay and n = 20 for restart decay. We set the bound for clip operator to 0.003, and the dropout ratio to 0.1 for drop operator. Note that one can always include more options of these values by adding new operator variants to the space (e.g. drop For all input operators, we use their default PyTorch implementations and hyper-parameters.