A Appendix A.1 More Ablations and Visualizations Effect of Blocking Gradient of f(s

As mentioned in Section 3.2, we compare the performance of different detectors with or without blocking the gradient of f(s We attribute this to the unstable training caused by the gradient from the denominator, so they are blocked out by default in the experiments. Figure 1 visualizes the searched parameterized functions for different detectors on the COCO benchmark [5]. The dots on each line represent the control points for each parameterized function. It can be observed that loss functions for different detectors seem to differ from each other. Their intrinsic differences can lead to distinct loss functions.