Review for NeurIPS paper: Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation

Weaknesses: v) It is difficult to attribute the source of empirical gains, since the paper is presenting both a problem-specific architecture search space and a particular search method. The comparison to random is missing some potentially-important measures as it has no error bars or plot of the distribution. Though the comparison to evolutionary methods in Fig 2. is a good experiment along these lines, the (missing) random comparison is especially important [a]. The comparison to random is against the *best* model found by random search, instead of error bars or any modeling of the search space. This'd be important for comparisons that separate out the search vs design space as in [a,b].