Weaknesses: The idea of predicting IoUs between predictions and ground-truths has already been explored by other works. It seems to me the main differences between this and prior works are: 1) Prior works such as IoU-Net still have a separate branch for classification but this one does not. The contribution of each modification to the final performance is unclear. Does the separate branch for classification hurt the performance? What if we just ignore the separate branch for classification in prior works?

Expert reviewers agree that this paper has a novel contribution and lean toward accept. The reviewers like the proposed idea and find the empirical results strong. But the presentation of the paper can be significantly improved. The authors should follow the reviewers' suggestions and significantly revise the paper. In particular, the authors should improve the presentation of the paper following R4's suggestions and add the additional analysis and clarifications as discussed in the rebuttal.

One-stage detector basically formulates object detection as dense classification and localization (i.e., bounding box regression). The classification is usually optimized by Focal Loss and the box location is commonly learned under Dirac delta distribution. A recent trend for one-stage detectors is to introduce an \emph{individual} prediction branch to estimate the quality of localization, where the predicted quality facilitates the classification to improve detection performance. This paper delves into the \emph{representations} of the above three fundamental elements: quality estimation, classification and localization. Two problems are discovered in existing practices, including (1) the inconsistent usage of the quality estimation and classification between training and inference, and (2) the inflexible Dirac delta distribution for localization.