any ground-truth visual relationship annotations, avoiding the challenging manual annotation of visual relationships;

We thank all the reviewers for their efforts and constructive comments! Below we address the important and common issues. On the other hand, the probing loss can further help improve the performance. As mentioned by R4, "this paper introduces a new and BLEU between captions (query image) and reference captions (retrieved images) in Table B. We see that'Obj.+Rel.' Table B: Results on 1K query images randomly sampled from MSCOCO.