A Appendix

These results show the improvements of rankers on the Codex code generation model. We further conducted an experiment with a GPT -Neo 1.3B model for the ranker. It took almost 4 days to complete 7 epochs of training with 16 V100 GPUs running in parallel. And still, it didn't reach the performance achieved by the CodeBERT model. CodeBERT based ranker took only 12 hours to train for 30 epochs (with 16 GPUs).