LR, is important for good generalization in real-life deep nets.

The right to be forgotten calls for efficient machine unlearning techniques that make trained machine learning models forget a cohort of data.

It is well known that modern deep neural networks are powerful enough to memorize datasets even when the labels have been randomized.

These results show the improvements of rankers on the Codex code generation model. We further conducted an experiment with a GPT -Neo 1.3B model for the ranker. It took almost 4 days to complete 7 epochs of training with 16 V100 GPUs running in parallel. And still, it didn't reach the performance achieved by the CodeBERT model. CodeBERT based ranker took only 12 hours to train for 30 epochs (with 16 GPUs).

Atypically to standard NN applications, financial industry practitioners use such models equally to replicate market prices and to value other financial instruments. In other words, low training losses are as important as generalization capabilities.