The dominant approaches to text representation in natural language rely on learning embeddings on massive corpora which have convenient properties such as compositionality and distance preservation.

We thank the reviewers for their helpful comments and remarks. Reviewer 3 suggests to assess'if the data augmentation scheme is actually helpful for a range of tasks' Also, when compared with other methods, we achieve better performance. Reviewer 1. Thank you for spotting the typos on lines 11, 58, 88 and Table 1. We will update the paper accordingly. 'Question: are the F1 scores here (T able 2) stratified at all by "extreme" vs not?' 'The proposed method regularizes fixed BERT embeddings to be heavy tailed, however most SOTA methods fine-tune Reviewer 3. Thank you for spotting the typos in Table 1.

The dominant approaches to text representation in natural language rely on learning embeddings on massive corpora which have convenient properties such as compositionality and distance preservation.

The dominant approaches to text representation in natural language rely on learning embeddings on massive corpora which have convenient properties such as compositionality and distance preservation.