We thank the reviewers for their valuable comments! Unclear if the proposed method is better than only using LSH. Thank you for the suggestions. ALSH significantly outperforms the E2LSH and the Reformer LSH scheme. SMYRF-BERT base (see also Table 2).

We study transformer language models, analyzing attention heads whose attention patterns are spread out, and whose attention scores depend weakly on content. We argue that the softmax denominators of these heads are stable when the underlying token distribution is fixed. By sampling softmax denominators from a "calibration text", we can combine together the outputs of multiple such stable heads in the first layer of GPT2-Small, approximating their combined output by a linear summary of the surrounding text. This approximation enables a procedure where from the weights alone - and a single calibration text - we can uncover hundreds of first layer neurons that respond to high-level contextual properties of the surrounding text, including neurons that didn't activate on the calibration text.

We thank the reviewers for their valuable comments! Unclear if the proposed method is better than only using LSH. Thank you for the suggestions. ALSH significantly outperforms the E2LSH and the Reformer LSH scheme. SMYRF-BERT base (see also Table 2).