Appendix

Weights and activation engaged inmatrix multiplication are also quantized. B.2 Problematicquantizationnodes In this subsection, we give some simple and direct studies to elaborate on the most problematic tensors (outputs ofLayerNorm structures andGELU). On the one hand, we compare the cosine similarity between the FP value and the quantized one foreachoutput. For example, "Query" means disabling the quantization of output at Query modulesacross12layers. Infact,wefindthatthisnotonly emergesinfine-tuned models but is also obvious in the pre-trained ones.