The Pitfalls of Publishing in the Age of LLMs: Strange and Surprising Adventures with a High-Impact NLP Journal

In the dawn of the age of Large Language Models (LLMs), already much has been said about how researchers are making use of LLMs to author articles. For example, according to an article in Scientific American [1], "One percent of scientific articles published in 2023 showed signs of generative AI's potential involvement, according to a recent analysis." However, far less has been said about how reviewers are now abusing their role, sometimes with the editor's collusion. Here is our report of a case in point. We submitted a manuscript on domain-independent deception detection to a highly respected journal. As a consequence of a reviewer's use of an LLM, we both received a most peculiar review and also lost the promised confidentiality regarding our submission.