Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities. However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques. To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries. Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited. To address this, we present ETHIC, a novel benchmark designed to assess LLMs' ability to leverage the entire context. Our benchmark comprises 2,648 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. Our evaluations reveal significant performance drops in contemporary LLMs, highlighting a critical challenge in managing long contexts. Our benchmark is available at https://github.com/dmis-lab/ETHIC.

Consider developing AI mindful of each stakeholder to benefit the environment and all present and future ecosystem members, human and non-human alike.

Artificial Intelligence (AI), defined as "a system's ability to interpret external data correctly, to learn from such data, and to use those learnings to achieve specific goals and tasks through flexible adaptation" (Kaplan and Haenlein 2019, p. 17), is one of the most popular topics across a variety of academic disciplines, industry sectors, and business functions, and widely influences society at large. While many will first think of computational, organizational, or technological issues related to AI, there is an entire set of ethical dimensions triggered by this new era which urgently need to be analyzed, discussed, and reflected upon. As pointed out by Martin and Freeman (2004, p. 353) "business ethicists are uniquely positioned to analyze the relationship between business, technology, and society". There are many examples where inappropriate use of AI has resulted in unethical outcomes and behavior. Examples include image recognition services which make offensive classifications of minorities due to biased algorithms; Microsoft's AI chatbot Tay which became racist and adopted hate speech after only one day; and Amazon's facial recognition technology which simply failed to recognize users with darker skin colors.

As news broke of a new app called DeepNude, which allowed anyone to alter a photo of a woman to make them appear nude, I found myself deeply disturbed by the speed at which deepfakes are evolving. Such a tangible and accessible tool highlights the darker side of AI, computer vision, and other machine learning techniques in the wrong hands. And while there are some incredibly overt examples of how deepfakes can be used to doctor videos of individuals, their appearance, their activities, as well as what they are saying, the accessibility of this technology has mostly been in the hands of the technical few that understood it. Even with this knowledge, the time it takes to generate convincing deepfakes was also a barrier. But DeepNude showed that altering images can be done in seconds versus the days it previously took on incredibly powerful machines out of reach of the general public.

The proposed changes to the ACM Code of Ethics and Professional Conduct, as discussed by Don Gotterbarn et al. in "ACM Code of Ethics: A Guide for Positive Action"1 (Digital Edition, Jan. 2018), are generally misguided and should be rejected by the ACM membership. The changes attempt to, for example, create real obligations on members to enforce hiring quotas/priorities with debatable efficacy while ACM members are neither HR specialists nor psychologists; create "safe spaces for all people," a counterproductive concept causing problems in a number of universities; counter harassment while not being lawyers or police officers; enforce privacy while not being lawyers; ensure "the public good" while not being elected leaders; encourage acceptance of "social responsibilities" while not defining them or being elected leaders or those charged with implementing government policy; and monitor computer systems integrated into society for "fair access" while not being lawyers or part of the C-suite. ACM is a computing society, not a society of activists for social justice, community organizers, lawyers, police officers, or MBAs. The proposed changes add nothing related specifically to computing and far too much related to these other fields, and also fail to address, in any significant new way, probably the greatest ethical hole in computing today--security and hacking. If the proposed revised Code is ever submitted to a vote by the membership, I will be voting against it and urge other members to do so as well.

This article is part of an MIT SMR initiative exploring how technology is reshaping the practice of management. Smart machines need ethics, too: Remember that movie in which a computer asked an impossibly young Matthew Broderick, "Shall we play a game?" Four decades later, it turns out that global thermonuclear war may be the least likely of a slew of ethical dilemmas associated with smart machines -- dilemmas with which we are only just beginning to grapple. The worrisome lack of a code of ethics for smart machines has not been lost on Alphabet, Amazon, Facebook, IBM, and Microsoft, according to a report by John Markoff in The New York Times. The five tech giants (if you buy Mark Zuckerberg's contention that he isn't running a media company) have formed an industry partnership to develop and adopt ethical standards for artificial intelligence -- an effort that Markoff infers is motivated as much to head off government regulation as to safeguard the world from black-hearted machines. On the other hand, the first of a century's worth of quinquennial reports from Stanford's One Hundred Year Study on Artificial Intelligence (AI100) throws the ethical ball into the government's court.