Despite advances in generating fluent texts, existing pretraining models tend to attach incoherent event sequences to involved entities when generating narratives such as stories and news. We conjecture that such issues result from representing entities as static embeddings of superficial words, while neglecting to model their ever-changing states, i.e., the information they carry, as the text unfolds. Therefore, we extend the Transformer model to dynamically conduct entity state updates and sentence realization for narrative generation. We propose a contrastive framework to learn the state representations in a discrete space, and insert additional attention layers into the decoder to better exploit these states. Experiments on two narrative datasets show that our model can generate more coherent and diverse narratives than strong baselines with the guidance of meaningful entity states.

Legal documents are typically long and written in legalese, which makes it particularly difficult for laypeople to understand their rights and duties. While natural language understanding technologies can be valuable in supporting such understanding in the legal domain, the limited availability of datasets annotated for deontic modalities in the legal domain, due to the cost of hiring experts and privacy issues, is a bottleneck. To this end, we introduce, LEXDEMOD, a corpus of English contracts annotated with deontic modality expressed with respect to a contracting party or agent along with the modal triggers. We benchmark this dataset on two tasks: (i) agent-specific multi-label deontic modality classification, and (ii) agent-specific deontic modality and trigger span detection using Transformer-based (Vaswani et al., 2017) language models. Transfer learning experiments show that the linguistic diversity of modal expressions in LEXDEMOD generalizes reasonably from lease to employment and rental agreements. A small case study indicates that a model trained on LEXDEMOD can detect red flags with high recall. We believe our work offers a new research direction for deontic modality detection in the legal domain.

The composition of the academic workforce thus shapes what advances are made and who benefits from them,20,21 in part because demographic diversity in science is known to accelerate innovation and improve problem solving.17,31 Despite a continued emphasis on broadening participation, women faculty in the U.S. remain underrepresented relative to women's share of the U.S. population by more than a factor of two, and Black, Hispanic, and Native faculty by more than a factor of five.37,40 Women's underrepresentation among computing researchers also persists internationally. For example, women are estimated to comprise less than 10% of contributors to international computer science journals.25 On one hand, there are generational problems, in which faculty diversity changes slowly because it takes many years for diversity increases at the earliest stages of training to propagate up to more senior levels.16 On the other hand, there are structural and social climate problems in the U.S.,1 in which members of underrepresented groups who aspire to or have a faculty career are pushed or pulled out of the community, which may counteract efforts to address generational problems. In concert, these two effects may lead to a persistent overrepresentation of majority groups5 despite efforts to the contrary. We consider a third class of problem, which exists because most faculty are hired via searches that focus on a particular subfield of computing--for example, artificial intelligence (AI). As a result, field-level demographic dynamics such as gender, racial, and socioeconomic representation are in fact driven by diversity differences across computing's subfields and the representation of those subfields among the suppliers of future faculty.8 For example, faculty searches in subfields with fewer women than other subfields are less likely to increase a department's gender diversity.

SPEAKING at the University of Cambridge in 1980, Stephen Hawking considered the possibility of a theory of everything that would unite general relativity and quantum mechanics – our two leading descriptions of reality – into one neat, all-encompassing equation. We would need some help, he reckoned, from computers. Then he made a provocative prediction about these machines' growing abilities. "The end might not be in sight for theoretical physics," said Hawking. "But it might be in sight for theoretical physicists."

In the majority of fiction that concerns artificial intelligence (AI), it replaces us in almost every industry. Often, only the artistic fields are left untouched. Usually, a robot or AI showing capabilities to create a work is seen as a completion, at the pinnacle of intelligence, considered almost human. Yet, AI and the arts are already flirting in reality, whether in music creation or visual arts, it is becoming more and more present. However, on August 26, 2022, there was a rude shock when, at a visual arts competition in Colorado, the winning image was entirely designed by an AI.

According to Meta, Galactica can "summarize academic papers, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more." But soon after its launch, it was pretty easy for outsiders to prompt the model to provide "scientific research" on the benefits of homophobia, anti-Semitism, suicide, eating glass, being white, or being a man. Meanwhile, papers on AIDS or racism were blocked. As my colleague Will Douglas Heaven writes in his story about the debacle: "Meta's misstep--and its hubris--show once again that Big Tech has a blind spot about the severe limitations of large language models." Not only was Galactica's launch premature, but it shows how insufficient AI researchers' efforts to make large language models safer have been. Meta might have been confident that Galactica outperformed competitors in generating scientific-sounding content.

According to the Global Burden of Disease list provided by the World Health Organization (WHO), mental disorders are among the most debilitating disorders.To improve the diagnosis and the therapy effectiveness in recent years, researchers have tried to identify individual biomarkers. Gathering neurobiological data however, is costly and time-consuming. Another potential source of information, which is already part of the clinical routine, are therapist-patient dialogues. While there are some pioneering works investigating the role of language as predictors for various therapeutic parameters, for example patient-therapist alliance, there are no large-scale studies. A major obstacle to conduct these studies is the availability of sizeable datasets, which are needed to train machine learning models. While these conversations are part of the daily routine of clinicians, gathering them is usually hindered by various ethical (purpose of data usage), legal (data privacy) and technical (data formatting) limitations. Some of these limitations are particular to the domain of therapy dialogues, like the increased difficulty in anonymisation, or the transcription of the recordings. In this paper, we elaborate on the challenges we faced in starting our collection of therapist-patient dialogues in a psychiatry clinic under the General Data Privacy Regulation of the European Union with the goal to use the data for Natural Language Processing (NLP) research. We give an overview of each step in our procedure and point out the potential pitfalls to motivate further research in this field.

Scoring systems, as a type of predictive model, have significant advantages in interpretability and transparency and facilitate quick decision-making. As such, scoring systems have been extensively used in a wide variety of industries such as healthcare and criminal justice. However, the fairness issues in these models have long been criticized, and the use of big data and machine learning algorithms in the construction of scoring systems heightens this concern. In this paper, we propose a general framework to create fairness-aware, data-driven scoring systems. First, we develop a social welfare function that incorporates both efficiency and group fairness. Then, we transform the social welfare maximization problem into the risk minimization task in machine learning, and derive a fairness-aware scoring system with the help of mixed integer programming. Lastly, several theoretical bounds are derived for providing parameter selection suggestions. Our proposed framework provides a suitable solution to address group fairness concerns in the development of scoring systems. It enables policymakers to set and customize their desired fairness requirements as well as other application-specific constraints. We test the proposed algorithm with several empirical data sets. Experimental evidence supports the effectiveness of the proposed scoring system in achieving the optimal welfare of stakeholders and in balancing the needs for interpretability, fairness, and efficiency.

The raise of machine learning and deep learning led to significant improvement in several domains. This change is supported by both the dramatic rise in computation power and the collection of large datasets. Such massive datasets often include personal data which can represent a threat to privacy. Membership inference attacks are a novel direction of research which aims at recovering training data used by a learning algorithm. In this paper, we develop a mean to measure the leakage of training data leveraging a quantity appearing as a proxy of the total variation of a trained model near its training samples. We extend our work by providing a novel defense mechanism. Our contributions are supported by empirical evidence through convincing numerical experiments.

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models. Warning: this paper contains examples that may be offensive or upsetting.