Numerous evaluation metrics have been developed for natural language generation tasks, but their effectiveness in evaluating stories is limited as they are not specifically tailored to assess intricate aspects of storytelling, such as fluency and interestingness. In this paper, we introduce DELTASCORE, a novel methodology that employs perturbation techniques for the evaluation of nuanced story aspects. Our central proposition posits that the extent to which a story excels in a specific aspect (e.g., fluency) correlates with the magnitude of its susceptibility to particular perturbations (e.g., the introduction of typos). Given this, we measure the quality of an aspect by calculating the likelihood difference between pre- and post-perturbation states using pre-trained language models. We compare DELTASCORE with existing metrics on storytelling datasets from two domains in five fine-grained story aspects: fluency, coherence, relatedness, logicality, and interestingness. DELTASCORE demonstrates remarkable performance, revealing a surprising finding that a specific perturbation proves highly effective in capturing multiple aspects.

Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs. We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

This demo features a user interface for authoring stories and story fragments for use by the Minstrel Remixed story generation system. It also demonstrates Minstrel Remixed in use, allowing users to author story fragments and then have Minstrel Remixed expand these fragments and generate stories based on them. The focus is on the interface for story-fragment authoring, which exposes Minstrel's graph- of-frames knowledge representation format to the user in an interactive manner. It also exposes Minstrel Remixed's story generation capabilities as they exist currently, including the Author-Level Planning (ALP) and Transform Adapt Recall Methods (TRAM) systems.

This demo features a user interface for authoring stories and story fragments for use by the Minstrel Remixed story generation system. It also demonstrates Minstrel Remixed in use, allowing users to author story fragments and then have Minstrel Remixed expand these fragments and generate stories based on them. The focus is on the interface for story-fragment authoring, which exposes Minstrel's graph- of-frames knowledge representation format to the user in an interactive manner. It also exposes Minstrel Remixed's story generation capabilities as they exist currently, including the Author-Level Planning (ALP) and Transform Adapt Recall Methods (TRAM) systems.

The first major story generation system, which preceded Minstrel and which While ongoing progress in digital entertainment also received significant attention, is Tale-Spin (Meehan technology continues, commercial designers still largely 1977). Like Minstrel, this system generates stories which eschew systems for procedural story generation, preferring satisfy user-submitted requirements. Tale-Spin creates instead to generate content by hand. In the academic English stories by planning a method for the main literature, projects such as (Appling & Riedl 2009, Roberts character to achieve her or his goal, using inferences and & Isbell 2009) continue to investigate ways to improve the rules to generate a large number of details about a story nuances of interactive storytelling while others attempt to (many of which do little contribute to an audience create their own systems to investigate ways to use experience). This contrasts nicely with Minstrel, which knowledge from interactive narrative and story generation performs no logical inferences and which performs all in new fields such as playable games (Drachen & Hitchens actions from the point of view of an author, manipulating et al. 2009, Sullivan, Mateas & Wardrip-Fruin 2009).