While large language models (LLMs) equipped with techniques like chain-of-thought prompting have demonstrated impressive capabilities, they still fall short in their ability to reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We introduce MuSR, a dataset for evaluating language models on multistep soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable LLMs are released. Second, our dataset instances are free text narratives corresponding to real-world domains of reasoning; this makes it simultaneously much more challenging than other synthetically-crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.

This story is adapted from Death of an Author, a murder-mystery novella written by Aidan Marchine, a collaboration between author Stephen Marche and three artificial intelligence tools: ChatGPT, Sudowrite, and Cohere. Gus Dupin, walking along the stillness of Stony Lake in the gathering night, recognized the sleek motorboat approaching his dock. A girl in a bright yellow sundress jumped off and sprinted to his mailbox, dropping in an envelope before running back. As she set off into the lake, she yelled "an honest-to-God letter" over her shoulder. Gus Dupin was not accustomed to receiving letters or messages of any kind.

Take Black Mirror's dystopian tech commentary, The Good Place's philosophical exploration of the after-life, and the workplace antics of The Office, mash them together, and you have Amazon's Upload. It takes place in a world that could easily be 10 years from now -- self driving cars are commonplace, the Earth is polluted and over-crowded, and, oh yeah, you can also achieve digital immortality by uploading your consciousness to the cloud. Upload, which premieres today, is an entirely new territory for Greg Daniels, the genius writer behind The Office, and Parks and Rec (not to mention a long run on The Simpsons). But it's a world that's clearly been percolating in his mind for years. It's bold and raunchy in a way a network sitcom never could be, and it defies being classified into a single genre.

John Scalzi is known for his witty science fiction thrillers. Old Man's War and its sequels are his take on military science fiction, while last year's Collapsing Empire was a new foray into space opera. His latest novel Head On is a techno-thriller involving robotic sports leagues and murder, and it's a book that's particularly relevant in our own, technological world. Head On is the sequel to Scalzi's 2014 thriller Lock In and an accompanying novella, Unlocked: An Oral History of Haden's Syndrome. In each, he introduces readers to a world that's experienced a medical catastrophe: a flu pandemic infected and killed millions of people around the world, and left some of the survivors with Haden's Syndrome, a condition that left them "locked in" to their bodies.

This is an example of a probability distribution because it specifies the probability for each possible state of the random variable murderer. In this case the distribution is over a random variable which has two possible states. We will sometimes use the notation to denote the distribution over the random variable murderer. This can be viewed as a shorthand notation for the combination of and . At this point it is useful to introduce a pictorial representation of a probability distribution that will help to explain some of the later calculations.