ABSTRACT This report analyzes the potential for large language models (LLMs) to expedite accurate replication of published message effects studies. We tested LLM-powered participants (personas) by replicating 133 experimental findings from 14 papers containing 45 recent studies in the Journal of Marketing (January 2023-May 2024). We used a new software tool, Viewpoints AI (https://viewpoints.ai/), that takes study designs, stimuli, and measures as input, automatically generates prompts for LLMs to act as a specified sample of unique personas, and collects their responses to produce a final output in the form of a complete dataset and statistical analysis. The underlying LLM used was Anthropic's Claude Sonnet 3.5. We generated 19,447 AI personas to replicate these studies with the exact same sample attributes, study designs, stimuli, and measures reported in the original human research. Our LLM replications successfully reproduced 76% of the original main effects (84 out of 111), demonstrating strong potential for AI-assisted replication of studies in which people respond to media stimuli. When including interaction effects, the overall replication rate was 68% (90 out of 133). The use of LLMs to replicate and accelerate marketing research on media effects is discussed with respect to the replication crisis in social science, potential solutions to generalizability problems in sampling subjects and experimental conditions, and the ability to rapidly test consumer responses to various media stimuli. We also address the limitations of this approach, particularly in replicating complex interaction effects in media response studies, and suggest areas for future research and improvement in AI-assisted experimental replication of media effects. STUDY OVERVIEW AND RELATED WORK Research about the effectiveness of media messages is increasingly difficult, attributable to both administrative challenges (e.g., stimulus acquisition and creation, data management demands of digital trace data, acquisition of participants and especially those in special groups like children, minorities and international groups), as well as requirements to deal with new and critical challenges to the very nature of social research, as exemplified by existential issues of replication and reproducibility, and the ability to generalize findings across people, media stimuli and experimental contexts. We briefly review these issues with an eye toward our current test of whether new LLM tools may help solve the problems mentioned, and with significant advantages in cost, time, and research personnel.

Registered reports have been proposed as a way to move from eye-catching and surprising results and toward methodologically sound practices and interesting research questions. However, none of the top-twenty artificial intelligence journals support registered reports, and no traces of registered reports can be found in the field of artificial intelligence. Is this because they do not provide value for the type of research that is conducted in the field of artificial intelligence? Registered reports have been touted as one of the solutions to the problems surrounding the reproducibility crisis. They promote good research practices and combat data dredging1.

Uncovering anomalies in attributed networks has recently gained popularity due to its importance in unveiling outliers and flagging adversarial behavior in a gamut of data and network science applications including {the Internet of Things (IoT)}, finance, security, to list a few. The present work deals with uncovering anomalous edges in attributed graphs using two distinct formulations with complementary strengths, which can be easily distributed, and hence efficient. The first relies on decomposing the graph data matrix into low rank plus sparse components to markedly improve performance. The second broadens the scope of the first by performing robust recovery of the unperturbed graph, which enhances the anomaly identification performance. The novel methods not only capture anomalous edges linking nodes of different communities, but also spurious connections between any two nodes with different features. Experiments conducted on real and synthetic data corroborate the effectiveness of both methods in the anomaly identification task.

Paris – Native Americans and Polynesians bridged vast expanses of open ocean around the year 1200 and mingled, leaving incontrovertible proof of their encounter in the DNA of present-day populations, scientists revealed Wednesday. Whether peoples from what is today Colombia or Ecuador drifted thousands of kilometers to tiny islands in the middle of the Pacific, or whether seafaring Polynesians sailed upwind to South America and then back again is still unknown. But what is certain, according to a study in Nature, is that the hook up took place hundreds of years before Europeans set foot in either region, and left individuals scattered across French Polynesia with signature traces of the New World in their DNA. "These findings change our understanding of one of the most unknown chapters in the history of our species' great continental expansions," senior author Andreas Moreno-Estrada, principal investigator at Mexico's National Laboratory of Genomics for biodiversity, said. Archeologists and historians have tussled for decades over whether Oceana islanders and native Americans crossed paths during the Middle Ages, and how, if they did, that contact might have unfolded.

Science has always hinged on the idea that researchers must be able to prove and reproduce the results of their research. Simply put, that is what makes science...science. Yet in recent years, as computing power has increased, the cloud has taken shape, and data sets have grown, a problem has appeared: it has becoming increasingly difficult to generate the same results consistently--even when researchers include the same dataset. "One basic requirement of scientific results is reproducibility: shake an apple tree, and apples will fall downwards each and every time," observes Kai Zhang, an associate professor in the department of statistics and operations research at The University of North Carolina, Chapel Hill. "The problem today is that in many cases, researchers cannot replicate existing findings in the literature and they cannot produce the same conclusions. This is undermining the credibility of scientists and science. It is producing a crisis."

In Part 1, I'll introduce the problem of bad medicine. As you review the evidence for it, you'll also get a good sense of why it's so entrenched. In Part 2, I'll examine the implications of the problem on artificial intelligence. In his history of Bad Medicine, David Wootton argued, "We can only think about medical progress if we start with the long tradition of medical failure."

Sometimes it seems surprising that science functions at all. In 2005, medical science was shaken by a paper with the provocative title "Why most published research findings are false."1 Written by John Ioannidis, a professor of medicine at Stanford University, it didn't actually show that any particular result was wrong. Instead, it showed that the statistics of reported positive findings was not consistent with how often one should expect to find them. As Ioannidis concluded more recently, "many published research findings are false or exaggerated, and an estimated 85 percent of research resources are wasted."2 It's likely that some researchers are consciously cherry-picking data to get their work published.

Editor's note: See also part 1 of 17 More Must-Know Data Science Interview Questions and Answers. Overfitting is when you build a predictive model that fits the data "too closely", so that it captures the random noise in the data rather than true patterns. As a result, the model predictions will be wrong when applied to new data. We frequently hear about studies that report unusual results (especially if you listen to Wait Wait Don't Tell Me), or see findings like "an orange used car is least likely to be a lemon", or learn that studies overturn previous established findings (eggs are no longer bad for you). Many such studies produce questionable results that cannot be repeated.

Predictive models ground many state-of-the-art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross-validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross-validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg $\pm$10% for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data.

Editor's note: See also part 1 of 17 More Must-Know Data Science Interview Questions and Answers. Overfitting is when you build a predictive model that fits the data "too closely", so that it captures the random noise in the data rather than true patterns. As a result, the model predictions will be wrong when applied to new data. We frequently hear about studies that report unusual results (especially if you listen to Wait Wait Don't Tell Me), or see findings like "an orange used car is least likely to be a lemon", or learn that studies overturn previous established findings (eggs are no longer bad for you). Many such studies produce questionable results that cannot be repeated.