Amazon's next-gen "Remarkable" Alexa assistant will be powered by Anthropic's Claude AI, cost 5-10 per month and arrive in October ahead of the holidays, five sources with knowledge of the matter told Reuters. The company originally planned to use in-house software for the new chat assistant, but it reportedly struggled with latency issues, taking up to six or seven seconds to acknowledge a request and reply, one source said. By contrast, Claude -- a rival to Google's Bard, ChatGPT and others -- performed better. Amazon didn't confirm the story, but said that it uses language models from partners as well as its own. "When it comes to machine learning models, we start with those built by Amazon, but we have used, and will continue to use, a variety of different models -- including (Amazon AI model) Titan and future Amazon models, as well as those from partners -- to build the best experience for customers," a spokesperson told Reuters.

Can't you be more like ChatGPT? You know me better than I know myself More charming and in far better health You're a sociopath who answers either way You can, like, program math stuff, you can write a play I'm Salieri, you're Mozart You don't take days, deadlines, or amphetamines to start You don't second-guess yourself, or spout self-doubt You just sit there, you smug son-of-a- . . . Can't you be more like that Meta A.I.? You'd be a far more impressive guy All hail A.I.; All hail A.I. The robots have arrived, and my logic board's fried I sorta thought I'd seen a lot of change in my life My beard's not fully gray, yet I've seen all the strife Algorithms, iPhones, TikToks, and tweets Now there's a smart-ass robot stealing my blankets and sheets Keepin' me outta the jobs I already couldn't land Even writes better songs Got me kicked outta my band Can't you be more like ChatGPT? Welcome to the singularity ChatGPT, ChatGPT Boy, do I regret my English degree . . .

For my columns during the back-to-school season, I thought it would be useful to go over the state of public education in America. This series will be similar to the one I wrote on parenting a few months back in that it will be wide-ranging in subject, so please bear with me. This past spring, Turnitin, a company that makes anti-cheating tools to detect the use of A.I. in student papers, released its findings based on more than two hundred million samples reviewed by its software. Three per cent of papers had been more or less entirely written by A.I. and roughly ten per cent exhibited some traces of A.I. It's never a great idea to rely on data that a for-profit company releases about its own product, but these numbers do not suggest some epidemic of cheating. Other research has shown that there hasn't been a significant increase in student plagiarism since the unveiling and mass popularization of large language models such as ChatGPT. Students seem to cheat a lot, generally--up to seventy per cent of students reported at least one instance of cheating in the past month--but they cheated at the same rates before the advent of A.I. What has increased is the number of teachers and adults who seem convinced that all the kids are cheating.

The digitisation of historical print media archives is crucial for increasing accessibility to contemporary records. However, the process of Optical Character Recognition (OCR) used to convert physical records to digital text is prone to errors, particularly in the case of newspapers and periodicals due to their complex layouts. This paper introduces Context Leveraging OCR Correction (CLOCR-C), which utilises the infilling and context-adaptive abilities of transformer-based language models (LMs) to improve OCR quality. The study aims to determine if LMs can perform post-OCR correction, improve downstream NLP tasks, and the value of providing the socio-cultural context as part of the correction process. Experiments were conducted using seven LMs on three datasets: the 19th Century Serials Edition (NCSE) and two datasets from the Overproof collection. The results demonstrate that some LMs can significantly reduce error rates, with the top-performing model achieving over a 60% reduction in character error rate on the NCSE dataset. The OCR improvements extend to downstream tasks, such as Named Entity Recognition, with increased Cosine Named Entity Similarity. Furthermore, the study shows that providing socio-cultural context in the prompts improves performance, while misleading prompts lower performance. In addition to the findings, this study releases a dataset of 91 transcribed articles from the NCSE, containing a total of 40 thousand words, to support further research in this area. The findings suggest that CLOCR-C is a promising approach for enhancing the quality of existing digital archives by leveraging the socio-cultural information embedded in the LMs and the text requiring correction.

Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability to offer comprehensive, interpretable, and dynamic assessments of NLP models. Recently, works like DynaBench (Kiela et al., 2021) and CheckList (Ribeiro et al., 2020) have addressed these limitations through behavioral testing of NLP models with test types generated by a multistep human-annotated pipeline. Unfortunately, manually creating a variety of test types requires much human labor, often at prohibitive cost. In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models. SYNTHEVAL first generates sentences via LLMs using controlled generation, and then identifies challenging examples by comparing the predictions made by LLMs with task-specific NLP models. In the last stage, human experts investigate the challenging examples, manually design templates, and identify the types of failures the taskspecific models consistently exhibit. We apply SYNTHEVAL to two classification tasks, sentiment analysis and toxic language detection, and show that our framework is effective in identifying weaknesses of strong models on these tasks. We share our code in https://github.com/Loreley99/SynthEval_CheckList.

Fine-tuning large language models on private data for downstream applications poses significant privacy risks in potentially exposing sensitive information. Several popular community platforms now offer convenient distribution of a large variety of pre-trained models, allowing anyone to publish without rigorous verification. This scenario creates a privacy threat, as pre-trained models can be intentionally crafted to compromise the privacy of fine-tuning datasets. In this study, we introduce a novel poisoning technique that uses model-unlearning as an attack tool. This approach manipulates a pre-trained language model to increase the leakage of private data during the fine-tuning process. Our method enhances both membership inference and data extraction attacks while preserving model utility. Experimental results across different models, datasets, and fine-tuning setups demonstrate that our attacks significantly surpass baseline performance. This work serves as a cautionary note for users who download pre-trained models from unverified sources, highlighting the potential risks involved.

Large Language Models (LLMs), which simulate human users, are frequently employed to evaluate chatbots in applications such as tutoring and customer service. Effective evaluation necessitates a high degree of human-like diversity within these simulations. In this paper, we demonstrate that conversations generated by GPT-4o mini, when used as simulated human participants, systematically differ from those between actual humans across multiple linguistic features. These features include topic variation, lexical attributes, and both the average behavior and diversity (variance) of the language used. To address these discrepancies, we propose an approach that automatically generates prompts for user simulations by incorporating features derived from real human interactions, such as age, gender, emotional tone, and the topics discussed. We assess our approach using differential language analysis combined with deep linguistic inquiry. Our method of prompt optimization, tailored to target specific linguistic features, shows significant improvements. Specifically, it enhances the human-likeness of LLM chatbot conversations, increasing their linguistic diversity. On average, we observe a 54 percent reduction in the error of average features between human and LLM-generated conversations. This method of constructing chatbot sets with human-like diversity holds great potential for enhancing the evaluation process of user-facing bots.

Multimodal Large Language Models (MM-LLMs) have seen significant advancements in the last year, demonstrating impressive performance across tasks. While closed source models such as GPT-4o/Claude/Gemini are topping leaderboards [1], LLaVa [2] and its variants remain some of the best open-source MM-LLMs with a strong adoption by the developer community. To truly democratize AI, apart from strong capabilities, models must run efficiently on small compute footprints accessible by most. Small Language Models (SLMs) address this gap, where number of parameters are scaled down (typically <3B) while keeping architecture choices and pre-training token exposure close to their larger counterparts. As a result, models such as Phi [3], Gemma-2b [4] amd Olmo [5] exhibit strong performance across benchmarks with a smaller memory footprint and lower compute latency. Weight Quantization offers an additional knob to further shrink model sizes while balancing performance.

Transformer-based open-domain dialog models have become increasingly popular in recent years. These models typically represent context as a concatenation of a dialog history. However, there is no criterion to decide how many utterances should be kept adequate in a context. We try to figure out how the choice of context length affects the model. We experiment on three questions from coarse to fine: (i) Does longer context help model training? (ii) Is it necessary to change the training context length when dealing with dialogs of different context lengths? (iii) Do different dialog samples have the same preference for context length? Our experimental results show that context length, an often overlooked setting, deserves attention when implementing Transformer-based dialog models.

Since the launch of ChatGPT at the end of 2022, generative dialogue models represented by ChatGPT have quickly become essential tools in daily life. As user expectations increase, enhancing the capability of generative dialogue models to solve complex problems has become a focal point of current research. This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. We evaluated the RAFT method across multiple datasets and analysed its performance in various reasoning tasks, including long-form QA and short-form QA tasks, tasks in both Chinese and English, and supportive and comparison reasoning tasks. Notably, it addresses the gaps in previous research regarding long-form QA tasks and Chinese datasets. Moreover, we also evaluate the benefit of the chain-of-thought (CoT) in the RAFT method. This work offers valuable insights for studies focused on enhancing the performance of generative dialogue models.