We present TapeAgents, an agent framework built around a granular, structured log tape of the agent session that also plays the role of the session's resumable state. In TapeAgents we leverage tapes to facilitate all stages of the LLM Agent development lifecycle. The agent reasons by processing the tape and the LLM output to produce new thought and action steps and append them to the tape. The environment then reacts to the agent's actions by likewise appending observation steps to the tape. By virtue of this tape-centred design, TapeAgents can provide AI practitioners with holistic end-to-end support. At the development stage, tapes facilitate session persistence, agent auditing, and step-by-step debugging. Post-deployment, one can reuse tapes for evaluation, fine-tuning, and prompt-tuning; crucially, one can adapt tapes from other agents or use revised historical tapes. In this report, we explain the TapeAgents design in detail. We demonstrate possible applications of TapeAgents with several concrete examples of building monolithic agents and multi-agent teams, of optimizing agent prompts and finetuning the agent's LLM. We present tooling prototypes and report a case study where we use TapeAgents to finetune a Llama-3.1-8B form-filling assistant to perform as well as GPT-4o while being orders of magnitude cheaper. Lastly, our comparative analysis shows that TapeAgents's advantages over prior frameworks stem from our novel design of the LLM agent as a resumable, modular state machine with a structured configuration, that generates granular, structured logs and that can transform these logs into training text -- a unique combination of features absent in previous work.

Children across the US will likely spend the coming days and weeks saying goodbye to an AI-powered friend named Moxie. The small dog-sized companion bot--which used a ChatGPT-style large language model and expressive features to hold open-ended conversations with children--will soon be taken offline due to its creator's financial struggles. The decision to abandon the 799 product four years after its release, first reported by Aftermath, has left some customers bemoaning the loss of an artificial friend and others angrily demanding refunds. Videos of confused, crying children saying goodbye to their companion flooding social media. It's part of a larger trend of companies cutting off software support for hardware to cut costs.

The crowd-sourced review site Yelp unveiled a new feature that uses AI and customer reviews to rate common facets of nightlife and food-related business. The new Review Insights feature is available now on the iOS version of the Yelp app, according to the company's official blog. Review Insights aggregates customer reviews and feeds them into a large language model (LLM), which will assign specific aspects -- like the vibe or service time -- a rating out of 100. Supposedly it will be able to infer customer sentiment about these parts of a business "even when a review doesn't explicitly mention one of the topics." Yelp will also be adding an AI-powered homepage in the coming weeks.

OpenAI has now launched its new AI model called Sora, which can generate realistic videos from text-based prompts. The tool is available to both ChatGPT Plus and ChatGPT Pro subscribers via sora.com. ChatGPT Plus subscribers can generate up to 50 priority videos in 720p resolution up to five seconds long in duration, while ChatGPT Pro subscribers can generate unlimited videos with up to 500 priority videos in 1080p resolution up to 20 seconds long in duration. ChatGPT Pro users can also generate up to five videos simultaneously and download generated videos without watermarks on them. All videos generated via Sora will have C2PA metadata to indicate that they've been created using AI.

Sora, an artificial intelligence (AI) video generator program created by startup OpenAI in 2021, is making waves as it has now moved out of the research phase and has been officially released to the public under the new name of Sora Turbo. The launch has triggered an online frenzy among users, causing the company to temporarily halt new account creations after finding itself overwhelmed by an avalanche of traffic. Sora uses text prompts to create content, similar to other content creation programs such as ChatGPT. Unlike traditional AI programs which produce written responses, Sora creates high-quality videos based on a user's text input. Adding it to @everartai asap so you can bring your images to life pic.twitter.com/wMehxOc8cm

For more than two years, every new AI announcement has lived in the shadow of ChatGPT. No model from any company has eclipsed or matched that initial fever. But perhaps the closest any firm has come to replicating the buzz was this past February, when OpenAI first teased its video-generating AI model, Sora. Tantalizing clips--woolly mammoths kicking up clouds of snow, Pixar-esque animations of adorable fluffy critters--promised a stunning future, one in which anyone can whip up high-quality clips by typing simple text prompts into a computer program. But Sora, which was not immediately available to the public, remained just that: a teaser.

Document content extraction is crucial in computer vision, especially for meeting the high-quality data needs of large language models (LLMs) and retrieval-augmented generation (RAG) technologies. However, current document parsing methods suffer from significant limitations in terms of diversity and comprehensive evaluation. To address these challenges, we introduce OmniDocBench, a novel multi-source benchmark designed to advance automated document content extraction. OmniDocBench includes a meticulously curated and annotated high-quality evaluation dataset comprising nine diverse document types, such as academic papers, textbooks, slides, among others. Our benchmark provides a flexible and comprehensive evaluation framework with 19 layout category labels and 14 attribute labels, enabling multi-level assessments across entire datasets, individual modules, or specific data types. Using OmniDocBench, we perform an exhaustive comparative analysis of existing modular pipelines and multimodal end-to-end methods, highlighting their limitations in handling document diversity and ensuring fair evaluation. OmniDocBench establishes a robust, diverse, and fair evaluation standard for the document content extraction field, offering crucial insights for future advancements and fostering the development of document parsing technologies. The codes and dataset is available in https://github.com/opendatalab/OmniDocBench.

Large Language Models (LLMs) have demonstrated impressive capabilities in code completion tasks, where they assist developers by predicting and generating new code in real-time. However, existing LLM-based code completion systems primarily rely on the immediate context of the file being edited, often missing valuable repository-level information, user behaviour and edit history that could improve suggestion accuracy. Additionally, challenges such as efficiently retrieving relevant code snippets from large repositories, incorporating user behavior, and balancing accuracy with low-latency requirements in production environments remain unresolved. In this paper, we propose ContextModule, a framework designed to enhance LLM-based code completion by retrieving and integrating three types of contextual information from the repository: user behavior-based code, similar code snippets, and critical symbol definitions. By capturing user interactions across files and leveraging repository-wide static analysis, ContextModule improves the relevance and precision of generated code. We implement performance optimizations, such as index caching, to ensure the system meets the latency constraints of real-world coding environments. Experimental results and industrial practise demonstrate that ContextModule significantly improves code completion accuracy and user acceptance rates.

Metaphor serves as an implicit approach to convey information, while enabling the generalized comprehension of complex subjects. However, metaphor can potentially be exploited to bypass the safety alignment mechanisms of Large Language Models (LLMs), leading to the theft of harmful knowledge. In our study, we introduce a novel attack framework that exploits the imaginative capacity of LLMs to achieve jailbreaking, the J\underline{\textbf{A}}ilbreak \underline{\textbf{V}}ia \underline{\textbf{A}}dversarial Me\underline{\textbf{TA}} -pho\underline{\textbf{R}} (\textit{AVATAR}). Specifically, to elicit the harmful response, AVATAR extracts harmful entities from a given harmful target and maps them to innocuous adversarial entities based on LLM's imagination. Then, according to these metaphors, the harmful target is nested within human-like interaction for jailbreaking adaptively. Experimental results demonstrate that AVATAR can effectively and transferablly jailbreak LLMs and achieve a state-of-the-art attack success rate across multiple advanced LLMs. Our study exposes a security risk in LLMs from their endogenous imaginative capabilities. Furthermore, the analytical study reveals the vulnerability of LLM to adversarial metaphors and the necessity of developing defense methods against jailbreaking caused by the adversarial metaphor. \textcolor{orange}{ \textbf{Warning: This paper contains potentially harmful content from LLMs.}}

Thomas C. Schelling, awarded the 2005 Nobel Memorial Prize in Economic Sciences, pointed out that ``individuals decisions (micromotives), while often personal and localized, can lead to societal outcomes (macrobehavior) that are far more complex and different from what the individuals intended.'' The current research related to large language models' (LLMs') micromotives, such as preferences or biases, assumes that users will make more appropriate decisions once LLMs are devoid of preferences or biases. Consequently, a series of studies has focused on removing bias from LLMs. In the NLP community, while there are many discussions on LLMs' micromotives, previous studies have seldom conducted a systematic examination of how LLMs may influence society's macrobehavior. In this paper, we follow the design of Schelling's model of segregation to observe the relationship between the micromotives and macrobehavior of LLMs. Our results indicate that, regardless of the level of bias in LLMs, a highly segregated society will emerge as more people follow LLMs' suggestions. We hope our discussion will spark further consideration of the fundamental assumption regarding the mitigation of LLMs' micromotives and encourage a reevaluation of how LLMs may influence users and society.