Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose Blockwise Learning Rate (LR), a strategy that tailors the LR to each block's sharpness, accelerating large language model (LLM) pre-training. By integrating Blockwise LR into AdamW, we consistently achieve lower terminal loss and nearly $2\times$ speedup compared to vanilla AdamW. We demonstrate this acceleration across GPT-2 and LLaMA, with model sizes ranging from 0.12B to 1.1B and datasets of OpenWebText and MiniPile. Finally, we incorporate Blockwise LR into Adam-mini (Zhang et al., 2024), a recently proposed memory-efficient variant of Adam, achieving a combined $2\times$ speedup and $2\times$ memory saving. These results underscore the potential of exploiting the sharpness disparity to improve LLM training.

Microsoft announced that it is making some features available for free in its Copilot AI assistant. Everyone now has unlimited access to Voice and Think Deeper, which is powered by OpenAI's o1 model. Copilot got the Voice feature, which allows users to have conversations with the AI assistant, in October 2024. Think Deeper is intended to parse complicated queries, such as assessing the pros and cons of major home purchases, taking cost and long-term value into account. "We are working hard to scale unlimited access to advanced features to as many people as possible, as quickly as possible," the blog post noted.

When OpenAI announced Deep Research at start of February, the company promised to bring the tool to Plus users "in about a month," and now it's doing exactly that. Starting today, the feature, which you can use to prompt ChatGPT to create in-depth reports on nearly any subject, is rolling out to Plus, Team, Edu and Enterprise users. Previously, you needed a 200 per month Pro plan to try out Deep Research. For the time being, Plus users will get 10 Deep Research queries per month included with their plan. For Pro subscribers, OpenAI is increasing the monthly limit to 120, up from 100 previously.

I'm a psychologist, and AI is coming for my job. The signs are everywhere: a client showing me how ChatGPT helped her better understand her relationship with her parents; a friend ditching her in-person therapist to process anxiety with Claude; a startup raising 40 million to build a super-charged-AI-therapist. The other day on TikTok, I came across an influencer sharing how she doesn't need friends; she can just vent to God and ChatGPT. "ChatGPT talked me out of self-sabotaging." "It knows me better than any human walking this earth."

Remember when AI came onto the scene and shook, well, everybody? Most people were fascinated by it, and others thought (or wished) the AI hype would die out eventually. The fact of the matter is AI is here to stay, and it might be time to start learning how to use it to your advantage. Just imagine saving time at work--that's what AI can help you do, as well as attending meetings for you and checking off your never-ending list of to-dos from your boss. The only downside is if you're new to the tool, you probably don't know which platform to use.

A Bangladeshi worker was eager to get started at their new OpenAI job--completing basic online tasks in exchange for consistent income, while getting into cryptocurrency investing at the same time. After connecting with the startup on Telegram and creating an account through a ChatGPT-branded app, they invested crypto into the platform and began a months-long job working for "Aiden" from "OpenAI." The work was performed through the website "OpenAi-etc," and internal conversations were held on Telegram. It was simple: Invest some crypto, complete a few tasks, and earn daily profits based on what was invested. Over the course of this worker's time with the company, mentors continuously encouraged them to invest more money into the fund and recruit more Bangladeshi people to the team.

DeepSeek is looking to press home its advantage. The Chinese startup triggered a 1 trillion-plus sell-off in global equities markets last month with a cut-price AI reasoning model that outperformed many Western competitors. Now, the Hangzhou-based firm is accelerating the launch of the successor to January's R1 model, according to three people familiar with the company. Deepseek had planned to release R2 in early May but now wants it out as early as possible, two of them said, without providing specifics. The company says it hopes the new model will produce better coding and be able to reason in languages beyond English.

These authors contributed equally: Y un Hao, Che Fan. Here, we introduce AlloyGAN, a closed-loop framework that integrates Large Language Model (LLM)-assisted text mining with Conditional Generative Adversarial Networks (CGANs) to enhance data diversity and improve inverse design. For metallic glasses, the framework predicts thermodynamic properties with discrepancies of less than 8% from experiments, demonstrating its robustness. By bridging generative AI with domain knowledge and validation workflows, AlloyGAN offers a scalable approach to accelerate the discovery of materials with tailored properties, paving the way for broader applications in materials science. Materials design typically involves two fundamental problems: forward and inverse problems. The forward problem focuses on understanding the relationship between composition, processing conditions, and material properties. This understanding enables researchers to optimize alloy compositions and processing conditions to achieve enhanced performance. Conversely, the inverse problem is more prevalent in material design and poses the question: "Given the desired material properties, what composition and processing conditions are required to achieve them?" The inverse problem is particularly challenging for multi-component materials due to the vast composition space and complex interactions among components. Traditional "trial-and-error" experimental approaches are often prohibitively time-consuming and cost-ineffective [1] for such problems. Addressing these challenges thus requires innovative approaches to efficiently navigate the composition space and identify optimal solutions for materials design.

Workload traces are essential to understand complex computer systems' behavior and manage processing and memory resources. Since real-world traces are hard to obtain, synthetic trace generation is a promising alternative. This paper proposes a first-of-a-kind approach that relies on training a large language model (LLM) to generate synthetic workload traces, specifically microservice call graphs. To capture complex and arbitrary hierarchical structures and implicit constraints in such traces, we show how to fine-tune LLMs to generate recursively, making call graph generation a sequence of easier steps. To further enforce learning constraints in traces and generate uncommon situations, we argue for applying additional instruction tuning steps to align our model with the desired trace features. Our evaluation results show that we can generate diverse realistic traces under various conditions and outperform existing methods in accuracy and validity. We demonstrate that our synthetically generated traces can effectively replace real data to optimize important microservice management tasks. Additionally, our model adapts to downstream trace-related tasks, such as predicting key trace features and infilling missing data.

Retrieval-Augmented Generation (RAG) enhances the performance of Large Language Models (LLMs) by incorporating external knowledge. However, LLMs still encounter challenges in effectively utilizing the knowledge from retrieved documents, often being misled by irrelevant or noisy information. To address this issue, we introduce RankCoT, a knowledge refinement method that incorporates reranking signals in generating CoT-based summarization for knowledge refinement based on given query and all retrieval documents. During training, RankCoT prompts the LLM to generate Chain-of-Thought (CoT) candidates based on the query and individual documents. It then fine-tunes the LLM to directly reproduce the best CoT from these candidate outputs based on all retrieved documents, which requires LLM to filter out irrelevant documents during generating CoT-style summarization. Additionally, RankCoT incorporates a self-reflection mechanism that further refines the CoT outputs, resulting in higher-quality training data. Our experiments demonstrate the effectiveness of RankCoT, showing its superior performance over other knowledge refinement models. Further analysis reveals that RankCoT can provide shorter but effective refinement results, enabling the generator to produce more accurate answers. All code and data are available at https://github.com/NEUIR/RankCoT.