Large Language Model
How much power and water does AI use? Google, Mistral weigh in
How badly does AI harm the environment? We now have some answers to that question, as both Google and Mistral have published their own self-assessments of the environmental impact of an AI query. In July, Mistral, which publishes its own AI models, published a self-evaluation of the environmental impact of training and querying its model in terms of the amount of carbon dioxide (CO2) produced, the amount of water consumed, and the amount of material consumed. Google took a slightly different approach, publishing the amount of power and water a Gemini query consumes, as well as how much CO2 it produces. Of course, there are caveats: Each report was self-generated, and not performed by an outside auditor.
OpenAI limits ChatGPT's role in mental health help
Mothers Against Media Addiction executive director Julie Scelfo joins'Fox & Friends First' to discuss the impact of screen time on kids' mental health and development as lawmakers are set to examine how screens impact learning in the classroom. More people are turning to artificial intelligence for support, even for mental health advice. It's easy to see why: tools like ChatGPT are free, fast, and always available. But mental health is a delicate issue, and AI isn't equipped to handle the complexities of real emotional distress. To address growing concerns, OpenAI has introduced new safety measures for ChatGPT.
In a first, Google has released data on how much energy an AI prompt uses
Earlier this year, MIT Technology Review published a comprehensive series on AI and energy, at which time none of the major AI companies would reveal their per-prompt energy usage. Google's new publication, at last, allows for a peek behind the curtain that researchers and analysts have long hoped for. The study focuses on a broad look at energy demand, including not only the power used by the AI chips that run models but also by all the other infrastructure needed to support that hardware. "We wanted to be quite comprehensive in all the things we included," said Jeff Dean, Google's chief scientist, in an exclusive interview with MIT Technology Review about the new report. Another large portion of the energy is used by equipment needed to support AI-specific hardware: The host machine's CPU and memory account for another 25% of the total energy used.
Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models
Zhang, Liyi, Snell, Jake, Griffiths, Thomas L.
Fine-tuning large language models (LLMs) with low-rank adaptaion (LoRA) is a cost-effective way to incorporate information from a specific dataset. However, it is often unclear how well the fine-tuned LLM will generalize, i.e., how well it will perform on unseen datasets. Methods have been proposed to improve generalization by optimizing with in-context prompts, or by using meta-learning to fine-tune LLMs. However, these methods are expensive in memory and computation, requiring either long-context prompts or saving copies of parameters and using second-order gradient updates. To address these challenges, we propose Amortized Bayesian Meta-Learning for LoRA (ABMLL). This method builds on amortized Bayesian meta-learning for smaller models, adapting this approach to LLMs while maintaining its computational efficiency. We reframe task-specific and global parameters in the context of LoRA and use a set of new hyperparameters to balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. ABMLL provides effective generalization and scales to large models such as Llama3-8B. Furthermore, as a result of using a Bayesian framework, ABMLL provides improved uncertainty quantification. We test ABMLL on Unified-QA and CrossFit datasets and find that it outperforms existing methods on these benchmarks in terms of both accuracy and expected calibration error.
CRED-SQL: Enhancing Real-world Large Scale Database Text-to-SQL Parsing through Cluster Retrieval and Execution Description
Duan, Shaoming, Wang, Zirui, Liu, Chuanyi, Zhu, Zhibin, Zhang, Yuhao, Han, Peiyi, Yan, Liang, Peng, Zewu
Recent advances in large language models (LLMs) have significantly improved the accuracy of Text-to-SQL systems. However, a critical challenge remains: the semantic mismatch between natural language questions (NLQs) and their corresponding SQL queries. This issue is exacerbated in large-scale databases, where semantically similar attributes hinder schema linking and semantic drift during SQL generation, ultimately reducing model accuracy. To address these challenges, we introduce CRED-SQL, a framework designed for large-scale databases that integrates Cluster Retrieval and Execution Description. CRED-SQL first performs cluster-based large-scale schema retrieval to pinpoint the tables and columns most relevant to a given NLQ, alleviating schema mismatch. It then introduces an intermediate natural language representation-Execution Description Language (EDL)-to bridge the gap between NLQs and SQL. This reformulation decomposes the task into two stages: Text-to-EDL and EDL-to-SQL, leveraging LLMs' strong general reasoning capabilities while reducing semantic deviation. Extensive experiments on two large-scale, cross-domain benchmarks-SpiderUnion and BirdUnion-demonstrate that CRED-SQL achieves new state-of-the-art (SOTA) performance, validating its effectiveness and scalability. Our code is available at https://github.com/smduan/CRED-SQL.git
STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples
Hu, Haiquan, Jiang, Jiazhi, Xu, Shiyou, Zeng, Ruhan, Wang, Tian
Evaluating large language models (LLMs) has become increasingly challenging as model capabilities advance rapidly. While recent models often achieve higher scores on standard benchmarks, these improvements do not consistently reflect enhanced real-world reasoning capabilities. Moreover, widespread overfitting to public benchmarks and the high computational cost of full evaluations have made it both expensive and less effective to distinguish meaningful differences between models. To address these challenges, we propose the \textbf{S}tructured \textbf{T}ransition \textbf{E}valuation \textbf{M}ethod (STEM), a lightweight and interpretable evaluation framework for efficiently estimating the relative capabilities of LLMs. STEM identifies \textit{significant transition samples} (STS) by analyzing consistent performance transitions among LLMs of the same architecture but varying parameter scales. These samples enable STEM to effectively estimate the capability position of an unknown model. Qwen3 model family is applied to construct the STS pool on six diverse and representative benchmarks. To assess generalizability. Experimental results indicate that STEM reliably captures performance trends, aligns with ground-truth rankings of model capability. These findings highlight STEM as a practical and scalable method for fine-grained, architecture-agnostic evaluation of LLMs.