Verifiable formal languages like Lean have profoundly impacted mathematical reasoning, particularly through the use of large language models (LLMs) for automated reasoning. A significant challenge in training LLMs for these formal languages is the lack of parallel datasets that align natural language with formal language proofs. To address this challenge, this paper introduces a novel framework for translating the Mathlib4 corpus (a unified library of mathematics in formal language Lean 4) into natural language. Building upon this, we employ a dual augmentation strategy that combines tactic-based and informal-based approaches, leveraging the Lean-jixia system, a Lean 4 analyzer. We present the results of this pipeline on Mathlib4 as Herald (Hierarchy and Retrieval-based Translated Lean Dataset). We also propose the Herald Translator, which is fine-tuned on Herald. Herald translator achieves a 93.2% accuracy (Pass@128) on formalizing statements in the miniF2F-test and a 22.5% accuracy on our internal graduate-level textbook dataset, outperforming InternLM2-Math-Plus-7B (74.0% and 7.5%) and TheoremLlama (50.1% and 4.0%). Furthermore, we propose a section-level translation framework for real-world applications. As a direct application of Herald translator, we have successfully translated a template section in the Stack project, marking a notable progress in the automatic formalization of graduate-level mathematical literature. Our model, along with the datasets, will be open-sourced to the public soon.

Hearst has become the latest major US publisher to sign an agreement to license its content to ChatGPT creator OpenAI. As part of a partnership announced on Tuesday, content from more than 60 Hearst-owned publications will appear in ChatGPT and other OpenAI products. Some of the publisher's more notable properties include Esquire, Cosmopolitan and Elle. It also owns newspapers like the San Francisco Chronicle. When Hearst content appears in ChatGPT, the software will provide citations and direct links.

A technical challenge of deep learning is recognizing target classes without seen data. Zero-shot learning leverages semantic representations such as attributes or class prototypes to bridge source and target classes. Existing standard zero-shot learning methods may be prone to overfitting the seen data of source classes as they are blind to the semantic representations of target classes. In this paper, we study generalized zero-shot learning that assumes accessible to target classes for unseen data during training, and prediction on unseen data is made by searching on both source and target classes. We propose a novel Deep Calibration Network (DCN) approach towards this generalized zero-shot learning paradigm, which enables simultaneous calibration of deep networks on the confidence of source classes and uncertainty of target classes.

There has been a string of articles recently about the end of generative AI hype. Our experts consider whether or not the bubble has burst. Joining the conversation this time are: Tom Dietterich (Oregon State University), Sabine Hauert (University of Bristol), Michael Littman (Brown University), and Marija Slavkovik (University of Bergen). Sabine Hauert: There have been a number of recent articles in the mainstream media talking about the fact that AI has not made any money, and that it might be all hype, or a bubble. Marija Slavkovik: There is this article by Cory Doctorow which asks what kind of bubble AI is. I really like his take that a lot of bubbles come and go; some of them leave us something useful and some of them just generate something for a brief moment in time, like excellent revenue for the investment bankers for example.

Large language models can do jaw-dropping things. But nobody knows exactly why. Two years ago, Yuri Burda and Harri Edwards, researchers at OpenAI, were trying to find out what it would take to get a large language model to do basic arithmetic. The models memorized the sums they saw but failed to solve new ones. By accident, Burda and Edwards left some of their experiments running for days rather than hours.

To win the race, the US is going to need access to a lot more electric power to serve data centers. AI data centers could add the equivalent of three New York Cities' worth of load to the grid by 2026, and they could more than double their share of US electricity consumption--to 9%--by the end of the decade. Artificial intelligence will thus contribute to a spike in power demand that the US hasn't seen in decades; according to one recent estimate, that demand--previously flat--is growing by around 2.5% per year, with data centers driving as much as 66% of the increase. Energy-hungry advanced AI chips are behind this growth. Three watt-hours of electricity are required for a ChatGPT query, compared with just 0.3 watt-hours for a simple Google search.

The idea is to represent each class as its mean/prototype within a learned embedding space, and then recognising new classes via softmax over distances to the prototypes. The model is trained by randomly randomly sampling classes and instances per episode. This is appealing due to its simplicity and speed compared to other influential few-shot methodologies [21,28]. Some insights are given about the connection to mixture density estimation in the case of Bregman divergence-based distances, linear models, and matching networks. The same framework extends relatively straightforwardly to zero-shot learning by making the class prototype be the result of a learned mapping from meta-data like attributes to the prototype vector.

Post rebuttal: I now understand the middle ground this paper is positioned, and the difference to propositional OO representations where you don't necessarily care which instance of an object type you're dealing with, which significantly reduces the dimensionality of learning transition dynamics. But this is still similar to other work on graph neural networks for model learning in fully relational representations, like Relation Networks by Santoro et al., and Interaction Networks by Battaglia et al. which in worst case learn T * n * (n-1) relations for n objects for T types of relations. However, this paper does do a nice job of formalizing from the OO-MDP and Propositional MDP setting as opposed to the two papers I mentioned which do not, and focus on the physical dynamics case. I am willing to increase my score based on this, but still do not think it is novel enough to be accepted. This is very similar to relational MDPs, but they learn transition dynamics in this relational attribute space rather than real state space.

The authors present an algorithm for zero-shot learning based on learning a linear projection between the features of a pre-trained convnet and a semantic space in which to do nearest neighbors classification. Overall, I found the exposition of the paper very confusing, and I am still struggling to understand the exact set-up in the experiments and theorems after several re-readings. I'm not 100% certain from the paper what the authors are actually doing and what they have available as training data. In particular, the authors denote the test images by D_u, which they are sure to point out are unlabeled. They then define l_i {(u)} to be the label of test point x_i {(u)} - how can the label be included in a set of test images that is unlabeled?

Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.