Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 3,000 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

We consider the general problem of online convex optimization with time-varying additive constraints in the presence of predictions for the next cost and constraint functions. A novel primal-dual algorithm is designed by combining a Follow-The-Regularized-Leader iteration with prediction-adaptive dynamic steps. The algorithm achieves $\mathcal O(T^{\frac{3-\beta}{4}})$ regret and $\mathcal O(T^{\frac{1+\beta}{2}})$ constraint violation bounds that are tunable via parameter $\beta\!\in\![1/2,1)$ and have constant factors that shrink with the predictions quality, achieving eventually $\mathcal O(1)$ regret for perfect predictions. Our work extends the FTRL framework for this constrained OCO setting and outperforms the respective state-of-the-art greedy-based solutions, without imposing conditions on the quality of predictions, the cost functions or the geometry of constraints, beyond convexity.

Department of Computer Science and Statistics Trinity College Dublin Ireland September 2019 Abstract Recently Jaouad Mourtada and St ephane Ga ıffas showed the anytime hedge algorithm has pseudo-regret O (log( d)/) if the cost vectors are generated by an i.i.d sequence in the cube [0, 1] d . This is remarkable because the Hedge algorithm was designed for the antagonistic setting. We prove a similar result for the anytime subgradient algorithm on the simplex. Given i.i.d cost vectors in the unit ball our pseudo-regret bound is O (1 /) and does not depend on the dimension of the problem. 1 Introduction and Related Work Online convex optimisation comes from Zinkevich [2003] who gave the following toy example: Suppose at the start of the year we must choose which of two crops to plant. Bananas grow best in warm weather and potatoes grow best in cold weather. We may plant all bananas or all potatoes or mix the two in any proportion. The average temperature for the year determines which of the two crops grows better and thus the profit for that year. Suppose also we have two neighbors, one who plants only potatoes every year and one who plants only bananas. Zincevich gave algorithms whereby after N seasons the more succesful of the two neighborhours will be only O ( N) more succesful than us. Taking averages the difference is O (1/ N) which decays to zero. Formally suppose a 1,a 2,... R d are cost vectors. On turn n we know only a 1,...,a n 1 and must select an action x n in the d-simplex S with a mind to minimising the sum null N i 1a i · x i. There exist algorithms whereby the regretnull N i 1a i · x i null N i 1a i · x has order O ( N) for all x S simultaneously. Remarkably these algorithms work when the only assumption is a uniform bound for the cost vectors. To extend the metaphor, there is a planting strategy that is effective even if the weather is selected antagonistically by nature. 1 arXiv:1909.05007v1 One popular model for predictable weather is to assume a 1,a 2,... are generated from some i.i.d sequence 1 of random variables with expectation a . In this case a 1,a 2,...,a n 1 provide estimates for a n and there are strategies where for E[a i] a the pseudo-regret null N i 1a · x i null N i 1a · x has expectation O (log N). Such algorithms exist even in the bandit setting, where on turn n we know only the sequence of costs a 1 · x 1,a 2 · x 2,...,a n 1 · x n 1 rather than the full information setting where we know the cost vectors a 1,a 2,...,a n 1 themselves. The bandit setting comes from Flaxman et al. [2004].