ChaosBench is published under the open source GNU General Public License. Further development and potential updates discussed in the limitations section will take place on the ChaosBench page. Furthermore, we are committed to maintaining and preserving the ChaosBench benchmark. Ongoing maintenance also includes tracking and resolving issues identified by the broader community after release. User feedback will be closely monitored via the GitHub issue tracker. All assets are hosted on GitHub and HuggingFace, which guarantees reliable and stable storage. Dataset: All our dataset, present and future (e.g., with more years, multi-resolution support, etc) are available at https://huggingface.co/datasets/LEAP/ChaosBench.

Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster preparedness and robust decision making amidst climate change. Yet, forecasting beyond the weather timescale is challenging because it deals with problems other than initial condition, including boundary interaction, butterfly effect, and our inherent lack of physical understanding. At present, existing benchmarks tend to have shorter forecasting range of up-to 15 days, do not include a wide range of operational baselines, and lack physics-based constraints for explainability. Thus, we propose ChaosBench, a challenging benchmark to extend the predictability range of data-driven weather emulators to S2S timescale. First, ChaosBench is comprised of variables beyond the typical surface-atmospheric ERA5 to also include ocean, ice, and land reanalysis products that span over 45 years to allow for full Earth system emulation that respects boundary conditions. We also propose physics-based, in addition to deterministic and probabilistic metrics, to ensure a physically-consistent ensemble that accounts for butterfly effect. Furthermore, we evaluate on a diverse set of physics-based forecasts from four national weather agencies as baselines to our data-driven counterpart such as ViT/ClimaX, PanguWeather, GraphCast, and FourCastNetV2. Overall, we find methods originally developed for weather-scale applications fail on S2S task: their performance simply collapse to an unskilled climatology. Nonetheless, we outline and demonstrate several strategies that can extend the predictability range of existing weather emulators, including the use of ensembles, robust control of error propagation, and the use of physics-informed models.

ChaosBench is published under the open source GNU General Public License. Further development and potential updates discussed in the limitations section will take place on the ChaosBench page. Furthermore, we are committed to maintaining and preserving the ChaosBench benchmark. Ongoing maintenance also includes tracking and resolving issues identified by the broader community after release. User feedback will be closely monitored via the GitHub issue tracker. All assets are hosted on GitHub and HuggingFace, which guarantees reliable and stable storage. Dataset: All our dataset, present and future (e.g., with more years, multi-resolution support, etc) are available at https://huggingface.co/datasets/LEAP/ChaosBench.

An AI weather program running for a single second on a desktop can match the accuracy of traditional forecasts that take hours or days on powerful supercomputers, claim its creators. Weather forecasting has, since the 1950s, relied on physics-based models that extrapolate from observations made using satellites, balloons and weather stations. But these calculations, known as numerical weather prediction (NWP), are extremely intensive and rely on vast, expensive and energy-hungry supercomputers. Microsoft has a new quantum computer – but does it actually work? In recent years, researchers have tried to streamline this process by applying AI.

A single researcher with a desktop computer will be able to deliver accurate weather forecasts using a new AI weather prediction approach that is tens of times faster and uses thousands of times less computing power than conventional systems. Weather forecasts are currently generated through a complex set of stages, each taking several hours to run on bespoke supercomputers, requiring large teams of experts to develop, maintain and deploy them. Aardvark Weather provides a blueprint to replace the entire process by training an AI on raw data from weather stations, satellites, weather balloons, ships and planes from around the world to enable it to make predictions. This offers the potential for vast improvements in forecast speed, accuracy and cost, according to research published on Thursday in Nature from the University of Cambridge, the Alan Turing Institute, Microsoft Research and the European Centre for Medium-Range Weather Forecasts (ECMWF). Richard Turner, a professor of machine learning at the University of Cambridge, said the approach could be used to quickly provide bespoke forecasts for specific industries or locations, for example predicting temperatures for African agriculture or wind speeds for a renewable energy company in Europe.

Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster preparedness and robust decision making amidst climate change. Yet, forecasting beyond the weather timescale is challenging because it deals with problems other than initial condition, including boundary interaction, butterfly effect, and our inherent lack of physical understanding. At present, existing benchmarks tend to have shorter forecasting range of up-to 15 days, do not include a wide range of operational baselines, and lack physics-based constraints for explainability. Thus, we propose ChaosBench, a challenging benchmark to extend the predictability range of data-driven weather emulators to S2S timescale. First, ChaosBench is comprised of variables beyond the typical surface-atmospheric ERA5 to also include ocean, ice, and land reanalysis products that span over 45 years to allow for full Earth system emulation that respects boundary conditions. We also propose physics-based, in addition to deterministic and probabilistic metrics, to ensure a physically-consistent ensemble that accounts for butterfly effect. Furthermore, we evaluate on a diverse set of physics-based forecasts from four national weather agencies as baselines to our data-driven counterpart such as ViT/ClimaX, PanguWeather, GraphCast, and FourCastNetV2. Overall, we find methods originally developed for weather-scale applications fail on S2S task: their performance simply collapse to an unskilled climatology. Nonetheless, we outline and demonstrate several strategies that can extend the predictability range of existing weather emulators, including the use of ensembles, robust control of error propagation, and the use of physics-informed models.

Given the interpretability, accuracy, and stability of numerical weather prediction (NWP) models, current operational weather forecasting relies heavily on the NWP approach. In the past two years, the rapid development of Artificial Intelligence (AI) has provided an alternative solution for medium-range (1-10 days) weather forecasting. Bi et al. (2023) (hereafter Bi23) introduced the first AI-based weather prediction (AIWP) model in China, named Pangu-Weather, which offers fast prediction without compromising accuracy. In their work, Bi23 made notable claims regarding its effectiveness in extreme weather predictions. However, this claim lacks persuasiveness because the extreme nature of the two tropical cyclones (TCs) examples presented in Bi23, namely Typhoon Kong-rey and Typhoon Yutu, stems primarily from their intensities rather than their moving paths. Their claim may mislead into another meaning which is that Pangu-Weather works well in predicting unusual typhoon paths, which was not explicitly analyzed. Here, we reassess Pangu-Weather's ability to predict extreme TC trajectories from 2020-2024. Results reveal that while Pangu-Weather overall outperforms NWP models in predicting tropical cyclone (TC) tracks, it falls short in accurately predicting the rarely observed sudden-turning tracks, such as Typhoon Khanun in 2023. We argue that current AIWP models still lag behind traditional NWP models in predicting such rare extreme events in medium-range forecasts.

We introduce GraphDOP, a new data-driven, end-to-end forecast system developed at the European Centre for Medium-Range Weather Forecasts (ECMWF) that is trained and initialised exclusively from Earth System observations, with no physics-based (re)analysis inputs or feedbacks. GraphDOP learns the correlations between observed quantities - such as brightness temperatures from polar orbiters and geostationary satellites - and geophysical quantities of interest (that are measured by conventional observations), to form a coherent latent representation of Earth System state dynamics and physical processes, and is capable of producing skilful predictions of relevant weather parameters up to five days into the future.

When Helene made landfall in Florida earlier this year, 234 people lost their lives to the worst hurricane to strike the US mainland since Katarina in 2005. It's natural disasters like that, and their growing intensity due to climate change, that have pushed scientists to develop more accurate weather forecasting systems. On Wednesday, Google's DeepMind division announced what may go down as the most significant advancement in the field in nearly eight decades of work. According to DeepMind, GenCast is not only better at providing daily and extreme weather forecasts than its previous AI weather program, but it also outperforms the best forecasting system in use right now, one that's maintained by the European Center for Medium-Range Weather Forecasts (ECMWF). In tests comparing the 15-day forecasts the two systems generated for weather in 2019, GenCast was, on average, more accurate than ECMWF's ENS system 97.2 percent of the time.

Today's weather forecasts rely on simulations that require a lot of computing power Google DeepMind claims its latest weather forecasting AI can make predictions faster and more accurately than existing physics-based simulations. GenCast is the latest in DeepMind's ongoing research project to use artificial intelligence to improve weather forecasting. The model was trained on four decades of historical data from the European Centre for Medium-Range Weather Forecasts's (ECMWF) ERA5 archive, which includes regular measurements of temperature, wind speed and pressure at various altitudes around the globe. Data up to 2018 was used to train the model and then data from 2019 was used to test its predictions against known weather. The company found that it beat ECMWF's industry-standard ENS forecast 97.4 per cent of the time in total, and 99.8 per cent of the time when looking ahead more than 36 hours.