WeatherReal: A Benchmark Based on In-Situ Observations for Evaluating Weather Models

Accurate weather forecasting plays a vital role in saving lives, aiding emergency management, and reducing the economic impact of severe weather events [Bauer et al., 2015]. The traditional paradigm of weather forecasting is numerical weather prediction (NWP), which focuses on nonlinear partial differential equations to simulate atmospheric dynamics and physical processes [Benjamin et al., 2019]. In recent years, with the advancement of artificial intelligence (AI) technology and the continuous accumulation of massive weather data, data-driven methods have been increasingly incorporated into various stages and different scales of weather forecasting [Ravuri et al., 2021, Schultz et al., 2021, Weyn et al., 2021]. Particularly in the past two years, numerous data-driven models addressing the short to mediumrange (0-10 day) forecasting problem have emerged [Bi et al., 2023, Lam et al., 2023, Chen et al., 2023, Lang et al., 2024]. These models have surpassed the operational Integrated Forecast System (IFS) from European Centre for Medium-Range Weather Forecasts (ECMWF) in metrics such as Root Mean Square Error (RMSE) and Anomaly Correlation Coefficient (ACC). These breakthroughs have instilled confidence that data-driven models can be significant tools for enhancing the accuracy and computational efficiency of weather forecasting.