Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset.

In a recent demo conducted over Zoom, I watched as the music composition app Dynascore transformed the entire emotional tenor of a short video multiple times in under a minute, all without altering a single frame of the visuals. What began in my short briefing as a very serious workout ad with a very serious soundtrack--something where you'd expect to see neon sweat pouring out of the athlete's head behind a Gatorade logo--quickly changed in tone to something a bit funnier. The machine intelligence engine inside Dynascore swapped out the action film theme music for Beethoven's somber Moonlight Sonata, suddenly transforming the video into a dark comedy. A few taps of the mouse on the other end of the Zoom window, a few seconds of rendering, and I was watching the same video with a modern pop song now layered over it, equally form-fitting to the burly close-ups on screen. This time, the result felt more like a music video.

We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards. On each task, models are ranked according to the Dynascore, a novel utility-based aggregation of these statistics, which users can customize to better reflect their preferences, placing more/less weight on a particular axis of evaluation or dataset. As state-of-the-art NLP models push the limits of traditional benchmarks, Dynaboard offers a standardized solution for a more diverse and comprehensive evaluation of model quality.