In this work, we introduce WinoGAViL: an online game of vision-andlanguage associations (e.g., betweenwerewolves and a full moon), used as a dynamic evaluation benchmark.
However, it is often argued that correct predictions in the tail are more "interesting" or "rewarding," but the community has not yet settled on a metric capturing this intuitive concept.