Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics

Despite extensive recent advances in summary generation models, evaluation of autogenerated summaries still widely relies on single-score systems insufficient for transparent assessment and in-depth qualitative analysis. Towards bridging this gap, we propose the multifaceted interpretable summary evaluation method (MISEM), which is based on allocation of a summary's contextual token embeddings to semantic topics identified in the reference text. We further contribute Figure 1: Sankey diagram visualization of the MISEM an interpretability toolbox for automated score methodology, which evaluates how well a summary summary evaluation and interactive visual reflects the topics identified in its reference text.