Rethinking Language Models as Symbolic Knowledge Graphs