This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

IDO DAGAN TITLE: Open Knowledge Graphs: Consolidating and Exploring Textual Information ABSTRACT: How can we capture effectively the information expressed in multiple texts? How can we allow people, as well as computer applications, to easily explore it? The current semantic NLP pipeline typically ends at the single sentence level, putting the burden on applications to consolidate related information that is spread across different texts. Further, semantic representations are often based on non-trivial pre-specified schemata, which require expert annotation and hence complicate the creation of large scale corpora for effective training. In this talk, I will outline a proposal for a novel open representation of the information exressed jointly by multiple texts, which we term Open Knowledge Graphs (OKG). First, we follow the spirit of "open" semantic approaches, such as Open Information Extraction (OIE) and more concretely the recent Question-Answer SRL (QA-SRL) paradigm, which represent semantic structure solely via natural language expressions.