This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.
We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.
"Drawing on your phone or computer can be slow and difficult -- so we created AutoDraw, a new web-based tool that pairs machine learning with drawings created by talented artists to help you draw," wrote Google Creative Lab's "creative technologist," Dan Motzenbecker, earlier this week. AutoDraw is one of Google's artificial intelligence (AI) experiments, working across platforms to let anyone, irrespective of their artistic flair, create something super quick with little more than a scribble. It guesses what you're trying to draw, then lets you pick from a list of previously created pictures. No worries!" is the general idea here. First up, AutoDraw is a super fun tool that gets increasingly addictive -- that much is clear. But what's also clear is that the tool is more a display of AI smarts than it is a tool to improve your artwork, because it would be just as easy to embody the exact same functionality within a text-based search engine. I mean, why bother drawing a crap dolphin with your finger when you could just type in the word "dolphin"? Because it wouldn't be nearly as much fun, and Google wouldn't get to show off its fancy new toys.