This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.
We propose a novel neural topic model in the Wasserstein autoencoders (WAE) framework. Unlike existing variational autoencoder based models, we directly enforce Dirichlet prior on the latent document-topic vectors. We exploit the structure of the latent space and apply a suitable kernel in minimizing the Maximum Mean Discrepancy (MMD) to perform distribution matching. We discover that MMD performs much better than the Generative Adversarial Network (GAN) in matching high dimensional Dirichlet distribution. We further discover that incorporating randomness in the encoder output during training leads to significantly more coherent topics. To measure the diversity of the produced topics, we propose a simple topic uniqueness metric. Together with the widely used coherence measure NPMI, we offer a more wholistic evaluation of topic quality. Experiments on several real datasets show that our model produces significantly better topics than existing topic models.
The ability of reasoning beyond data fitting is substantial to deep learning systems in order to make a leap forward towards artificial general intelligence. A lot of efforts have been made to model neural-based reasoning as an iterative decision-making process based on recurrent networks and reinforcement learning. Instead, inspired by the consciousness prior proposed by Yoshua Bengio, we explore reasoning with the notion of attentive awareness from a cognitive perspective, and formulate it in the form of attentive message passing on graphs, called neural consciousness flow (NeuCFlow). Aiming to bridge the gap between deep learning systems and reasoning, we propose an attentive computation framework with a three-layer architecture, which consists of an unconsciousness flow layer, a consciousness flow layer, and an attention flow layer. We implement the NeuCFlow model with graph neural networks (GNNs) and conditional transition matrices. Our attentive computation greatly reduces the complexity of vanilla GNN-based methods, capable of running on large-scale graphs. We validate our model for knowledge graph reasoning by solving a series of knowledge base completion (KBC) tasks. The experimental results show NeuCFlow significantly outperforms previous state-of-the-art KBC methods, including the embedding-based and the path-based. The reproducible code can be found by the link below.
Mr. Smith Goes to Washington 1939 TCM Tue. 7 p.m. Mean Streets 1973 Cinemax Sun. 6 a.m. Batman Begins 2005 AMC Sun. Throw Momma From the Train 1987 EPIX Sun. Die Hard 1988 IFC Sun. I Know What You Did Last Summer 1997 Starz Tue. Gone in 60 Seconds 2000 CMT Wed. 8 p.m., Thur. Total Recall 1990 Encore Thur. 2 a.m. A Fish Called Wanda 1988 Encore Thur. 2 p.m., 9 p.m. The World Is Not Enough 1999 EPIX Sat. 4 p.m. Look Who's Talking 1989 OVA Sun. Die Hard With a Vengeance 1995 IFC Thur. Oil-platform workers, including an estranged couple, and a Navy SEAL make a startling deep-sea discovery. A clueless politician falls in love with a waitress whose erratic behavior is caused by a nail stuck in her head. After glimpsing his future, an ambitious politician battles the agents of Fate itself to be with the woman he loves. To help a friend, a suburban baby sitter drives into downtown Chicago with her two charges and a neighbor. Two teenage baby sitters and a group of children spend a wild night ...
I wanted to review one of Netflix's new, original movies this week, but after checking out The Discovery, Sandy Wexler, and Win It All, I was uninspired. In any case, I dug into some other, much brighter gems to highlight. How about a couple of horror movies that wouldn't be out of place at a 1970s grindhouse? How about a couple of sci-fi movies that are based on actual ideas rather than explosions? I could go on, but don't just take it from me. Peep the whole curated list of 13 movies below--and then start streaming.
Wall-E fell in love with another robot in the movie named after him. Researchers have yet to create a sentient machine, but a breakthrough could be on the horizon. Peter Remine says he will know it's time to get serious about rights for robots "when a robot knocks on my door asking for some help." Remine, founder of the Seattle-based American Society for the Prevention of Cruelty to Robots, says the moment will come when a robot in an automobile factory "will become sentient, realize that it doesn't want to do that unfulfilling and dangerous job anymore, and ask for protection under state workers' rights." Bit by bit, we are growing more comfortable with digital devices in our daily lives.
The 2016 Summer Movie Preview is a snapshot of the films opening through early September. Release dates and other details, as compiled by Kevin Crust, are subject to change. The view of Earth from space and the information it reveals about humanity's effect on the planet are examined in this large-format science documentary. Business suddenly picks up for a London kosher baker when his young Muslim apprentice accidentally drops a stash of pot into the mixer. Written by Yehudah Jez Freedman and Jonathan Benson. The kidnapping of a beloved kitty forces two naive cousins to infiltrate a street gang. Written by Peele & Alex Rubens. In 1913 Cambridge, England, a young Indian math genius joins forces with an eccentric professor. Written and directed by Matthew Brown. Written by Lily Hollander, Anya Kochoff Romano. After an auto accident, a young woman finds herself in a life at odds with the one she remembers. Written by Doc Pedrolie and Victoria Arch. The famous writer's downward spiral is witnessed by a young reporter during the revolution. With Minka Kelly, Giovanni Ribisi, Joely Richardson, Adrian Sparks. A lonely lombax and a tiny robot team with the Galactic Rangers to save their world in this animated adventure.