In text-to-SQL task, seq-to-seq models often lead to sub-optimal performance due to limitations in their architecture. In this paper, we present a simple yet effective approach that adapts transformer-based seq-to-seq model to robust text-to-SQL generation. Instead of inducing constraint to decoder or reformat the task as slot-filling, we propose to train seq-to-seq model with Schema aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These denoising objectives acts as the auxiliary tasks for better modeling the structural data in S2S generation. In addition, we improve and propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of seq-to-seq model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. The results indicate that the capacity of vanilla seq-to-seq architecture for text-to-SQL may have been under-estimated.

This paper introduces a general framework for explicitly constructing universal deep neural models with inputs from a complete, separable, and locally-compact metric space $\mathcal{X}$ and outputs in the Wasserstein-1 $\mathcal{P}_1(\mathcal{Y})$ space over a complete and separable metric space $\mathcal{Y}$. We find that any model built using the proposed framework is dense in the space $C(\mathcal{X},\mathcal{P}_1(\mathcal{Y}))$ of continuous functions from $\mathcal{X}$ to $\mathcal{P}_1(\mathcal{Y})$ in the corresponding uniform convergence on compacts topology, quantitatively. We identify two methods in which the curse of dimensionality can be broken. The first approach constructs subsets of $C(\mathcal{X},\mathcal{P}_1(\mathcal{Y}))$ consisting of functions that can be efficiently approximated. In the second approach, given any fixed $f \in C(\mathcal{X},\mathcal{P}_1(\mathcal{Y}))$, we build non-trivial subsets of $\mathcal{X}$ on which $f$ can be efficiently approximated. The results are applied to three open problems lying at the interface of applied probability and computational learning theory. We find that the proposed models can approximate any regular conditional distribution of a $\mathcal{Y}$-valued random element $Y$ depending on an $\mathcal{X}$-valued random element $X$, with arbitrarily high probability. The proposed models are also shown to be capable of generically expressing the aleatoric uncertainty present in most randomized machine learning models. The proposed framework is used to derive an affirmative answer to the open conjecture of Bishop (1994); namely: mixture density networks are generic regular conditional distributions. Numerical experiments are performed in the contexts of extreme learning machines, randomized DNNs, and heteroscedastic regression.

We propose a novel reinforcement learning based framework PoBRL for solving multi-document summarization. PoBRL jointly optimizes over the following three objectives necessary for a high-quality summary: importance, relevance, and length. Our strategy decouples this multi-objective optimization into different subproblems that can be solved individually by reinforcement learning. Utilizing PoBRL, we then blend each learned policies together to produce a summary that is a concise and complete representation of the original input. Our empirical analysis shows state-of-the-art performance on several multi-document datasets. Human evaluation also shows that our method produces high-quality output.

This paper provides the details of implementing two important policy gradient methods to solve the inverted pendulum problem. These are namely the Deep Deterministic Policy Gradient (DDPG) and the Proximal Policy Optimization (PPO) algorithm. The problem is solved by using an actor-critic model where an actor-network is used to learn the policy function and a critic network is to evaluate the actor-network by learning to estimate the Q function. Apart from briefly explaining the mathematics behind these two algorithms, the details of python implementation are provided which helps in demystifying the underlying complexity of the algorithm. In the process, the readers will be introduced to OpenAI/Gym, Tensorflow 2.x and Keras utilities used for implementing the above concepts.

We present a method enabling a large number of agents to learn how to flock, which is a natural behavior observed in large populations of animals. This problem has drawn a lot of interest but requires many structural assumptions and is tractable only in small dimensions. We phrase this problem as a Mean Field Game (MFG), where each individual chooses its acceleration depending on the population behavior. Combining Deep Reinforcement Learning (RL) and Normalizing Flows (NF), we obtain a tractable solution requiring only very weak assumptions. Our algorithm finds a Nash Equilibrium and the agents adapt their velocity to match the neighboring flock's average one. We use Fictitious Play and alternate: (1) computing an approximate best response with Deep RL, and (2) estimating the next population distribution with NF. We show numerically that our algorithm learn multi-group or high-dimensional flocking with obstacles.

The impressive results of modern neural networks partly come from their non linear behaviour. Unfortunately, this property makes it very difficult to apply formal verification tools, even if we restrict ourselves to networks with a piecewise linear structure. However, such networks yields subregions that are linear and thus simpler to analyse independently. In this paper, we propose a method to simplify the verification problem by operating a partitionning into multiple linear subproblems. To evaluate the feasibility of such an approach, we perform an empirical analysis of neural networks to estimate the number of linear regions, and compare them to the bounds currently known. We also present the impact of a technique aiming at reducing the number of linear regions during training.

Continual Learning (CL) refers to a learning setup where data is non stationary and the model has to learn without forgetting existing knowledge. The study of CL for sequential patterns revolves around trained recurrent networks. In this work, instead, we introduce CL in the context of Echo State Networks (ESNs), where the recurrent component is kept fixed. We provide the first evaluation of catastrophic forgetting in ESNs and we highlight the benefits in using CL strategies which are not applicable to trained recurrent models. Our results confirm the ESN as a promising model for CL and open to its use in streaming scenarios.

Higher-order methods for dependency parsing can partially but not fully addresses the issue that edges in dependency tree should be constructed at the text span/subtree level rather than word level. % This shortcoming can cause an incorrect span covered the corresponding tree rooted at a certain word though the word is correctly linked to its head. In this paper, we propose a new method for dependency parsing to address this issue. The proposed method constructs dependency trees by directly modeling span-span (in other words, subtree-subtree) relations. It consists of two modules: the {\it text span proposal module} which proposes candidate text spans, each of which represents a subtree in the dependency tree denoted by (root, start, end); and the {\it span linking module}, which constructs links between proposed spans. We use the machine reading comprehension (MRC) framework as the backbone to formalize the span linking module in an MRC setup, where one span is used as a query to extract the text span/subtree it should be linked to. The proposed method comes with the following merits: (1) it addresses the fundamental problem that edges in a dependency tree should be constructed between subtrees; (2) the MRC framework allows the method to retrieve missing spans in the span proposal stage, which leads to higher recall for eligible spans. Extensive experiments on the PTB, CTB and Universal Dependencies (UD) benchmarks demonstrate the effectiveness of the proposed method. We are able to achieve new SOTA performances on PTB and UD benchmarks, and competitive performances to previous SOTA models on the CTB dataset. Code is available at https://github.com/ShannonAI/mrc-for-dependency-parsing.

Do you read me, HAL?" said astronaut Dave Bowman, desperately trying to keep his emotions in check. There was a pause and then, in an emotionless monotone, the computer responded. Was HAL, the all-powerful A.I. that controlled the Discovery One spacecraft, really ignoring him? "I'm sorry, Dave," HAL continued. "I'm afraid I can't do that." "What's the problem?" Dave asked. I've forgotten how to open them."

Unlike 2D data, 3D data is complex with more parameters and features. Collecting 3D data and transforming it from one representation to another is a tedious process. Thus 3D deep learning is more time consuming and error-prone than 2D Computer Vision. Though there are nicely-performing models, datasets, metrics, graphics tools, and visualization tools published in recent years, integrating different approaches is quite a non-trivial job for researchers and practitioners. In this scenario, NVIDIA introduced a PyTorch-based library named Kaolin and has recently released its latest optimized version.