I've done several experiments with a text-generating neural network called GPT-2. Trained at great expense by OpenAI (to the tune of tens of thousands of dollars worth of computing power), GPT-2 learned to imitate all kinds of text from the internet. I've interacted with the basic model, discovering its abilities to generate fan fiction, British snacks, or tea. I've also used a tool called gpt-2-simple that Max Woolf developed to make it easy to finetune GPT-2 on more specialized datasets - I've tried it on datasets like recipes or crochet. One of my favorite applications of GPT-2 and other text-generating neural nets is Dungeons and Dragons spells, creatures, character names, and character bios.

Depending on whether you're a glass-half-full or half-empty type of person, the words "artificial intelligence", "machine learning" or "deep learning" can either conjure up a future in which humans are completely subject to their robot overlords, or an idyllic vision of sipping cocktails on sun loungers while algorithms do all the heavy lifting. Reality is, as usual, a bit more complex than science fiction would have us believe. No, you're not likely to be enslaved by your coffee machine anytime soon. Nor is it quite time to hand in your notice and head for the nearest sandy beach. Artificial intelligence is a concept that feels overwhelming to many.

As part of enterprise artificial intelligence (AI) initiatives, data engineering teams are using a wide range of data analytics techniques, ranging from streaming analytics to machine learning to deep learning. This diversity in techniques has led to a corresponding diversity in software platforms and tools. Most data engineering teams are using data ingestion frameworks, such as Kafka; a combination of machine learning tools, such as Hadoop, Splunk, SAS Analytics, Spark, Python and R; and open-source deep learning packages, such as TensorFlow, Caffe and PyTorch. In traditional data analytics pipelines, data flows into enterprise environments from various internal and external sources and gets pre-processed and cleansed. Enterprises commonly use a "staging area" to store intermediate representations of pre-processed data.

As part of enterprise artificial intelligence (AI) initiatives, data engineering teams are using a wide range of data analytics techniques, ranging from streaming analytics to machine learning to deep learning. This diversity in techniques has led to a corresponding diversity in software platforms and tools. Most data engineering teams are using data ingestion frameworks, such as Kafka; a combination of machine learning tools, such as Hadoop, Splunk, SAS Analytics, Spark, Python and R; and open-source deep learning packages, such as TensorFlow, Caffe and PyTorch. In traditional data analytics pipelines, data flows into enterprise environments from various internal and external sources and gets pre-processed and cleansed. Enterprises commonly use a "staging area" to store intermediate representations of pre-processed data.

This paper investigates the convergence of learning dynamics in Stackelberg games. In the class of games we consider, there is a hierarchical game being played between a leader and a follower with continuous action spaces. We establish a number of connections between the Nash and Stackelberg equilibrium concepts and characterize conditions under which attracting critical points of simultaneous gradient descent are Stackelberg equilibria in zero-sum games. Moreover, we show that the only stable critical points of the Stackelberg gradient dynamics are Stackelberg equilibria in zero-sum games. Using this insight, we develop a gradient-based update for the leader while the follower employs a best response strategy for which each stable critical point is guaranteed to be a Stackelberg equilibrium in zero-sum games. As a result, the learning rule provably converges to a Stackelberg equilibria given an initialization in the region of attraction of a stable critical point. We then consider a follower employing a gradient-play update rule instead of a best response strategy and propose a two-timescale algorithm with similar asymptotic convergence guarantees. For this algorithm, we also provide finite-time high probability bounds for local convergence to a neighborhood of a stable Stackelberg equilibrium in general-sum games. Finally, we present extensive numerical results that validate our theory, provide insights into the optimization landscape of generative adversarial networks, and demonstrate that the learning dynamics we propose can effectively train generative adversarial networks.

The dominant automatic lexical stress detection method is to split the utterance into syllable segments using phoneme sequence and their time-aligned boundaries. Then we extract features from syllable to use classification method to classify the lexical stress. However, we can't get very accurate time boundaries of each phoneme and we have to design some features in the syllable segments to classify the lexical stress. Therefore, we propose a end-to-end approach using sequence to sequence model of transformer to estimate lexical stress. For this, we train transformer model using feature sequence of audio and their phoneme sequence with lexical stress marks. During the recognition process, the recognized phoneme sequence is restricted according to the original standard phoneme sequence without lexical stress marks, but the lexical stress mark of each phoneme is not limited. We train the model in different subset of Librispeech and do lexical stress recognition in TIMIT and L2-ARCTIC dataset. For all subsets, the end-to-end model will perform better than the syllable segments classification method. Our method can achieve a 6.36% phoneme error rate on the TIMIT dataset, which exceeds the 7.2% error rate in other studies.

This article presents a novel method for predicting suicidal ideation from Electronic Health Records (EHR) and Ecological Momentary Assessment (EMA) data using deep sequential models. Both EHR longitudinal data and EMA question forms are defined by asynchronous, variable length, randomly-sampled data sequences. In our method, we model each of them with a Recurrent Neural Network (RNN), and both sequences are aligned by concatenating the hidden state of each of them using temporal marks. Furthermore, we incorporate attention schemes to improve performance in long sequences and time-independent pre-trained schemes to cope with very short sequences. Using a database of 1023 patients, our experimental results show that the addition of EMA records boosts the system recall to predict the suicidal ideation diagnosis from 48.13% obtained exclusively from EHR-based state-of-the-art methods to 67.78%. Additionally, our method provides interpretability through the t-SNE representation of the latent space. Further, the most relevant input features are identified and interpreted medically.

Ultrasonic guided waves are commonly used to localize structural damage in infrastructures such as buildings, airplanes, bridges. Damage localization can be viewed as an inverse problem. Physical model based techniques are popular for guided wave based damage localization. The performance of these techniques depend on the degree of faithfulness with which the physical model describes wave propagation. External factors such as environmental variations and random noise are a source of uncertainty in wave propagation. The physical modeling of uncertainty in an inverse problem is still a challenging problem. In this work, we propose a deep learning based model for robust damage localization in presence of uncertainty. Wave data with uncertainty is simulated to reflect variations due to external factors and Gaussian noise is added to reflect random noise in the environment. After evaluating the localization error on test data with uncertainty, we observe that the deep learning model trained with uncertainty can learn robust representations. The approach shows potential for dealing with uncertainty in physical science problems using deep learning models.

Our interest focuses on developing statistical methods for analysis of brain structural connectomes. Nodes in the brain connectome graph correspond to different regions of interest (ROIs) while edges correspond to white matter fiber connections between these ROIs. Due to the high-dimensionality and non-Euclidean nature of the data, it becomes challenging to conduct analyses of the population distribution of brain connectomes and relate connectomes to other factors, such as cognition. Current approaches focus on summarizing the graph using either pre-specified topological features or principal components analysis (PCA). In this article, we instead develop a nonlinear latent factor model for summarizing the brain graph in both unsupervised and supervised settings. The proposed approach builds on methods for hierarchical modeling of replicated graph data, as well as variational auto-encoders that use neural networks for dimensionality reduction. We refer to our method as Graph AuTo-Encoding (GATE). We compare GATE with tensor PCA and other competitors through simulations and applications to data from the Human Connectome Project (HCP).

Deep Learning Models for Global Coordinate Transformations that Linearize PDEs Craig Gin 1, Bethany Lusch 2, Steven L. Brunton 1,3, and J. Nathan Kutz 1 1 Department of Applied Mathematics, University of Washington, Seattle, WA, 98195, USA 2 Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA 3 Department of Mechanical Engineering, University of Washington, Seattle, WA, 98195, USA (Received 11 November 2019) We develop a deep autoencoder architecture that can be used to find a coordinate transformation which turns a nonlinear PDE into a linear PDE. Our architecture is motivated by the linearizing transformations provided by the Cole-Hopf transform for Burgers equation and the inverse scattering transform for completely integrable PDEs. By leveraging a residual network architecture, a near-identity transformation can be exploited to encode intrinsic coordinates in which the dynamics are linear. The resulting dynamics are given by a Koopman operator matrix K. The decoder allows us to transform back to the original coordinates as well. Multiple time step prediction can be performed by repeated multiplication by the matrix K in the intrinsic coordinates. We demonstrate our method on a number of examples, including the heat equation and Burgers equation, as well as the substantially more challenging Kuramoto-Sivashinsky equation, showing that our method provides a robust architecture for discovering interpretable, linearizing transforms for nonlinear PDEs. Key Words: Koopman theory, deep neural nets, residual networks, linearizing transforms, Cole-Hopf transform 2010 Mathematics Subject Classification: 35A22, 35A35, 37M99, 65P99, 68T99 1 Introduction Partial differential equations (PDEs) provide a theoretical framework for modeling spatiotemporal systems across the biological, physical and engineering sciences. Analytic solution techniques are readily available for PDEs that are linear and have constant coefficients [12]. These PDEs include canonical models such as the heat equation, wave equation and Laplace's equation which are amenable to standard separation of variable techniques and linear superposition. In contrast, there is no general mathematical architecture for solving nonlinear PDEs as methods like separation of variables fail to hold, thus recourse to computational solutions is necessary. There are a few, but notable, exceptions: (i) the Cole-Hopf transformation [14, 6] for solving diffusively regularized Burgers equation, and (ii) the Inverse Scattering Transform (IST) [1] for solving a class of completely integrable PDEs such as Korteweg deVries (KdV), nonlinear Schr odinger arXiv:1911.02710v1 A deep autoencoder is used to find coordinate transformations to linearize PDEs. The encoder finds a set of intrinsic coordinates for which the dynamics are linear.