In this article, we propose a new hypothesis testing method for directed acyclic graph (DAG). While there is a rich class of DAG estimation methods, there is a relative paucity of DAG inference solutions. Moreover, the existing methods often impose some specific model structures such as linear models or additive models, and assume independent data observations. Our proposed test instead allows the associations among the random variables to be nonlinear and the data to be time-dependent. We build the test based on some highly flexible neural networks learners. We establish the asymptotic guarantees of the test, while allowing either the number of subjects or the number of time points for each subject to diverge to infinity. We demonstrate the efficacy of the test through simulations and a brain connectivity network analysis.

Normalizing flows are invertible neural networks with tractable change-of-volume terms, which allows optimization of their parameters to be efficiently performed via maximum likelihood. However, data of interest is typically assumed to live in some (often unknown) low-dimensional manifold embedded in high-dimensional ambient space. The result is a modelling mismatch since -- by construction -- the invertibility requirement implies high-dimensional support of the learned distribution. Injective flows, mapping from low- to high-dimensional space, aim to fix this discrepancy by learning distributions on manifolds, but the resulting volume-change term becomes more challenging to evaluate. Current approaches either avoid computing this term entirely using various heuristics, or assume the manifold is known beforehand and therefore are not widely applicable. Instead, we propose two methods to tractably calculate the gradient of this term with respect to the parameters of the model, relying on careful use of automatic differentiation and techniques from numerical linear algebra. Both approaches perform end-to-end nonlinear manifold learning and density estimation for data projected onto this manifold. We study the trade-offs between our proposed methods, empirically verify that we outperform approaches ignoring the volume-change term by more accurately learning manifolds and the corresponding distributions on them, and show promising results on out-of-distribution detection.

Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

Predicting molecular conformations (or 3D structures) from molecular graphs is a fundamental problem in many applications. Most existing approaches are usually divided into two steps by first predicting the distances between atoms and then generating a 3D structure through optimizing a distance geometry problem. However, the distances predicted with such two-stage approaches may not be able to consistently preserve the geometry of local atomic neighborhoods, making the generated structures unsatisfying. In this paper, we propose an end-to-end solution for molecular conformation prediction called ConfVAE based on the conditional variational autoencoder framework. Specifically, the molecular graph is first encoded in a latent space, and then the 3D structures are generated by solving a principled bilevel optimization program. Extensive experiments on several benchmark data sets prove the effectiveness of our proposed approach over existing state-of-the-art approaches. Code is available at https://github.com/MinkaiXu/ConfVAE-ICML21

This post covers model inference optimization or compression in breadth and hopefully depth as of March 2021. This includes engineering topics like model quantization and binarization, more research-oriented topics like knowledge distillation, as well as well-known-hacks. Each year, larger and larger models are able to find methods for extracting signal from the noise in machine learning. In particular, language models get larger every day. These models are computationally expensive (in both runtime and memory), which can be both costly when served out to customers or too slow or large to function in edge environments like a phone. Researchers and practitioners have come up with many methods for optimizing neural networks to run faster or with less memory usage.

If you look beyond image processing--it's one of the most common use cases for AI. And just like image processing, video processing uses established techniques like computer vision, object recognition, machine learning, and deep learning to enhance this process. Whether you use computer vision and NLP in video editing and generation, object recognition in video content auto-tagging tasks, machine learning to streamline AI video analysis, or deep learning to expedite real-time background removal, the use cases continue to grow by the day. Keep reading to learn what approach you can take when it comes to using AI in video processing. Let's start with the basics. Real-time video processing is an essential technology in surveillance systems using object and facial recognition.

The past month in ML research literature has brought surpising results such as the revival of MLPs as a competitive architecture for Computer Vision or the questioning of Batch Normalization as an all-good innocuous layer. Transformers are also (of course) on the plate: for self supervised learning on vision, as well as for sentence representation techniques and character-level language modelling. This is a monthly selection of recent ML research literature, backed by Zeta Alpha, where we're always keeping a close eye at the latest papers. Why Very simple MLP-based architectures suddenly work way better than they should, this has interesting implications and advances our knowledge about what makes Deep Learning work. Key insights You can probably solve ML by just scaling up. Well okay this is an oversimplification and exaggeration, but Rich Sutton's Bitter Lesson seems to be aging better than fine wine so far.

Xception: Deep Learning with Depthwise Separable Convolutions  (CVPR 2017) We present an interpretation of Inception modules in convolutional neural networks as being an intermediate step in-between regular convolution and the depthwise separable convolution operation (a depthwise convolution followed by a pointwise convolution). In this light, a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers. This observation leads us to propose a novel deep convolutional neural network architecture inspired by Inception, where Inception modules have been replaced with depthwise separable convolutions. We show that this architecture, dubbed Xception, slightly outperforms Inception V3 on the ImageNet dataset (which Inception V3 was designed for), and significantly outperforms Inception V3 on a larger image classification dataset comprising 350 million images and 17,000 classes. Since the Xception architecture has the same number of parameters as Inception V3, the performance gains are not due to increased capacity but rather to a more efficient use of model parameters. Paper:   https://arxiv.org/pdf/1610.02357.pdf

Today's artificial intelligence remains a long way from the supple, dynamic intelligence of AI ... [ ] characters from popular fiction, like The Jetsons. Modern artificial intelligence is capable of wonders. It can produce breathtaking original content: poetry, prose, images, music, human faces. Last year it produced a solution to the "protein folding problem," a grand challenge in biology that has stumped researchers for half a century. Yet today's AI still has fundamental limitations. Relative to what we would expect from a truly intelligent agent--relative to that original inspiration and benchmark for artificial intelligence, human cognition--AI has a long way to go. Critics like to point to these shortcomings as evidence that the pursuit of artificial intelligence is misguided or has failed.

Naver, the Seongnam, South Korean-based company that operates the eponymous search engine Naver, this week announced that it trained one of the largest AI language models of its kind, called HyperCLOVA. Naver claims that the system learned 6,500 times more Korean data than OpenAI's GPT-3 and contains 204 billion parameters, the parts of the machine learning model learned from historical training data. For the better part of a year, OpenAI's GPT-3 has remained among the largest AI language models ever created. Via an API, people have used it to automatically write emails and articles, summarize text, compose poetry and recipes, create website layouts, and generate code for deep learning in Python. But GPT-3 has key limitations, chief among them that it's only available in English.