We present a new class of neurons, ARNs, which give a cross entropy on test data that is up to three times lower than the one achieved by carefully optimized LSTM neurons. The explanations for the huge improvements that often are achieved are elaborate skip connections through time, up to four internal memory states per neuron and a number of novel activation functions including small quadratic forms. The new neurons were generated using automatic programming and are formulated as pure functional programs that easily can be transformed. We present experimental results for eight datasets and found excellent improvements for seven of them, but LSTM remained the best for one dataset. The results are so promising that automatic programming to generate new neurons should become part of the standard operating procedure for any machine learning practitioner who works on time series data such as sensor signals.

As a network-based functional approximator, we have proposed a "Lagrangian Density Space-Time Deep Neural Networks" (LDDNN) topology. It is qualified for unsupervised training and learning to predict the dynamics of underlying physical science governed phenomena. The prototypical network respects the fundamental conservation laws of nature through the succinctly described Lagrangian and Hamiltonian density of the system by a given data-set of generalized nonlinear partial differential equations. The objective is to parameterize the Lagrangian density over a neural network and directly learn from it through data instead of hand-crafting an exact time-dependent "Action solution" of Lagrangian density for the physical system. With this novel approach, can understand and open up the information inference aspect of the "Black-box deep machine learning representation" for the physical dynamics of nature by constructing custom-tailored network interconnect topologies, activation, and loss/cost functions based on the underlying physical differential operators. This article will discuss statistical physics interpretation of neural networks in the Lagrangian and Hamiltonian domains.

Mean-field games (MFGs) are a modeling framework for systems with a large number of interacting agents. They have applications in economics, finance, and game theory. Normalizing flows (NFs) are a family of deep generative models that compute data likelihoods by using an invertible mapping, which is typically parameterized by using neural networks. They are useful for density modeling and data generation. While active research has been conducted on both models, few noted the relationship between the two. In this work, we unravel the connections between MFGs and NFs by contextualizing the training of an NF as solving the MFG. This is achieved by reformulating the MFG problem in terms of agent trajectories and parameterizing a discretization of the resulting MFG with flow architectures. With this connection, we explore two research directions. First, we employ expressive NF architectures to accurately solve high-dimensional MFGs, sidestepping the curse of dimensionality in traditional numerical methods. Compared with other deep learning approaches, our trajectory-based formulation encodes the continuity equation in the neural network, resulting in a better approximation of the population dynamics. Second, we regularize the training of NFs with transport costs and show the effectiveness on controlling the model's Lipschitz bound, resulting in better generalization performance. We demonstrate numerical results through comprehensive experiments on a variety of synthetic and real-life datasets.

We consider a general non-linear model where the signal is a finite mixture of an unknown, possibly increasing, number of features issued from a continuous dictionary parameterized by a real nonlinear parameter. The signal is observed with Gaussian (possibly correlated) noise in either a continuous or a discrete setup. We propose an off-the-grid optimization method, that is, a method which does not use any discretization scheme on the parameter space, to estimate both the non-linear parameters of the features and the linear parameters of the mixture. We use recent results on the geometry of off-the-grid methods to give minimal separation on the true underlying non-linear parameters such that interpolating certificate functions can be constructed. Using also tail bounds for suprema of Gaussian processes we bound the prediction error with high probability. Assuming that the certificate functions can be constructed, our prediction error bound is up to log --factors similar to the rates attained by the Lasso predictor in the linear regression model. We also establish convergence rates that quantify with high probability the quality of estimation for both the linear and the non-linear parameters.

Modern search systems use several large ranker models with transformer architectures. These models require large computational resources and are not suitable for usage on devices with limited computational resources. Knowledge distillation is a popular compression technique that can reduce the resource needs of such models, where a large teacher model transfers knowledge to a small student model. To drastically reduce memory requirements and energy consumption, we propose two extensions for a popular sentence-transformer distillation procedure: generation of an optimal size vocabulary and dimensionality reduction of the embedding dimension of teachers prior to distillation. We evaluate these extensions on two different types of ranker models. This results in extremely compressed student models whose analysis on a test dataset shows the significance and utility of our proposed extensions.

The sketch is configured to update the metrics every eight seconds. You can see the Arduino sketch below and in the project's GitHub, NILM⁹ The actual energy disaggregation computations are hosted on a Raspberry Pi 4 which is connected over USB to the Arduino to fetch the aggregate metrics. The computations are comprised of running the tflite appliance inference models, trained and quantized per the steps described above, with pre- and post-processing steps. The inference models output predicted energy for each appliance from 599-sample sized windows of the aggregate apparent power input signal. These predictions are stored in a local CSV file and made available for downstream reporting and analysis.

Artificial intelligence (AI) is often presented in binary terms in both popular culture and political analysis. Either it represents the key to a futuristic utopia defined by the integration of human intelligence and technological prowess, or it is the first step toward a dystopian rise of machines. This same binary thinking is practiced by academics, entrepreneurs, and even activists in relation to the application of AI in combating climate change. The technology industry's singular focus on AI's role in creating a new technological utopia obscures the ways that AI can exacerbate environmental degradation, often in ways that directly harm marginalized populations. In order to utilize AI in fighting climate change in a way that both embraces its technological promise and acknowledges its heavy energy use, the technology companies leading the AI charge need to explore solutions to the environmental impacts of AI.

Today's industry is experiencing an extraordinary moment, in terms of evolution and efficiency of opportunities and potential for transformation enabled by the advent of new technologies that are now gathered under the umbrella of the so-called Industry 4.0. This term, which represents the advent of a new Industrial Revolution, the fourth, encompasses a set of innovation and technological developments, all potentially competing in transforming today's industrial processes from the very root, making them totally different from those of the past. However, if taken as stand-alone, these new techniques or developments may seem like only promising "add-ons" to today's factory, already highly automated/digitized and intelligent in its own way. All enabled by advanced connectivity, simulation tools, Edge and Cloud computing to support computation, IoT, automation and advanced robotics and new, more sustainable and efficient energy supply sources. But what represents the real keystone of the "4.0 Factory", however, is the very high degree of interconnection between all these technologies, both with each other and with all the different sub-processes and systems in the same chain of production, from logistics to production, from design to order management and quality check.

In less than a century, computing has transformed our society and helped spur countless innovations. We now carry in our back pockets computers that we could only have dreamed of a few decades ago. Machine-learning systems can analyze scenes and drive vehicles. And we can craft extraordinarily accurate representations of the real world--models that can be used to design nuclear reactors, simulate myriad greenhouse-gas emission scenarios, and launch a probe on a nine-year trip to study Pluto in an all-too-brief high-speed fly-by. We fundamentally owe these capabilities to our ability to build progressively better computing devices--the transistors and other components at the heart of computer chips.

The rise of vehicle automation has generated significant interest in the potential role of future automated vehicles (AVs). In particular, in highly dense traffic settings, AVs are expected to serve as congestion-dampeners, mitigating the presence of instabilities that arise from various sources. However, in many applications, such maneuvers rely heavily on non-local sensing or coordination by interacting AVs, thereby rendering their adaptation to real-world settings a particularly difficult challenge. To address this challenge, this paper examines the role of imitation learning in bridging the gap between such control strategies and realistic limitations in communication and sensing. Treating one such controller as an "expert", we demonstrate that imitation learning can succeed in deriving policies that, if adopted by 5% of vehicles, may boost the energy-efficiency of networks with varying traffic conditions by 15% using only local observations. Results and code are available online at https://sites.google.com/view/il-traffic/home.