Gradient-based hyperparameter optimization is an attractive way to perform meta-learning across a distribution of tasks, or improve the performance of an optimizer on a single task. However, this approach has been unpopular for tasks requiring long horizons (many gradient steps), due to memory scaling and gradient degradation issues. A common workaround is to learn hyperparameters online or split the horizon into smaller chunks. However, this introduces greediness which comes with a large performance drop, since the best local hyperparameters can make for poor global solutions. In this work, we enable non-greediness over long horizons with a two-fold solution. First, we share hyperparameters that are contiguous in time, and show that this drastically mitigates gradient degradation issues. Then, we derive a forward-mode differentiation algorithm for the popular momentum-based SGD optimizer, which allows for a memory cost that is constant with horizon size. When put together, these solutions allow us to learn hyperparameters without any prior knowledge. Compared to the baseline of hand-tuned off-the-shelf hyperparameters, our method compares favorably on simple datasets like SVHN. On CIFAR-10 we match the baseline performance, and demonstrate for the first time that learning rate, momentum and weight decay schedules can be learned with gradients on a dataset of this size. Code is available at https://github.com/polo5/NonGreedyGradientHPO

In the current era, an increasing number of machine learning models is generated for the automation of industrial processes. To that end, machine learning models are trained using historical data of each single asset leading to the development of asset-based models. To elevate machine learning models to a higher level of learning capability, domain adaptation has opened the door for extracting relevant patterns from several assets combined together. In this research we are focusing on translating the specific asset-based historical data (source domain) into data corresponding to one reference asset (target domain), leading to the creation of a multi-assets global dataset required for training domain invariant generic machine learning models. This research is conducted to apply domain adaptation to the ironmaking industry, and particularly for the creation of a domain invariant dataset by gathering data from different blast furnaces. The blast furnace data is characterized by multivariate time series. Domain adaptation for multivariate time series data hasn't been covered extensively in the literature. We propose MTS-CycleGAN, an algorithm for Multivariate Time Series data based on CycleGAN. To the best of our knowledge, this is the first time CycleGAN is applied on multivariate time series data. Our contribution is the integration in the CycleGAN architecture of a Long Short-Term Memory (LSTM)-based AutoEncoder (AE) for the generator and a stacked LSTM-based discriminator, together with dedicated extended features extraction mechanisms. MTS-CycleGAN is validated using two artificial datasets embedding the complex temporal relations between variables reflecting the blast furnace process. MTS-CycleGAN is successfully learning the mapping between both artificial multivariate time series datasets, allowing an efficient translation from a source to a target artificial blast furnace dataset.

We describe an application of machine learning to a real-world computer assisted labeling task. Our experimental results expose significant deviations from the IID assumption commonly used in machine learning. These results suggest that the common random split of all data into training and testing can often lead to poor performance.

We demonstrate two new important properties of the 1-path-norm of shallow neural networks. First, despite its non-smoothness and non-convexity it allows a closed form proximal operator which can be efficiently computed, allowing the use of stochastic proximal-gradient-type methods for regularized empirical risk minimization. Second, when the activation functions is differentiable, it provides an upper bound on the Lipschitz constant of the network. Such bound is tighter than the trivial layer-wise product of Lipschitz constants, motivating its use for training networks robust to adversarial perturbations. In practical experiments we illustrate the advantages of using the proximal mapping and we compare the robustness-accuracy trade-off induced by the 1-path-norm, L1-norm and layer-wise constraints on the Lipschitz constant (Parseval networks).

Sharks, especially great whites, were catapulted into the public eye with the release of the film Jaws in the summer of 1975. The film is the story of a massive great white that terrorizes a seaside community, and the image of the cover alone--the exposed jaws of a massive shark rising upward in murky water--is enough to inject fear into the hearts of would-be swimmers. Other thrillers have perpetuated the theme of sharks as villans. But where did our fear of sharks come from, and how far back does it go? We're going to need a bigger boat: Take a look at the design history of Jaws and its iconic cover https://t.co/dRdRPILF7L

The demand for AI continues to increase according to forecasts by International Data Corporation. Enterprises will adopt AI in 2020 with an estimated 16% surge compared to previous years. Diversity is enabling the growth of AI as companies rely on AI for decision-making with bias incidents reducing according to the IDC report. The customer experience from AI is growing as enterprises analyze interactions, and respond to queries in real-time. Automated AI systems are offering customer support, an area humans have faced challenges because of physical limitations.

With AI often thrown around as a buzzword in business circles, people often forget that machine learning is a means to an end, rather than an end in itself. For most companies, building an AI is not your true goal. Instead, AI implementation can provide you with the tools to meet your goals, be it better customer service through an intuitive chatbot or streamlining video production through synthetic voiceovers. To help shed light on some real-world applications of machine learning, this article introduces five innovative AI software that you should keep on eye on throughout 2020. Scanta is an AI startup with a very interesting history.

What if you could make software testing simple? What if it could be done without all the conversations, questions, defect reports, and metrics? We've been promised artificial intelligence (AI) as the solution to all problems related to testing, especially by those who have never tested--those who believe that what we do as testers is little more than tapping screens to make comparisons. Although I've stated that AI is coming and will change software testing forever (eventually), we're not there yet--not even close. But that doesn't mean we can't use AI to support our testing efforts.

The ability to cooperate through language is a defining feature of humans. As the perceptual, motory and planning capabilities of deep artificial networks increase, researchers are studying whether they also can develop a shared language to interact. From a scientific perspective, understanding the conditions under which language evolves in communities of deep agents and its emergent features can shed light on human language evolution. From an applied perspective, endowing deep networks with the ability to solve problems interactively by communicating with each other and with us should make them more flexible and useful in everyday life.

Data-driven prediction and physics-agnostic machine-learning methods have attracted increased interest in recent years achieving forecast horizons going well beyond those to be expected for chaotic dynamical systems. In a separate strand of research data-assimilation has been successfully used to optimally combine forecast models and their inherent uncertainty with incoming noisy observations. The key idea in our work here is to achieve increased forecast capabilities by judiciously combining machine-learning algorithms and data assimilation. We combine the physics-agnostic data-driven approach of random feature maps as a forecast model within an ensemble Kalman filter data assimilation procedure. The machine-learning model is learned sequentially by incorporating incoming noisy observations. We show that the obtained forecast model has remarkably good forecast skill while being computationally cheap once trained. Going beyond the task of forecasting, we show that our method can be used to generate reliable ensembles for probabilistic forecasting as well as to learn effective model closure in multi-scale systems.