What is optimization Some real-life situations where we need to optimize an objective The mathematical formalism of optimization How discrete optimization (Combinatorics) differs from continuous optimization Different approaches to solve a Combinatorics problem, including-- The simplest, perfect but slow'Brute Force' method. One of the fastest and practicable'Greedy' heuristic.A look-ahead mechanism to refine the greedy approach. Travelling Salesman Problem Other generic problems in discrete optimization, like the Knapsack Problem How metaheuristic approaches compare to heuristic solutions The nature-inspired class of metaheuristic approaches Ant Colony Optimization: its basis, modus operandi, algorithm and flow chart The R library to implement Ant Colony Optimization and other heuristic solutions Examples of Travelling Salesman Problems solved through different approaches ESSENTIAL: A moderate knowledge of Mathematics (High School level) BOOSTER: Familiarity with some programming language (preferably R) BOOSTER: Interest in solving puzzles and games involving logic BOOSTER: Basic knowhow on what Data Science is about Discrete Optimization is something all of us use in our daily activities when say, we order at a restaurant, decide which subject to study, take up a new activity… or look for a change. It comprises of choosing between alternatives that best suit some objective we have in mind. When such things are formalized, i.e. the objective and the ability of each choice to fulfill that objective are quantified, we get a mathematical expression of the problem we would optimize.

This paper is devoted to the prediction of solutions to a stochastic discrete optimization problem. Through an application, we illustrate how we can use a state-of-the-art neural machine translation (NMT) algorithm to predict the solutions by defining appropriate vocabularies, syntaxes and constraints. We attend to applications where the predictions need to be computed in very short computing time -- in the order of milliseconds or less. The results show that with minimal adaptations to the model architecture and hyperparameter tuning, the NMT algorithm can produce accurate solutions within the computing time budget. While these predictions are slightly less accurate than approximate stochastic programming solutions (sample average approximation), they can be computed faster and with less variability.

When samples have internal structure, we often see a mismatch between the objective optimized during training and the model's goal during inference. For example, in sequence-to-sequence modeling we are interested in high-quality translated sentences, but training typically uses maximum likelihood at the word level. Learning to recognize individual faces from group photos, each captioned with the correct but unordered list of people in it, is another example where a mismatch between training and inference objectives occurs. In both cases, the natural training-time loss would involve a combinatorial problem -- dynamic programming-based global sequence alignment and weighted bipartite graph matching, respectively -- but solutions to combinatorial problems are not differentiable with respect to their input parameters, so surrogate, differentiable losses are used instead. Here, we show how to perform gradient descent over combinatorial optimization algorithms that involve continuous parameters, for example edge weights, and can be efficiently expressed as integer, linear, or mixed-integer linear programs. We demonstrate usefulness of gradient descent over combinatorial optimization in sequence-to-sequence modeling using differentiable encoder-decoder architecture with softmax or Gumbel-softmax, and in weakly supervised learning involving a convolutional, residual feed-forward network for image classification.

Daniel Jiwoong Im 1, Yibo Jiang 2, and Nakul Verma 3 1 Janelia Research Campus, HHMI, Virgina 2 Harvard University, Massachusetts 3 Columbia University, New York Abstract Meta learning has emerged as an important framework for learning new tasks from just a few examples. The success of any meta-learning model depends on (i) its fast adaptation to new tasks, as well as (ii) having a shared representation across similar tasks. Here we extend the model-agnostic meta-learning (MAML) framework introduced by Finn et al. (2017) to achieve improved performance by analyzing the temporal dynamics of the optimization procedure via the Runge-Kutta method. This method enables us to gain fine-grained control over the optimization and helps us achieve both the adaptation and representation goals across tasks. By leveraging this refined control, we demonstrate that there are multiple principled ways to update MAML and show that the classic MAML optimization is simply a special case of second order Runge-Kutta method that mainly focuses on fast-adaptation. Experiments on benchmark classification, regression and reinforcement learning tasks show that this refined control helps attain improved results. 1 Introduction Building an intelligent system that can learn quickly on a new task with few examples or few experiences is one of the central goals of machine learning. Achieving this goal requires an agent that learns continuously while having the ability to adapt to new tasks with limited data. Meta-learning (Biggs, 1985) has emerged as a compelling framework that strives to attain this challenging goal. There are two main approaches to meta-learning: learning-to-optimize and learning-to-initialize the meta-model (usually encoded as deep network).

A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data, which makes them susceptible to estimation errors. We thus propose to replace each nominal distribution with an ambiguity set containing all distributions in its vicinity and to evaluate an \emph{optimistic likelihood}, that is, the maximum of the likelihood over all distributions in the ambiguity set. When the proximity of distributions is quantified by the Fisher-Rao distance or the Kullback-Leibler divergence, the emerging optimistic likelihoods can be computed efficiently using either geodesic or standard convex optimization techniques. We showcase the advantages of working with optimistic likelihoods on a classification problem using synthetic as well as empirical data.

Large-scale machine learning training suffers from two prior challenges, specifically for nuclear-norm constrained problems with distributed systems: the synchronization slowdown due to the straggling workers, and high communication costs. In this work, we propose an asynchronous Stochastic Frank Wolfe (SFW-asyn) method, which, for the first time, solves the two problems simultaneously, while successfully maintaining the same convergence rate as the vanilla SFW. We implement our algorithm in python (with MPI) to run on Amazon EC2, and demonstrate that SFW-asyn yields speed-ups almost linear to the number of machines compared to the vanilla SFW.

Deep Neural Networks (DNNs) are widely applied in a wide range of usecases. There is an increased demand for deploying DNNs on devices that do not have abundant resources such as memory and computation units. Recently, network compression through a variety of techniques such as pruning and quantization have been proposed to reduce the resource requirement. A key parameter that all existing compression techniques are sensitive to is the compression ratio (e.g., pruning sparsity, quantization bitwidth) of each layer. Traditional solutions treat the compression ratios of each layer as hyper-parameters, and tune them using human heuristic. Recent researchers start using black-box hyper-parameter optimizations, but they will introduce new hyper-parameters and have efficiency issue. In this paper, we propose a framework to jointly prune and quantize the DNNs automatically according to a target model size without using any hyper-parameters to manually set the compression ratio for each layer. In the experiments, we show that our framework can compress the weights data of ResNet-50 to be 836x smaller without accuracy loss on CIFAR-10, and compress AlexNet to be 205x smaller without accuracy loss on ImageNet classification.

More recent works introduced advanced multi-criteria optimization models [12, 21, 27] that can express more accurately several established cartographic principles, but still with the aim of a full automation of the map labeling process. While progress is made by incorporating more comprehensive cartographic rules for label placement, none of the above approaches includes decisions made by human experts - other than setting preferences, parameters, and priorities in the different scoring functions that control a single optimization run of the respective algorithm. A notable exception is the UserHints framework [7], where human interaction was integrated into solving the label number maximization problem in a fixed-position point labeling setting. In that system, two heuristic methods were implemented as labeling algorithms, and hence the evaluation could not assess the deviation from optimal solutions with respect to the objective function. Moreover, the authors did not consider the stability of the labeling under user interaction. Beyond the label placement problem, interactive optimization [22] and human-guided search [16] are of course techniques that are of general interest and more broadly applicable. 2 Popular GIS software like Mapbox 1, ArcGIS Pro 2, or QGIS 3 also provide labeling algorithms. Mapbox allows customized label modifications with data conditions, but no manual selection or drag-and-drop placement. The ArcGIS Pro documentation 4 states "Label positions are generated automatically.

Regression methods have provided a foundation for modeling structure-property relationships in materials science1,2,3,4,5,6. These methods take as input a data set of known material compositions along with some property (e.g. Each material, in turn, is described in terms of one or more features that represent aspects of structure, chemistry, bonding and/or microstructure in an abstract, high-dimensional space. The success of regression is based on its ability to capture the relative variation in a property as a function of the features, eventually culminating in the prediction of new materials with desired properties. Materials design is an optimization problem with the goal of maximizing (or minimizing) some desired property of a material, denoted by y, by varying certain features, denoted by x.

A highly influential ingredient of many techniques designed to exploit sparsity in numerical optimization is the so-called chordal extension of a graph representation of the optimization problem. The definitive relation between chordal extension and the performance of the optimization algorithm that uses the extension is not a mathematically understood task. For this reason, we follow the current research trend of looking at Combinatorial Optimization tasks by using a Machine Learning lens, and we devise a framework for learning elimination rules yielding high-quality chordal extensions. As a first building block of the learning framework, we propose an on-policy imitation learning scheme that mimics the elimination ordering provided by the (classical) minimum degree rule. The results show that our on-policy imitation learning approach is effective in learning the minimum degree policy and, consequently, produces graphs with desirable fill-in characteristics.