A novel neural network (NN) approach is proposed for constrained optimization. The proposed method uses a specially designed NN architecture and training/optimization procedure called Neural Optimization Machine (NOM). The objective functions for the NOM are approximated with NN models. The optimization process is conducted by the neural network's built-in backpropagation algorithm. The NOM solves optimization problems by extending the architecture of the NN objective function model. This is achieved by appropriately designing the NOM's structure, activation function, and loss function. The NN objective function can have arbitrary architectures and activation functions. The application of the NOM is not limited to specific optimization problems, e.g., linear and quadratic programming. It is shown that the increase of dimension of design variables does not increase the computational cost significantly. Then, the NOM is extended for multiobjective optimization. Finally, the NOM is tested using numerical optimization problems and applied for the optimal design of processing parameters in additive manufacturing.

By enabling multiple agents to cooperatively solve a global optimization problem in the absence of a central coordinator, decentralized stochastic optimization is gaining increasing attention in areas as diverse as machine learning, control, and sensor networks. Since the associated data usually contain sensitive information, such as user locations and personal identities, privacy protection has emerged as a crucial need in the implementation of decentralized stochastic optimization. In this paper, we propose a decentralized stochastic optimization algorithm that is able to guarantee provable convergence accuracy even in the presence of aggressive quantization errors that are proportional to the amplitude of quantization inputs. The result applies to both convex and non-convex objective functions, and enables us to exploit aggressive quantization schemes to obfuscate shared information, and hence enables privacy protection without losing provable optimization accuracy. In fact, by using a {stochastic} ternary quantization scheme, which quantizes any value to three numerical levels, we achieve quantization-based rigorous differential privacy in decentralized stochastic optimization, which has not been reported before. In combination with the presented quantization scheme, the proposed algorithm ensures, for the first time, rigorous differential privacy in decentralized stochastic optimization without losing provable convergence accuracy. Simulation results for a distributed estimation problem as well as numerical experiments for decentralized learning on a benchmark machine learning dataset confirm the effectiveness of the proposed approach.

Bayesian global optimization (BGO) is an efficient surrogate-assisted technique for problems involving expensive evaluations. A parallel technique can be used to parallelly evaluate the true-expensive objective functions in one iteration to boost the execution time. An effective and straightforward approach is to design an acquisition function that can evaluate the performance of a bath of multiple solutions, instead of a single point/solution, in one iteration. This paper proposes five alternatives of \emph{Probability of Improvement} (PoI) with multiple points in a batch (q-PoI) for multi-objective Bayesian global optimization (MOBGO), taking the covariance among multiple points into account. Both exact computational formulas and the Monte Carlo approximation algorithms for all proposed q-PoIs are provided. Based on the distribution of the multiple points relevant to the Pareto-front, the position-dependent behavior of the five q-PoIs is investigated. Moreover, the five q-PoIs are compared with the other nine state-of-the-art and recently proposed batch MOBGO algorithms on twenty bio-objective benchmarks. The empirical experiments on different variety of benchmarks are conducted to demonstrate the effectiveness of two greedy q-PoIs ($\kpoi_{\mbox{best}}$ and $\kpoi_{\mbox{all}}$) on low-dimensional problems and the effectiveness of two explorative q-PoIs ($\kpoi_{\mbox{one}}$ and $\kpoi_{\mbox{worst}}$) on high-dimensional problems with difficult-to-approximate Pareto front boundaries.

Across infrastructure domains, physical damage caused by storms and other weather events often requires costly and time-sensitive repairs to restore services as quickly as possible. While recent studies have used agent-based models to estimate the cost of repairs, the implemented strategies for assignment of repair crews to different locations are generally human-driven or based on simple rules. In order to find performant strategies, we continue with an agent-based model, but approach this problem as a combinational optimization problem known as the Minimum Weighted Latency Problem for multiple repair crews. We apply a partitioning algorithm that balances the assignment of targets amongst all the crews using two different heuristics that optimize either the importance of repair locations or the travel time between them. We benchmark our algorithm on both randomly generated graphs as well as data derived from a real-world urban environment, and show that our algorithm delivers significantly better assignments than existing methods.

Federated learning (FL) is a trending training paradigm to utilize decentralized training data. FL allows clients to update model parameters locally for several epochs, then share them to a global model for aggregation. This training paradigm with multi-local step updating before aggregation exposes unique vulnerabilities to adversarial attacks. Adversarial training is a popular and effective method to improve the robustness of networks against adversaries. In this work, we formulate a general form of federated adversarial learning (FAL) that is adapted from adversarial learning in the centralized setting. On the client side of FL training, FAL has an inner loop to generate adversarial samples for adversarial training and an outer loop to update local model parameters. On the server side, FAL aggregates local model updates and broadcast the aggregated model. We design a global robust training loss and formulate FAL training as a min-max optimization problem. Unlike the convergence analysis in classical centralized training that relies on the gradient direction, it is significantly harder to analyze the convergence in FAL for three reasons: 1) the complexity of min-max optimization, 2) model not updating in the gradient direction due to the multi-local updates on the client-side before aggregation and 3) inter-client heterogeneity. We address these challenges by using appropriate gradient approximation and coupling techniques and present the convergence analysis in the over-parameterized regime. Our main result theoretically shows that the minimum loss under our algorithm can converge to $\epsilon$ small with chosen learning rate and communication rounds. It is noteworthy that our analysis is feasible for non-IID clients.

Contact-implicit trajectory optimization offers an appealing method of automatically generating complex and contact-rich behaviors for robot manipulation and locomotion. The scalability of such techniques has been limited, however, by the challenge of ensuring both numerical reliability and physical realism. In this paper, we present preliminary results suggesting that the Iterative Linear Quadratic Regulator (iLQR) algorithm together with the recently proposed pressure-field-based hydroelastic contact model enables reliable and physically realistic trajectory optimization through contact. We use this approach to synthesize contact-rich behaviors like quadruped locomotion and whole-arm manipulation. Furthermore, open-loop playback on a Kinova Gen3 robot arm demonstrates the physical accuracy of the whole-arm manipulation trajectories. Code is available at https://bit.ly/ilqr_hc and videos can be found at https://youtu.be/IqxJKbM8_ms.

Artificial Bee Colony (ABC) algorithm is a Swarm Intelligence optimization algorithm inspired by the functioning of honey bees trying to find the best nectar resources surrounding their bee hive. Derviş Kara-Bogaz first proposed this algorithm in 2005. This algorithm has been used in many forms of optimization of complex non-linear functions. As you will see soon, this algorithm is dependent on the randomness of the situation, it is a great domain for applying better strategies to find the optimal point even faster. You will also notice that if the algorithm has a hint that the point is somehow a local minimum, it has a strategy to even discard it.

We propose a new spectrum allocation strategy, aided by unsupervised learning, for multiuser terahertz communication systems. In this strategy, adaptive sub-band bandwidth is considered such that the spectrum of interest can be divided into sub-bands with unequal bandwidths. This strategy reduces the variation in molecular absorption loss among the users, leading to the improved data rate performance. We first formulate an optimization problem to determine the optimal sub-band bandwidth and transmit power, and then propose the unsupervised learning-based approach to obtaining the near-optimal solution to this problem. In the proposed approach, we first train a deep neural network (DNN) while utilizing a loss function that is inspired by the Lagrangian of the formulated problem. Then using the trained DNN, we approximate the near-optimal solutions. Numerical results demonstrate that comparing to existing approaches, our proposed unsupervised learning-based approach achieves a higher data rate, especially when the molecular absorption coefficient within the spectrum of interest varies in a highly non-linear manner.

In this letter we propose a data-driven approach to optimizing the algebraic connectivity of a team of robots. While a considerable amount of research has been devoted to this problem, we lack a method that scales in a manner suitable for online applications for more than a handful of agents. To that end, we propose a supervised learning approach with a convolutional neural network (CNN) that learns to place communication agents from an expert that uses an optimization-based strategy. We demonstrate the performance of our CNN on canonical line and ring topologies, 105k randomly generated test cases, and larger teams not seen during training. We also show how our system can be applied to dynamic robot teams through a Unity-based simulation. After training, our system produces connected configurations over an order of magnitude faster than the optimization-based scheme for teams of 10-20 agents.

For general quadratically-constrained quadratic programming (QCQP), we propose a parabolic relaxation described with convex quadratic constraints. An interesting property of the parabolic relaxation is that the original non-convex feasible set is contained on the boundary of the parabolic relaxation. Under certain assumptions, this property enables one to recover near-optimal feasible points via objective penalization. Moreover, through an appropriate change of coordinates that requires a one-time computation of an optimal basis, the easier-to-solve parabolic relaxation can be made as strong as a semidefinite programming (SDP) relaxation, which can be effective in accelerating algorithms that require solving a sequence of convex surrogates. The majority of theoretical and computational results are given in the next part of this work [57].