As diverse high-performance computing (HPC) systems are built, many opportunities arise for applications to solve larger problems than ever before. Given the significantly increased complexity of these HPC systems and application tuning, empirical performance tuning, such as autotuning, has emerged as a promising approach in recent years. Despite its effectiveness, autotuning is often a computationally expensive approach. Transfer learning (TL)-based autotuning seeks to address this issue by leveraging the data from prior tuning. Current TL methods for autotuning spend significant time modeling the relationship between parameter configurations and performance, which is ineffective for few-shot (that is, few empirical evaluations) tuning on new tasks. We introduce the first generative TL-based autotuning approach based on the Gaussian copula (GC) to model the high-performing regions of the search space from prior data and then generate high-performing configurations for new tasks. This allows a sampling-based approach that maximizes few-shot performance and provides the first probabilistic estimation of the few-shot budget for effective TL-based autotuning. We compare our generative TL approach with state-of-the-art autotuning techniques on several benchmarks. We find that the GC is capable of achieving 64.37% of peak few-shot performance in its first evaluation. Furthermore, the GC model can determine a few-shot transfer budget that yields up to 33.39$\times$ speedup, a dramatic improvement over the 20.58$\times$ speedup using prior techniques.

Survival analysis studies and predicts the time of death, or other singular unrepeated events, based on historical data, while the true time of death for some instances is unknown. Survival trees enable the discovery of complex nonlinear relations in a compact human comprehensible model, by recursively splitting the population and predicting a distinct survival distribution in each leaf node. We use dynamic programming to provide the first survival tree method with optimality guarantees, enabling the assessment of the optimality gap of heuristics. We improve the scalability of our method through a special algorithm for computing trees up to depth two. The experiments show that our method's run time even outperforms some heuristics for realistic cases while obtaining similar out-of-sample performance with the state-of-the-art.

Graph matching is one of the most significant graph analytic tasks in practice, which aims to find the node correspondence across different graphs. Most existing approaches rely on adjacency matrices or node embeddings when matching graphs, whose performances are often sub-optimal because of not fully leveraging the multi-modal information hidden in graphs, such as node attributes, subgraph structures, etc. In this study, we propose a novel and effective graph matching method based on a differentiable hierarchical optimal transport (HOT) framework, called DHOT-GM. Essentially, our method represents each graph as a set of relational matrices corresponding to the information of different modalities. Given two graphs, we enumerate all relational matrix pairs and obtain their matching results, and accordingly, infer the node correspondence by the weighted averaging of the matching results. This method can be implemented as computing the HOT distance between the two graphs -- each matching result is an optimal transport plan associated with the Gromov-Wasserstein (GW) distance between two relational matrices, and the weights of all matching results are the elements of an upper-level optimal transport plan defined on the matrix sets. We propose a bi-level optimization algorithm to compute the HOT distance in a differentiable way, making the significance of the relational matrices adjustable. Experiments on various graph matching tasks demonstrate the superiority and robustness of our method compared to state-of-the-art approaches.

Deep Learning (DL) has enabled a paradigm shift in wireless communication system with data driven end-to-end (E2E) learning and optimization of the Physical Layer (PHY). By leveraging the representation learning of DL, E2E systems exhibit enhanced adaptability and performance in complex wireless environments, fulfilling the demands of 5G and beyond network systems and applications. The evolution of data-driven techniques in the PHY has enabled advanced semantic applications across various modalities including text, image, audio, video, and multi-modal transmissions. These applications transcend from traditional bit-level communication to semantic-level intelligent communication systems, which are capable of understanding and adapting to the context and intent of the data transmission. Although PHY as a DL architecture for data-driven E2E communication is a key factor in enabling semantic communication systems (SemCom), and various studies in recent years have surveyed them separately, their combination has not been thoroughly reviewed. Additionally, these are emerging fields that are still in their infancy, with several techniques having been developed and evolved in recent years. Therefore, this article provides a holistic review of data-driven PHY for E2E communication system, and their enabling semantic applications across different modalities. Furthermore, it identifies critical challenges and prospective research directions, providing a pivotal reference for future development of DL in PHY and SemCom.

We propose a methodology, based on machine learning and optimization, for selecting a solver configuration for a given instance. First, we employ a set of solved instances and configurations in order to learn a performance function of the solver. Secondly, we formulate a mixed-integer nonlinear program where the objective/constraints explicitly encode the learnt information, and which we solve, upon the arrival of an unknown instance, to find the best solver configuration for that instance, based on the performance function. The main novelty of our approach lies in the fact that the configuration set search problem is formulated as a mathematical program, which allows us to a) enforce hard dependence and compatibility constraints on the configurations, and b) solve it efficiently with off-the-shelf optimization tools.

During the architectural analysis, abstract artifacts need to be precisely identified and specified in order to efficiently guide the development, evolution and deployment of the overall system. Considering such an early stage, architectural decisions become even more challenging due to the lack of knowledge about the system but, at the same time, they are crucial to fulfill the many quality criteria imposed [12]. Artificial intelligence techniques and, more specifically, metaheuristics, can support software engineers in their decision processes by providing them with effective methods to explore a great deal of software designs, each one determined by a different trade-off among the required quality aspects. Such a scenario can be viewed as one of the goals of the search-based software engineering (SBSE) field[14], in which optimization techniques are applied to the resolution of software engineering (SE) tasks conveniently reformulated as search problems. However, solving human-centered activities in a fully automated way seems to be unrealistic, especially for those related to the analysis phase. Certainly, trying to capture the richness of human knowledge only by means of software metrics still represents an unresolved matter to the SE community [32]. Hence, most of the evaluation methods proposed at the architectural level strongly rely on the expert's judgment [10], making extremely difficult to precisely formulate a quantitative fitness function. Given the relevance of the software architect for the design process, searchbased approaches should benefit from his/her knowledge and expertise in order to address the optimization problem in the same way s/he would do it. Interactive optimization [21] constitutes a compelling paradigm here.

Fine-tuning is the primary methodology for tailoring pre-trained large language models to specific tasks. As the model's scale and the diversity of tasks expand, parameter-efficient fine-tuning methods are of paramount importance. One of the most widely used family of methods is low-rank adaptation (LoRA) and its variants. LoRA encodes weight update as the product of two low-rank matrices. Despite its advantages, LoRA falls short of full-parameter fine-tuning in terms of generalization error for certain tasks. We introduce Chain of LoRA (COLA), an iterative optimization framework inspired by the Frank-Wolfe algorithm, to bridge the gap between LoRA and full parameter fine-tuning, without incurring additional computational costs or memory overheads. COLA employs a residual learning procedure where it merges learned LoRA modules into the pre-trained language model parameters and re-initilize optimization for new born LoRA modules. We provide theoretical convergence guarantees as well as empirical results to validate the effectiveness of our algorithm. Across various models (OPT and llama-2) and seven benchmarking tasks, we demonstrate that COLA can consistently outperform LoRA without additional computational or memory costs.

The motivation for these problems is to enable efficient computation of heuristic estimators to formally evaluate and reason about different quantities of deep neural networks in the interest of AI alignment [3]. Our solutions involve reframing the matrix completion problems as a semidefinite program (SDP) and using recent advances in spectral bundle methods for fast, efficient, and scalable SDP solving. Proving that this task is at least as hard as dense matrix multiplication or positive semidefinite testing would count as a resolution. Question 2 (fast "approximate squaring"): Given A R The core idea is to formulate both questions as semidefinite programs (SDP) and use a spectral bundle method [1, 5, 9-11] to implicitly solve the SDP or obtain a certificate of infeasibility. In the case where the SDP is infeasible, our method computes an upper bound quantifying the degree to which the SDP is infeasible.

Many real world optimization problems are formulated as mixed-variable optimization problems (MVOPs) which involve both continuous and discrete variables. MVOPs including dimensional variables are characterized by a variable-size search space. Depending on the values of dimensional variables, the number and type of the variables of the problem can vary dynamically. MVOPs and variable-size MVOPs (VMVOPs) are difficult to solve and raise a number of scientific challenges in the design of metaheuristics. Standard metaheuristics have been first designed to address continuous or discrete optimization problems, and are not able to tackle (V)MVOPs in an efficient way. The development of metaheuristics for solving such problems has attracted the attention of many researchers and is increasingly popular. However, to our knowledge there is no well established taxonomy and comprehensive survey for handling this important family of optimization problems. This paper presents a unified taxonomy for metaheuristic solutions for solving (V)MVOPs in an attempt to provide a common terminology and classification mechanisms. It provides a general mathematical formulation and concepts of (V)MVOPs, and identifies the various solving methodologies than can be applied in metaheuristics. The advantages, the weaknesses and the limitations of the presented methodologies are discussed. The proposed taxonomy also allows to identify some open research issues which needs further in-depth investigations.

Optimizing service schedules is pivotal to the reliable, efficient, and inclusive on-demand mobility. This pressing challenge is further exacerbated by the increasing needs of an aging population, the over-subscription of existing services, and the lack of effective solution methods. This study addresses the intricacies of service scheduling, by jointly optimizing rider trip planning and crew scheduling for a complex dynamic mobility service. The resulting optimization problems are extremely challenging computationally for state-of-the-art methods. To address this fundamental gap, this paper introduces the Joint Rider Trip Planning and Crew Shift Scheduling Problem (JRTPCSSP) and a novel solution method, called AGGNNI-CG (Attention and Gated GNN- Informed Column Generation), that hybridizes column generation and machine learning to obtain near-optimal solutions to the JRTPCSSP with the real-time constraints of the application. The key idea of the machine-learning component is to dramatically reduce the number of paths to explore in the pricing component, accelerating the most time-consuming component of the column generation. The machine learning component is a graph neural network with an attention mechanism and a gated architecture, that is particularly suited to cater for the different input sizes coming from daily operations. AGGNNI-CG has been applied to a challenging, real-world dataset from the Paratransit system of Chatham County in Georgia. It produces dramatic improvements compared to the baseline column generation approach, which typically cannot produce feasible solutions in reasonable time on both medium-sized and large-scale complex instances. AGGNNI-CG also produces significant improvements in service compared to the existing system.