Graph neural networks (GNNs) have achieved superior performance in various applications, but training dedicated GNNs can be costly for large-scale graphs. Some recent work started to study the pre-training of GNNs. However, none of them provide theoretical insights into the design of their frameworks, or clear requirements and guarantees towards their transferability. In this work, we establish a theoretically grounded and practically useful framework for the transfer learning of GNNs. Firstly, we propose a novel view towards the essential graph information and advocate the capturing of it as the goal of transferable GNN training, which motivates the design of EGI (Ego-Graph Information maximization) to analytically achieve this goal.

Human intelligence is characterized not only by the capacity to learn complex skills, but the ability to rapidly adapt and acquire new skills within an ever-changing environment. In this work we study how the learning of modular solutions can allow for effective generalization to both unseen and potentially differently distributed data. Our main postulate is that the combination of task segmentation, modular learning and memory-based ensembling can give rise to generalization on an exponentially growing number of unseen tasks. We provide a concrete instantiation of this idea using a combination of: (1) the Forget-Me-Not Process, for task segmentation and memory based ensembling; and (2) Gated Linear Networks, which in contrast to contemporary deep learning techniques use a modular and local learning mechanism. We demonstrate that this system exhibits a number of desirable continual learning properties: robustness to catastrophic forgetting, no negative transfer and increasing levels of positive transfer as more tasks are seen.

One desired capability for machines is the ability to transfer their understanding of one domain to another domain where data is (usually) scarce. Despite ample adaptation of transfer learning in many deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analysis to address these fundamental questions. Through a series of analysis on transferring to block-shuffled images, we separate the effect of feature reuse from learning high-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.

This work considers the problem of transfer learning in the context of reinforcement learning. Specifically, we consider training a policy in a reduced order system and deploying it in the full state system. The motivation for this training strategy is that running simulations in the full-state system may take excessive time if the dynamics are complex. While transfer learning alleviates the computational issue, the transfer guarantees depend on the discrepancy between the two systems. In this work, we consider a class of cascade dynamical systems, where the dynamics of a subset of the state-space influence the rest of the states but not vice-versa. The reinforcement learning policy learns in a model that ignores the dynamics of these states and treats them as commanded inputs. In the full-state system, these dynamics are handled using a classic controller (e.g., a PID). These systems have vast applications in the control literature and their structure allows us to provide transfer guarantees that depend on the stability of the inner loop controller. Numerical experiments on a quadrotor support the theoretical findings.

--Forest fires pose a significant threat to the environment, human life, and property. Early detection and response are crucial to mitigating the impact of these disasters. However, traditional forest fire detection methods are often hindered by our reliability on manual observation and satellite imagery with low spatial resolution. This paper emphasizes the role of transfer learning in enhancing forest fire detection in India, particularly in overcoming data collection challenges and improving model accuracy across various regions. We compare traditional learning methods with transfer learning, focusing on the unique challenges posed by regional differences in terrain, climate, and vegetation. Transfer learning can be categorized into several types based on the similarity between the source and target tasks, as well as the type of knowledge transferred. One key method is utilizing pre-trained models for efficient transfer learning, which significantly reduces the need for extensive labeled data. We outline the transfer learning process, demonstrating how researchers can adapt pre-trained models like MobileNetV2 for specific tasks such as forest fire detection. India is home to a vast and diverse range of forests, covering over 70 million hectares of land [1]. These forests are crucial not only for the country's ecosystem and biodiversity but also provide livelihoods for millions of people, particularly in rural areas. However, India's forests are facing a growing threat from forest fires, which can have devastating consequences for the environment, human life, and property [2]. Forest fires are a major concern in India, particularly during the summer months when temperatures are high and humidity is low. According to the Indian government, forest fires affect over 50, 000 hectares of land annually, causing significant economic losses and damage to the environment [3]. The country's forests are also home to a wide range of wildlife, including many endangered species which are threatened by fires. Figure 1 illustrates some images of the Uttarakhand, India, forest fire. Early detection and response are critical to mitigating the impact of forest fires. Traditional methods of forest fire detection, such as manual observation and satellite imagery with low spatial resolution, are often limited in their ability to detect fires quickly and accurately [4]. Manual observation is time-consuming and labour-intensive and may not be feasible in remote or inaccessible areas [5]. Satellite imagery with low spatial resolution may not be able to detect small fires or fires in areas with dense vegetation. In recent years, advances in deep learning and computer vision have enabled the development of more effective methods for forest fire detection. Convolutional neural networks (CNNs), in particular, have shown great promise in image classification tasks [6]-[10], including fire detection [4].

The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explored transfer learning using neural networks. In weight transfer, a model trained on the source domain is used as an initialization point for a network to be trained on the target domain. In deep metric learning, the source domain is used to construct an embedding that captures class structure in both the source and target domains. In few-shot learning, the focus is on generalizing well in the target domain based on a limited number of labeled examples.

Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization, such as hyperparameter optimization. Typically, BO relies on conventional Gaussian process (GP) regression, whose algorithmic complexity is cubic in the number of evaluations. As a result, GP-based BO cannot leverage large numbers of past function evaluations, for example, to warm-start related BO runs. We propose a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesian linear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared deep neural net. Experiments show that the neural net learns a representation suitable for warm-starting the black-box optimization problems and that BO runs can be accelerated when the target black-box function (e.g., validation loss) is learned together with other related signals (e.g., training loss).

The paper tackles multi-task learning problems where there are non-linear relationships between tasks. The relationships between tasks is encoded as a set of non-linear constraints that the outputs of each task must satisfy (e.g . In a nutshell, he proposed technique can be summarized as: use kernel regression to make predictions for each task independently, then project the prediction vector onto the constrained set. Overall, I like the idea of being able to take advantage of non-linear relationships between tasks. However, I am not sure how practical it is to specify the non-linear constraints between tasks in practice.

I went back and read the main paper one more time. This hybrid approach robustly outperforms every few-shot learning and every deep metric learning method previously proposed on k-ITL. " 2) L144-147: "In contrast, weight adaptation determines model parameters using both source and target domain data. We explore a straightforward hybrid, adapted embeddings, which unifies embedding methods and weight adaptation by using the target-domain support set for model-parameter adaptation" In plain English, this is just saying: "We use the test *and* train set to train embeddings in contrast to the standard practice of only using the train set" and it empirically worked slightly better. It's a no brainer that the performance increases as you also train on more (k) test data.

This paper applies both multi-task training and transfer learning to AutoML. The paper extends the ideas presented in the Neural Architectura Search (NAS) technique (Barret Zoph and Quoc V. Le. The authors maintain the two-layer solution, with one network "the controller" choosing the architectural parameters for the "child" network which is used to solve the targeted task. The performance of the child network is fed back to the controller network to influence its results. The novelty of this paper is in the way this two-layer solution is used.