Algorithm 1 Association Graph Learning (TRAININGTIME) Require: {Dtrt }Tt=1: Training sets of all tasks; T: Number of tasks; C: Number of all classes; E: Shared feature extractor; WT,WC: Parameters of metric functions in the association graph; L: Number of GNN layers; {Wl}Ll=1: Parameters of all GNN layers; {ft}Tt=1: Task-specific classifiers; λ: Learning rate. For clarity, we provide the algorithms during training and test in Algorithm 1 and Algorithm 2, respectively. Algorithm 2 Association Graph Learning (TESTTIME) Require: xt: one test instance from the t-th task; E: Trained the feature extractor; GT,GC: Trained task and class graph; L: Number of GNN layers; {Wl}Ll=1: Trained parameters of all GNN layers; ft: The trained task-specific classifier. In this section, we provide the class assignment of all datasets under different missing rates. Table B.1, B.2, B.3 shows the class assignment for Office-Home, Office-Caltechand ImageCLEF, respectively.

In this paper, we focus on multi-task classification, where related classification tasks share the same label space and are learned simultaneously. In particular, we tackle a new setting, which is more realistic than currently addressed in the literature, where categories shift from training to test data. Hence, individual tasks do not contain complete training data for the categories in the test set. To generalize to such test data, it is crucial for individual tasks to leverage knowledge from related tasks. To this end, we propose learning an association graph to transfer knowledge among tasks for missing classes.