Tangent Model Composition for Ensembling and Continual Fine-tuning

The computational architecture of Transformers [52] has been leveraged extensively to co-opt Tangent Model Composition (TMC) is a method to combine the compositional structure of data through prompts or tokens component models independently fine-tuned around [29, 55], but still the activations of trained models do a pre-trained point. Component models are tangent vectors not appear to be meaningfully composable. Compositionality to the pre-trained model that can be added, scaled, of neural activity would allow one to combine activations or subtracted to support incremental learning, ensembling, from different models to capture novel concepts, or or unlearning. Component models are composed at inference incorporate knowledge from different data without having time via scalar combination, reducing the cost of ensembling to re-train or fine-tune the core models. This would enable to that of a single model. TMC improves accuracy open-universe classification and, more generally, combinatorial by 4.2% compared to ensembling non-linearly finetuned expansion of the hypothesis space. Continual learning models at a 2.5 to 10 reduction of inference cost, could be performed simply by composing models trained on growing linearly with the number of component models.