tosca
Sculpting [CLS] Features for Pre-Trained Model-Based Class-Incremental Learning
Yildirim, Murat Onur, Yildirim, Elif Ceren Gok, Vanschoren, Joaquin
Class-incremental learning requires models to continually acquire knowledge of new classes without forgetting old ones. Although pre-trained models have demonstrated strong performance in class-incremental learning, they remain susceptible to catastrophic forgetting when learning new concepts. Excessive plasticity in the models breaks generalizability and causes forgetting, while strong stability results in insufficient adaptation to new classes. This necessitates effective adaptation with minimal modifications to preserve the general knowledge of pre-trained models. To address this challenge, we first introduce a new parameter-efficient fine-tuning module 'Learn and Calibrate', or LuCA, designed to acquire knowledge through an adapter-calibrator couple, enabling effective adaptation with well-refined feature representations. Second, for each learning session, we deploy a sparse LuCA module on top of the last token just before the classifier, which we refer to as 'Token-level Sparse Calibration and Adaptation', or TOSCA. This strategic design improves the orthogonality between the modules and significantly reduces both training and inference complexity. By leaving the generalization capabilities of the pre-trained models intact and adapting exclusively via the last token, our approach achieves a harmonious balance between stability and plasticity. Extensive experiments demonstrate TOSCA's state-of-the-art performance while introducing ~8 times fewer parameters compared to prior methods.
Readers react: Why movie stars still matter
Is at Sea Without a News Anchor" [June 17]. Mary McNamara's view of network TV news hit the bull's eye and what she decries is, unfortunately, the result of the 24/7 news cycle and the out-of-control rampage of "get-it-now" satellite-delivered shout-casters. My list of the great ones would have included Chet Huntley and David Brinkley, both of whom brought dignity to the screen (much as Walter Cronkite did) as well as NBC News' original glass ceiling-buster, Pauline Frederick. Mary McNamara's column about a lack of a national news anchor that people can trust and respect has been true for years. That's the reason I abandoned TV news long ago and replaced it with NPR news. For me and many others, it is the only broadcast news that we pay attention to. Here I thought the malice America is facing has to do with a never-ending war in Afghanistan, high unemployment, etc., but no, the problem facing American is a lack of a good talking head. McNamara's piece bemoaning trusted anchors of ...
- Asia > Afghanistan (0.25)
- North America > United States > California > Los Angeles County > Los Angeles (0.06)
- North America > United States > District of Columbia > Washington (0.05)