Note, the direct FC link and SC detour form a cyclic loop inherited from both SC and FC topology.

It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT.

This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling.

Even a well-calibrated LLM may struggle to estimate uncertainty during fine-tuning duetocatastrophic forgetting ofgeneral knowledge [84].

Large Language Models (LLMs) have the capacity to store and recall facts.

Exploration in sparse-reward reinforcement learning (RL) is difficult due to the need for long, coordinated sequences of actions in order to achieve any reward.

"Rule-of-Thumb" generation, show that our method consistently exceeds previous