Offline Learning and Forgetting for Reasoning with Large Language Models