Regularizing Second-Order Influences for Continual Learning