A Adaptations of Algorithm 1 for different problems

A.1 Stochastic gradient descent We extend Algorithm 1 to stochastic gradient descent (SGD). Algorithm 2 provides the framework for teleportation in SGD. A.2 Data transformation Algorithm 3 here modifies Algorithm 1 to allow transformations on both parameters and data. The group actions on data at all teleportation steps can be precomposed as a function f and applied to the input data at inference time. In this section, we derive the group actions for the test functions and multi-layer neural networks.