Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences

Open in new window