our work interesting, timely and novel, and that our results demonstrate the fundamental limitations of Transformer

We thank the reviewers for their detailed comments and their useful suggestions. In this rebuttal, we report results on larger transformer models. We study the less understood issues related to how well TLMs are able to perform long chains of reasoning. This directly motivates us to investigate if language models can also learn certain reasoning strategies. We will add this discussion to the paper.