Fig. R1: Loss over effective learning time for the linear problem and different values of initial connectivity strength g and target value ˆz

We thank the reviewers for their positive and constructive feedback on the paper. We also addressed all the other comments, and they will appear in the revised version. We trained an LSTM network on the NLP task of sentiment analysis (Fig. R2). As in our paper, we found that the resulting changes to the network weights are of low rank. Note that one may not observe this behavior for any task and network off the shelf.