Appendix A Distribution of Class Labels Across Each Probing Task

For the probing tasks such as Sentence Length, Tree Depth, and Top Constituents, we balance the number of classes to overcome the imbalance problem. We balance the classes for these three probing tasks as follows: (i) Sentence Length: 3-classes ( 5, 5-8 and 9), (ii) TreeDepth: 3-classes (5, 6-7 and 8), and TopConstituents: 2-classes (1, 2). For the probing tasks such as Sentence Length, Tree Depth, and Top Constituents, we balance the number of classes to overcome the imbalance problem. Figure 6 displays the common samples between class labels of pair of probing tasks. This reports whether the cells are balanced across class labels of different probing tasks. We also implemented the Iterative Null-Space Projection (INLP) method (Ravfogel et al., 2020) to verify whether our removal method performance is similar to previously proposed method.