MLPs Compass: What is learned when MLPs are combined with PLMs?