Towards Improving the Performance of Pre-Trained Speech Models for Low-Resource Languages Through Lateral Inhibition