Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings