Adaptive Inference: Theoretical Limits and Unexplored Opportunities