Efficient Post-Training Augmentation for Adaptive Inference in Heterogeneous and Distributed IoT Environments