Fast Inference for Augmented Large Language Models