A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts