Decoupled Context Processing for Context Augmented Language Modeling