Writing in the Margins: Better Inference Pattern for Long Context Retrieval