Context Length Alone Hurts LLM Performance Despite Perfect Retrieval