Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge