Towards Integrating Personal Knowledge into Test-Time Predictions