Continual Learning with Query-Only Attention