Modularity through Attention: Efficient Training and Transfer of Language-Conditioned Policies for Robot Manipulation