Learning to Recall with Transformers Beyond Orthogonal Embeddings