Interweaving Memories of a Siamese Large Language Model