On the Role of Memory in Robust Opinion Dynamics