Following the Human Thread in Social Navigation