When does deep learning fail and how to tackle it? A critical analysis on polymer sequence-property surrogate models