When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected