Position: Stop Acting Like Language Model Agents Are Normal Agents