The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding