$\textit{GeoHard}$: Towards Measuring Class-wise Hardness through Modelling Class Semantics