Convergence to Nash Equilibrium and No-regret Guarantee in (Markov) Potential Games